Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate TOC class and remove its usage from codebase #7615

Merged
merged 18 commits into from May 10, 2023

Commits on May 9, 2023

  1. depend: imphookapi: rework TOC-style tuple detection

    Rework TOC-style 3-tuple detection in `add_datas` and `binaries`;
    instead of checking for instance of `TOC` class, check if the
    length of the first entry is three.
    
    This will allow us to replace the `TOC` class with regular list.
    rokm committed May 9, 2023
    Configuration menu
    Copy the full SHA
    609d9ba View commit details
    Browse the repository at this point in the history
  2. building: splash: remove reference to TOC class

    Instantiate `self.binaries` as a regular list instead of an
    instance of the `TOC` class; we do not need any of its functionality
    here.
    rokm committed May 9, 2023
    Configuration menu
    Copy the full SHA
    132f673 View commit details
    Browse the repository at this point in the history
  3. building: splash: clean up string formatting in logger method calls

    Use the lazy string formatting offered by logger method calls
    instead of directly formatting strings using the % syntax.
    rokm committed May 9, 2023
    Configuration menu
    Copy the full SHA
    24e57a7 View commit details
    Browse the repository at this point in the history
  4. depend: remove use of TOC class in get_bootstrap_modules

    We are returning a simple list of modules, so there's no need for
    TOC class there as we know the list has no duplicates.
    rokm committed May 9, 2023
    Configuration menu
    Copy the full SHA
    e0e0ae0 View commit details
    Browse the repository at this point in the history
  5. depend: remove optional TOC argument to _make_toc() helper

    There is exactly one call that passes the existing TOC in our
    codebase, and even there we can simply append the output to the
    existing TOC.
    
    The returned toc is now a list as opposed to instance of the TOC
    class. This removes implict deduplication functionality, but
    should not be an issue at this step, because entries are generated
    from the module graph (and so there should not be any duplicates
    here).
    rokm committed May 9, 2023
    Configuration menu
    Copy the full SHA
    ec13909 View commit details
    Browse the repository at this point in the history
  6. building: have Tree inherit from list instead of TOC

    Turn `Tree` class into child of regular `list` instead of `TOC`
    class, as part of on-going `TOC` class depreciation.
    
    As `Tree` describes the contents of the filesystem, implicit TOC
    de-duplication should not be required in the first place.
    
    Also initialize `Target.dependencies` with plain `list`.
    rokm committed May 9, 2023
    Configuration menu
    Copy the full SHA
    73e202e View commit details
    Browse the repository at this point in the history
  7. building: begin replacing TOC class with normalization helper function

    For now, the normalization helper function is a stub that internally
    uses the TOC class; this allows us to gradually limit/encapsulate
    the use of TOC class within the codebase.
    rokm committed May 9, 2023
    Configuration menu
    Copy the full SHA
    2792fc2 View commit details
    Browse the repository at this point in the history
  8. building: main: move extension processing further up in workflow

    Process extensions (rename them to full filenames) immediately
    after we obtain the list of extensions from the modulegraph.
    rokm committed May 9, 2023
    Configuration menu
    Copy the full SHA
    7634f85 View commit details
    Browse the repository at this point in the history
  9. building: main: remove direct uses of TOC class

    Instead of using TOC class, perform explicit TOC list normalization
    at certain build stages.
    rokm committed May 9, 2023
    Configuration menu
    Copy the full SHA
    1f2bed8 View commit details
    Browse the repository at this point in the history
  10. building: work around the issue with passing code cache around

    When `Analysis.pure` TOC list was an instance of the `TOC` class,
    we could dynamically add an attribute `_code_cache` to it, and use
    that to pass the code cache dictionary around. The `PYZ` writer
    would then check for the presence of the attribute on the received
    arguments and extend its internal code cache accordingly.
    
    Now that the TOC list is a plain `list`, the custom attribute
    cannot be added anymore. To work around this, we store new
    dictionary in global `CONF['code_cache']`, where we associate
    the list's `id()` with its code cache dictionary. This gives us
    the same semantics as the old approach.
    rokm committed May 9, 2023
    Configuration menu
    Copy the full SHA
    5d9cab5 View commit details
    Browse the repository at this point in the history
  11. tests: fix CONF override in test_issue_2492 and test_issue_5131

    The CONF override in `test_issue_2492` and `test_issue_5131`
    should initialize the newly-introduced `code_cache` dictionary.
    rokm committed May 9, 2023
    Configuration menu
    Copy the full SHA
    b59792f View commit details
    Browse the repository at this point in the history
  12. building: splash: fix detection of tkinter usage

    The `Splash` target is trying to detect whether user's code already
    uses `_tkinter` by looking for extension in the `binaries` TOC.
    However, it uses the `filenames` property of the `TOC` class, which
    is not available anymore now that the TOC lists have been switched
    to plain `list`.
    
    Futhermore, as it is trying to look up `_tkinter` extension by
    module name, this means that the detection has been effectively
    broken since pyinstaller#7273 where we pushed conversion of extension module
    names to full extension filenames into `Analysis`. So the search
    in `binaries` needs to be done with full extension filename rather
    than module name.
    rokm committed May 9, 2023
    Configuration menu
    Copy the full SHA
    7527eaf View commit details
    Browse the repository at this point in the history
  13. building: implement TOC list normalization helpers

    Replace the TOC list normalization stubs with proper implementation
    that encodes priority information. This for example ensures that
    if a file is collected both as a DATA and BINARY, we always treat
    is as BINARY (and hence subject it to additional binary processing).
    rokm committed May 9, 2023
    Configuration menu
    Copy the full SHA
    f83dad6 View commit details
    Browse the repository at this point in the history
  14. building: EXE: remove the work-around for merging PYZ.dependencies

    We now have TOC normalization with priority system in place, so
    we can directly extend the EXE's TOC with entries from all passed
    targets' dependencies.
    rokm committed May 9, 2023
    Configuration menu
    Copy the full SHA
    0603e01 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    e6f6f1f View commit details
    Browse the repository at this point in the history

Commits on May 10, 2023

  1. building: add deprecation warning to TOC class

    And ignore it in the TOC class unit tests.
    rokm committed May 10, 2023
    Configuration menu
    Copy the full SHA
    69e6130 View commit details
    Browse the repository at this point in the history
  2. docs: update documentation on TOC lists

    Update documentation on TOC lists now that the TOC class has
    been officially deprecated.
    rokm committed May 10, 2023
    Configuration menu
    Copy the full SHA
    cf95497 View commit details
    Browse the repository at this point in the history
  3. building: ensure TOC de-duplication when dest_name contains pardir loops

    Ensure that TOC is properly de-duplicated even if dest_name
    contains loops with parent directory path components. For example,
    `numpy/core/../../numpy.libs/libquadmath-2d0c479f.so.0.0.0` and
    `numpy/linalg/../../numpy.libs/libquadmath-2d0c479f.so.0.0.0`
    should be considered duplicates, as they are both normalized to
    `numpy.libs/libquadmath-2d0c479f.so.0.0.0`.
    
    Therefore, we now have the TOC normalization helpers to always
    sanitize the `dest_name` using `os.path.normpath` (with `pathlib`
    lacking the equivalent functionality), so that the entries are
    properly de-duplicated and that destination name is always in
    its compact/normalized form.
    
    We should probably also look into path normalization in the
    `bindepend.getImports` function, but at the end of the day,
    the TOC normalization serves as the last guard against problematic
    entries.
    rokm committed May 10, 2023
    Configuration menu
    Copy the full SHA
    97bc98e View commit details
    Browse the repository at this point in the history