Discard header files (and static libraries on Windows) of torch
#666
Conversation
Yeah, and are not really the same. The first tells PyInstaller to simply collect the whole package directory and treat is as data files (there's shared libraries and extensions in there, but in PyInstaller 6.x those get sorted out by automatic binary-vs-data reclassification). The second tells PyInstaller to collect just data files, so shared libs are not necessarily collected (I think it depends on the name). That
But if we want the hook to be compatible with older PyInstaller versions, this should be done only for PyInstaller >= 6.0, while for older versions, the old approach should be used. That said, are you sure that headers and .lib files are not required by |
|
I've tried wrapping a function using Added: I didn't try packing an application that requires JIT. I just deleted the static libraries and headers then ran the codes. Besides, this is only tested on Windows. |
|
@rokm Do you have any idea? Or maybe we can enable this only on |
If you're asking about the failing linux test - you probably need to add I'll look at refactoring the hook, though. It's been on my TODO list for a while, albeit not exactly a high-priority thing... |
|
@rokm I've fixed it with |
|
Strictly speaking, the output of And as I wrote earlier, Maybe one more thing to consider - do we need to collect the data files that make it through the filter at all? On linux, most remaining files seem to be .pyi ones (which might be needed if there is a lazy-loader involved somewhere), .cmake files (which are likely unnecessary), a |
|
.pyi is unnecessary, too, I think Maybe a better solution could be like this? datas, binaries, hiddenimports = collect_all('torch', exclude_datas=["**/*.h", "**/*.hpp", "**/*.cuh", "**/*.lib", "**/*.pyi"]) |
I explicitly don't want to see |
|
Then what about using three separated calls one by one? |
Yes, that's what I originally described in #666 (comment) |
Sorry for making you explain again... |
No problem, it's good to see that someone else is willing to take a stab at modernizing that hook. |
|
I'd like to ask one thing: how does Update: from PyInstaller.utils.hooks import logger, collect_data_files, is_module_satisfies, collect_dynamic_libs, collect_submodules
datas = collect_data_files("torch", excludes=["**/*.h", "**/*.hpp", "**/*.cuh",
"**/*.lib", "**/*.cpp", "**/*.pyi", "**/*.cmake"])
binaries = collect_dynamic_libs("torch")
hiddenimports = collect_submodules("torch")
# With torch 2.0.0, PyInstaller's modulegraph analysis hits the recursion limit.
# So, unless the user has already done so, increase it automatically.
if is_module_satisfies('torch >= 2.0.0'):
import sys
new_limit = 5000
if sys.getrecursionlimit() < new_limit:
logger.info("hook-torch: raising recursion limit to %d", new_limit)
sys.setrecursionlimit(new_limit)I've tried this, all |
Hidden imports by itself won't solve the .py files problem. That's why you also need See https://pyinstaller.org/en/stable/hooks.html#hook-global-variables |
|
I still suspect we could do without the These new modifications, however, implicitly rely on functionality that's available only in recent PyInstaller versions; the module collection mode requires PyInstaller >= 5.3, and the shared lib parent directory preservation works across all OSes only as of PyInstaller 6.0. So I think we should put this new code block under Also, don't forget to add a news fragment for the change log (something along the lines of "Modernize the torch for I see you already ran oneshot test for the latest |
|
|
Yeah, I can reproduce this locally. It is probably due to the fact that we are analyzing all collected modules (like we should), and somehow we trigger the hook for |
|
https://github.com/CarlGao4/pyinstaller-hooks-contrib/actions/runs/7085858732 It's fixed after I added six as a dependency |
That's a way to work around it, yeah. But ultimately this is a bug that should be fixed (presumably) in PyInstaller's modulegraph. It's not your problem to fix, though. One thing I've noticed when investigating the above issue is that we now generate missing-hidden-import warnings, although those modules do seem to be collected in the end: Can you add a |
|
Nice work @CarlGao4, thank you! |
|
I have no idea what's going on here so if you think this ready Rok then just go ahead and hit the merge button. My vote would be arbitrary at this point. |
…23.11 (#82) Bumps [pyinstaller-hooks-contrib](https://github.com/pyinstaller/pyinstaller-hooks-contrib) from 2023.10 to 2023.11. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/releases">pyinstaller-hooks-contrib's releases</a>.</em></p> <blockquote> <h2>2023.11</h2> <p>Please see the <a href="https://www.github.com/pyinstaller/pyinstaller-hooks-contrib/tree/master/CHANGELOG.rst">changelog</a> for more details</p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/blob/master/CHANGELOG.rst">pyinstaller-hooks-contrib's changelog</a>.</em></p> <blockquote> <h2>2023.11 (2023-12-20)</h2> <p>New hooks</p> <pre><code> * Add a hook for ``freetype`` that collects the shared library that is bundled with ``freetype-py`` PyPI wheels. (`[#674](pyinstaller/pyinstaller-hooks-contrib#674) <https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/674>`_) * Add a hook for ``z3c.rml`` that collects the required subset of Bitstream Vera TTF fonts from the ``reportlab`` package. (`[#674](pyinstaller/pyinstaller-hooks-contrib#674) <https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/674>`_) * Add hook for ``eth_rlp``. (`[#672](pyinstaller/pyinstaller-hooks-contrib#672) <https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/672>`_) * Add hook for ``eth_typing`` which requires its package metadata. (`[#656](pyinstaller/pyinstaller-hooks-contrib#656) <https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/656>`_) * Add hook for ``eth_utils`` to collect its embedded JSON files. (`[#656](pyinstaller/pyinstaller-hooks-contrib#656) <https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/656>`_) * Add hook for ``rlp``. (`[#672](pyinstaller/pyinstaller-hooks-contrib#672) <https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/672>`_) * Add hook for ``sspilib`` that collects submodules of ``sspilib.raw``, most of which are cythonized extensions. (`[#669](pyinstaller/pyinstaller-hooks-contrib#669) <https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/669>`_) <p>Updated hooks </code></pre></p> <ul> <li>Modernize the hook for <code>torch</code> and reduce the amount of unnecessarily collected data files (header files and static libraries). Requires PyInstaller >= 6.0. (<code>[#666](pyinstaller/pyinstaller-hooks-contrib#666) <https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/666></code>_)</li> <li>Update <code>pyarrow</code> hook to collect all of the package's submodules. (<code>[#662](pyinstaller/pyinstaller-hooks-contrib#662) <https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/662></code>_)</li> <li>Update <code>rtree</code> hook for compatibility with <code>Rtree >= 1.1.0</code>. (<code>[#657](pyinstaller/pyinstaller-hooks-contrib#657) <https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/657></code>_)</li> <li>Update <code>sudachipy</code> hook for <code>sudachipy</code> 0.6.8. (<code>[#673](pyinstaller/pyinstaller-hooks-contrib#673) <https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/673></code>_)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/3bd34d400863c8b0466bf1e84a57b6814e13476e"><code>3bd34d4</code></a> Release v2023.11</li> <li><a href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/11dff78740e2dc7b6685388c70c8e9ba0e07a03a"><code>11dff78</code></a> hooks: add hook for freetype</li> <li><a href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/7916492b83f98674529594083e2a6ba1e294ccbd"><code>7916492</code></a> hooks: add hook for z3c.rml</li> <li><a href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/937ca15b5f4d17ed0421e4a3fe2bcafd234954a9"><code>937ca15</code></a> Scheduled weekly dependency update for week 51 (<a href="https://redirect.github.com/pyinstaller/pyinstaller-hooks-contrib/issues/671">#671</a>)</li> <li><a href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/6a303f6c0bd7c2c51adf0418ed0cef19d647bb76"><code>6a303f6</code></a> tests: fix deprecation warnings in test_sudachipy</li> <li><a href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/6cb34e6a78b8a1368297cbef80bf96605cafe15b"><code>6cb34e6</code></a> hooks: sudachipy: add hiddenimports for v0.6.8</li> <li><a href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/97800abfb16f3e4a54bcc38f752b924a9e93d9ba"><code>97800ab</code></a> tests: repin sudachipy to latest version</li> <li><a href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/23cd820137a8905b018cc1c56d4e2d0a347bd9f4"><code>23cd820</code></a> ci: install sudachidict packages on sudachipy update</li> <li><a href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/f8aaa39b4dac28b8da62f3b2f5633425ac5fd153"><code>f8aaa39</code></a> tests: fix dependencies of sudachipy test</li> <li><a href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/f9c592dfcd7491cf6a6d33adde89fdf413d0914f"><code>f9c592d</code></a> hooks: add hook for eth_rlp</li> <li>Additional commits viewable in <a href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/compare/2023.10...2023.11">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
I've read #531 and found a new solution.
We know thousands of headers are included in the wheel of torch, which are provided for situations like libraries which would be compiled when being installed as they should use the same version of CUDA, because none of source codes of PyTorch is included. So those header files can be simply filtered to reduce file amount in the packed application (so it can be moved or copied easier)
I still encountered a problem, though. After modifying these, tests on Linux will fail:
Click to expand