-
-
Notifications
You must be signed in to change notification settings - Fork 9.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide hooks freezing with for PyInstaller? #17184
Comments
Thanks @htgoebel. Providing a hook seems like a good thing for numpy to do. Things that would make sense to exclude I can think of:
I can imagine there's a use case somewhere for all of those, but for normal freezing I doubt any of that is useful. So providing an easy config for PyInstaller users to get a slimmed down package may make a lot of sense. |
@rgommers Sound good. Please let me know if you need assistance. One point the documentation might not be verbose enough: hooks are handled per module, so you could e.g. exclude |
Related discussion on PyInstaller hooks in #16163 |
I'm already on this - I'm going to migrate our current suite to numpy. |
Removing the milestone. Please correct me if I am wrong that the hooks are developed/maintained in If you need input from us, please don't hesitate to ping. |
@seberg we're waiting on one PR (pyinstaller/pyinstaller#5168) to be finished and merged before @bwoodsend sends a PR with our entire numpy compatability suite.
Currently that is the case, yes. However the idea is to move the hooks into numpy for multiple reasons, the main one being that PyInstaller has a much slower release cycle than numpy. If the hooks are in NumPy, any changes required in the hooks for a specific version of numpy will go live when that version of numpy is released, instead of a few months afterwards. And on top of that, the hooks only have to support one version of NumPy - the version they're bundled with. |
TBH I'm not so sure that this is necessary. I stopped working on PR (pyinstaller/pyinstaller#5168) in favour of my Conda support PR (still a work in progress) which will fix our issues with Conda+numpy in a more general way. Conda's numpy was our main incentive to transfer responsibility for this hook to someone who would know which DLLs numpy needs. Official PyPI (and even unofficial mkl) wheels are trivial for us to support. Because all DLLs numpy uses are kept in numpy's installation in from PyInstaller.utils.hooks import collect_dynamic_libs
# Collect all DLLs inside numpy's installation folder, dump them into built app's root.
binaries = collect_dynamic_libs("numpy", ".")
# Submodules PyInstaller can't detect (probably because they're only imported by extension modules).
hiddenimports = ['numpy.core._dtype_ctypes']
# Remove testing and building code and packages which are referenced but aren't really dependencies.
excludedimports = [
"scipy", "pytest", "nose", "distutils", "f2py", "setuptools", "numpy.f2py",
"numpy.distutils",
] This is:
So you can have the hook if you want it, but it's not such a big deal if this doesn't happen. Certainly, I'd agree that you keep this off any milestones. I'll submit a PR to numpy with the finalised hook once I get round to finishing up on PyInstaller's end (which will probably take me ages). |
@bwoodsend Even if the hook you propose looks trivial, it is not. See e.g. the list excluded imports, which needs to be maintained. And as @Legorooj wrote, PyInstaller has a much slower release cycle than numpy. Thus IMHO the best place to maintain the hook is the numpy project. |
I think we are happy to include it inside NumPy. Just to note that the 1.20 release is imminent, so if you would like a hook included it would be good to have a PR by next weekend. (That timeline is not quite set in stone, and I guess we may be able to do a bit later, as this probably touches no other code, so backporting is likely fine even after an rc is released. But it is not ideal.) |
Alright then. Including it in NumPy it is then. But it won't happen imminently so certainly don't wait for us. We will keep a copy of the hook we send to you for a period of time. You're copy will have precedence over ours so older NumPy versions (which don't contain the hook) will use our copy (which we know is compatible) but if later on you change NumPy and your copy of the hook then updated NumPys (which will contain the hook) will correctly use your updated copy of the hook. So in theory, PyInstaller will always choose the hook which matches the version of NumPy installed without the need to backport this into older versions. |
I am a bit curious whether a pluggy based system (what e.g. pytest uses), might be a simple long term solution allowing to develop hooks also as third party libraries. |
Hooks are declared with a |
Oh, yeah, that is what I was wondering. So you could already provide a hook with a faster release cadence. But I guess it doesn't really matter too much. It is probably also fine to include it into NumPy ( at least assuming there are not recurring issues around including conda hacks). |
Thinking about testing the hook ... I don't suppose that there are any plans to move NumPy's test suite out of the |
We tend to like the current structure, since the tests related to random are in numpy/random, the tests related to linalg are in numpy/linalg, etc. If we did refactor the tests to move to tests/random, tests/linalg, ... it would mean rebasing many of the open PRs. On the other hand, breaking out the tests might make the lives of projects like Numba and CuPy (and PyInstaller) easier. |
If you remove
It also loses the ability for testing on end user installs, where we often need it to detect issues with (e.g.) building against a different BLAS library. I believe that's the original reason for everything being in-tree. If we move it out, we'd need to install it as a separate package with matching version numbering to the main package.
Only if we restructure the tests and make it easier to import overrides I think. For reasonable reusability we'd need some configurability like https://github.com/data-apis/array-api-tests/. |
Sumodules of a package have to be imported from the same package. If I have two copies of NumPy findable in
Don't quote me on this but last time I did a similar refactoring, I found that if I did it as two commits, the first which uses only
Asking users to substitute their version into and download https://codeload.github.com/numpy/numpy/zip/refs/tags/v1.22.0 then pull out the But I don't really expect you to rip up the floorboards for PyInstaller's sake. I've got my answer. AFAIK, numpy doesn't do much lazy/delayed importing so a simple test will probably do. |
Ugh, I never liked that we changed to adding |
Another option may be to seperate the tests out from the main numpy package, into an external package which you could install via |
This will make PyInstaller a test dependency. Shall I make it a hard requirement and put it in |
I would prefer a soft |
some -> one? It'd be great for this kind of test to be orthogonal in test job definition, so if it fails then it fails in a single place. |
I'd rather have a separate package (e.g., We may do this at some point, maybe, but I think we need to have better reasons for doing it than just for PyInstaller's benefit. Making the test suite portable to other array libraries would be one such reason. Or binary/wheel size - if it'd give a significant size reduction (not sure it will), then that matters. Size matters quite a bit actually - a 1 MB wheel size reduction save ~1 PB/yr bandwidth usage on PyPI alone. |
IIRC, we discussed doing this when making the transition from nose to pytest. |
You must have had this suggestion before but just running |
Adding this special hook file tells PyInstaller what files a self contained NumPy application needs to run. A test is encluded to verify that the hook still finds all necessary files. Closes numpy#17184.
We do try to strip wheels we ship, see MacPython/numpy-wheels#19. Has that gone missing again? |
That's actually kind of what I mean, apologies; I assumed you'd be familiar with the concept of setuptool's extra's. What I was suggesting was splitting out the tests into a different python package, which would be an extra dependency of numpy. You'd still be able to install it via An example of use for this is PyInstaller's encryption feature - it needs an extra dependency so we recommend people install |
Adding this special hook file tells PyInstaller what files a self contained NumPy application needs to run. A test is included to verify that the hook still finds all necessary files. Closes numpy#17184.
Adding this special hook file tells PyInstaller what files a self contained NumPy application needs to run. A test is included to verify that the hook still finds all necessary files. Closes numpy#17184.
Adding this special hook file tells PyInstaller what files a self contained NumPy application needs to run. A test is included to verify that the hook still finds all necessary files. Closes numpy#17184.
Adding this special hook file tells PyInstaller what files a self contained NumPy application needs to run. A test is included to verify that the hook still finds all necessary files. Closes numpy#17184.
Adding this special hook file tells PyInstaller what files a self contained NumPy application needs to run. A test is included to verify that the hook still finds all necessary files. Closes numpy#17184.
Adding this special hook file tells PyInstaller what files a self contained NumPy application needs to run. A test is included to verify that the hook still finds all necessary files. Closes numpy#17184.
Adding this special hook file tells PyInstaller what files a self contained NumPy application needs to run. A test is included to verify that the hook still finds all necessary files. Closes numpy#17184.
PyInstaller maintainer here.
PyInstaller includes a "hook" for adapting it to the special needs and methods of numpy. This works quite well, anyway if numpy changes, it might take some time until PyInstaller catches up.
PyInstaller 4.0 (released a few weeks ago) added support for package developers to provide hooks with their packages. Thus numpy could deliver hooks including or excluding modules and files as needed. If you are interested in providing PyInstaller hooks with numpy, please have a look at package developers to provide hooks with their packages.
The text was updated successfully, but these errors were encountered: