can't find torchtext dylib files #375
Comments
|
It's in That said, those warnings are result of our dependency scanner lacking support for proper interpretation of rpaths, and are often just warnings. Those libraries may end up collected anyway, due to other dependency chains. In fact, trying to freeze a simple
|
|
this works, thank you! |
|
Romk, Can you please explain further your comment : " The only way to get the program working is to replace the libraries in top-level directory with symbolic links" What top-level directory? I am asking because I think I'm seeing the same problem. When I run pyinstaller I see the following warnings: Then when I run the code I get the following messages: |
Top-level application directory of a |
|
I think I'm not understanding something as it's still not working. (same missing dylib errrors) (I am actually trying a onefile build, but did test a onedir build with the same results) I am running pyinstaller from the same directory where I created the linked files. And here is the pyinstaller command:
|
If you find yourself adding your python's site-packages to search path, you're likely doing something wrong. If it actually solves some kind of problem, then you either have not activated your virtual environment, or you have installed PyInstaller in an environment that is different from your target virtual environment. In either case, you're not going to be in for a pleasant experience - better sort out your environments before trying anything else.
Are these really necessary?
Don't use onefile until you have a onedir build up and running. The latter is much easier to debug in terms of checking for missing files.
Just for the record, these symlinks are not what we were discussing earlier in this thread. We were talking about manually modifying the generated The error you were getting,
indicates that Create a onedir build and check what the actual situation is. (Although it is likely that |
Any further suggestions? |
This pile of errors is probably due to duplicated copies, as was discussed before in this thread. So you'll need to replace the copies in top-level directory with symlinks to the copies in corresponding sub-directories. This goes for the torch dylibs that I listed earlier in #375 (comment), but may also apply to the |
|
Thanks - I now have a onedir build that is able to run! After running pyinstaller I needed to:
Now that this is running, how do I make these same changes with a onefile build??? |
Since PyInstaller does not support symlinks in a onefile build, the only way to make this work in a onefile build at the moment is to perform these cleanup steps at the beginning of your python program. EDIT: and the missing |
|
Thanks so much - I now have it fully working as a single file distribution ! Your responsiveness has been amazing. |
|
I also encountered this. I found that there is no hook for torchaudio but only for torch and torchvision. The hook for torch is very simple: But all those binaries are packed. And when I run the program, I got this error: Sometimes it is not the error above: I tried to use otool and found the copied binary is different from the oringinal: The error is same when I use I can't copy them after the build is finished, because I need a |
You can modify the generated .app bundle and then re-sign it manually... |
|
Is there a similar workaround (for either the |
This issue and its work-around are macOS-specific, so no. If you have problems on Windows, open a new issue and provide the error traceback (as well as python and pyinstaller version), so we can see what's going on. |
|
Still having this issue with torch 1.12.1, pyinstaller 5.6.2 and pyinstaller-hooks-contrib 2022.13:
I added the symlinks, but the error stays the same. I deleted the libs and symlinks in the top folder, same issue. I deleted the lib folder in pytorch in the onedir, same issue. Not sure what else to do, does anyone have an idea? |
When you were creating symlinks and removing libs, did you actually see That said, if I freeze a simple |
|
Sure, I actually made a test with a new environment, so let me share it with you: I created a conda env with only pyinstaller and pytorch. I used conda to install pytorch, with:
Then I wrote a simple script, that runs correctly: Then, I just ran pyinstaller with its default settings:
And this is the full error I have: It did collect the libs in pytorch/lib: libc10.dylib And in the "base" directory, I also have the mkl libs: libmkl_intel_ilp64.1.dylib |
|
So first, unless you absolutely have to use Anaconda, I suggest you to switch to python.org python + pip-installed packages when building with PyInstaller - because in addition to spotty Anaconda support (on top of already less-than-ideal torch support), you'll also end up with bloated builds, because we don't handle symlinks correctly, and so versioned dylibs collected from anaconda installation will end up as separate copies (instead of a single copy and symlinks). Back to the issue, though: the problem is that This can be fixed by manually adjusting the rpath to point to correct directory: Unfortunately, that's not the only lib with such problem. (and so on for any other MKL library that has problem finding Changing/adding rpath invalidates the signature (even if it is just ad-hoc one), so you might need to re-sign the modified libraries as well. If you did not use signing identity with PyInstaller, then you can just re-sing with ad-hoc identity, which is what PyInstaller uses by default. |
|
As per your instructions, I recreated the environment with pip. However, I still have an issue with torchvision. I could create a new issue for it, but I wanted to show it to you here: I found on stackoverflow to change the compat.py file in pyinstaller and replaced
with
however, it doesn't look like a great solution, and doesn't solve the issue anyway. |
|
This looks like an encoding problem with the source file that's required by torch's JIT. Based on the traceback, I think it should be Has this file been collected by PyInstaller (since we typically do not collect sources, and we do not have a |
|
I only have the .pyc file in torchvision in the pyinstaller, so it has not been collected. I really appreciate all your swift help, thank you. |
Are you building with
Like before, there was likely more to that error than only last line (e.g., missing dependency). If you copied the whole torchvision folder, you probably overwrote the shared libraries that PyInstaller processed. Instead of manual copying, can you try adding |
|
Rebuilding the env with venv instead of conda and using |
|
I had the same problem, and I solved it by rm & ln -s the dylibs in the torch.
How can I do this exactly? Which option should I use? |
There's no option for this. You need to implement the required steps in your python code, at the start of your program (using |
|
I have written code like this. import os
import stanza
import sys
for filename in sorted(os.listdir('.')):
print(f"check {filename}")For --onedir, os.remove and os.symlink are possible. And segmentfaults could be resolved. $ pyinstaller main.py --onedir
$ cd dist/main
$ ./main
check PIL
check base_library.zip
check certifi
check charset_normalizer
check google
check lib-dynload
check libXau.6.0.0.dylib
check libbrotlicommon.1.0.9.dylib
(snip)In --onefile how can I find the following files?
$ pyinstaller main.py --onefile
$ cd dist/main
$ ./main
check main |
Instead of using |
|
Wow, it worked. Thank you very much. |
|
Has this issue been fixed since torch and torchaudio, torchtext 2.0 has been released? I'm no longer using macOS, but I had to manually copy the missing DLLs on Windows when I encountered this problem on macOS. I have just tested again, and my Windows application functions well without any further operation after packing with PyInstaller, so I guess that the issue will also be fixed once pytorch (and other torch utilities) is updated to 2.0 Can anybody try that? |
Pyinstaller builds successfully, however when I run the exe I get an error related to missing torchtext dylib files.
I get the following warnings during the freezing:
I can find the last 4 dylib files (except
libc++.1.dylib) in my env/torch module and can copy them manually to the dist folder but that doesn't seem like a good workaround and I still can't findlibc++.1.dylib. Updating path / binaries in .spec file doesn't help.Any ideas on how to resolve this?
pyinstaller --version:4.83.90.11.21.10.2MacOS 12.1-->The text was updated successfully, but these errors were encountered: