Segmentation fault when accessing a large object after successful reading of that object from file

[Gateway2745]

System: M1 Macbook Pro (macOS Monterey), miniconda

I have frozen a single python script and its associated data (using the --add-data option) to a single dir (using --one-dir). This freezing runs without any errors.

Inside this script, I have a function which reads in one of the data files. Specifically, I have state_dict = torch.load('weights/model.pth', map_location='cpu') , which is a PyTorch function to read model weights from a saved file.
This reading of weights also runs successfully.

However, when I have a statement that accesses these weights such as x=state_dict['config'] or even print(state_dict), the program crashes with zsh: segmentation fault. There is no other output.

The python script runs successfully without pyinstaller, which is why I do not suspect any memory issues.
Could you please help me out? Thank you.

Additional: I had also encountered another error previously stating that libomp.dylib was missing, although it was already there in another folder. So, I copied libomp.dylib to the folder where pyinstaller wasn't able to find it. This duplication of libomp.dylib raised a warning Initializing libomp.dylib, but found libomp.dylib already initialized which I silenced by export KMP_DUPLICATE_LIB_OK=TRUE. The warning mentioned that this may cause issues. Could this be the reason behind seg fault?

Similar Issue - https://stackoverflow.com/questions/64575925/pyinstaller-exe-crashing-during-run-without-error-message

[rokm]

Can you check if libraries listed in pyinstaller/pyinstaller-hooks-contrib#375 (comment) are duplicated in your build?

[Gateway2745]

@rokm Yes, they are duplicated! Let me try making the top-level files as symlinks to the ones in torch/lib/, as you suggested. Thank you.

[Gateway2745]

@rokm Thank you, it now works in --one-dir mode! In my case, I only had to move libomp.dylib to torch/lib/libomp.dylib, and then create a symlink libomp.dylib -> torch/lib/libomp.dylib.

However, to get it working for --one-file mode, I did not fully understand where we need to make these changes, as you mentioned here. Can you please explain this? Thank you.

[rokm]

@rokm Thank you, it now works in --one-dir mode! In my case, I only had to move libomp.dylib to torch/lib/libomp.dylib, and then create a symlink libomp.dylib -> torch/lib/libomp.dylib.

However, to get it working for --one-file mode, I did not fully understand where we need to make these changes, as you mentioned here. Can you please explain this? Thank you.

You need to make the changes (remove the duplicates, create the links) programmatically, at the start of your python program. I.e., remove the offending files in the temporary directory indicated by sys._MEIPASS, and link corresponding files from os.path.join(sys._MEIPASS, 'torch', 'lib') to sys._MEIPASS.

It's inelegant and wasteful, but it's currently the only way for onefile mode if you really have to use it.

[rokm]

Or, you could remove the offending duplicates if you build via .spec file instead of directly from .py. In that case, you could filter Analysis.binaries and/or Analysis.datas before they are passed to PYZ, EXE, COLLECT, etc., to ensure that the duplicates are not collected in the first place.

You'll still need to create symlinks programmatically at the program start, though.

[Gateway2745]

@rokm Thank you so much! I've got it working as a --onefile too now!