macOS Pyinstaller: libc++abi: terminating with uncaught exception of type c10::Error: Type c10::intrusive_ptr<ConvPackedParamsBase<2>> could not be converted to any of the known types

[CaptainStiggz]

🐛 Describe the bug

I'm trying to package a python app using pyinstaller.

Torch: 1.12.1
Pyinstaller: 5.3
OS: macOS 12.5 (M1)

[UPDATE] This appears to be a problem with torchvision. If I remove the dependency, no crash.

When I try to run my UNIX executable, I get the following:

[41110] WARNING: file already exists but should not: /var/folders/9l/0lk2nvvs7j39q893l1k0qvcc0000gn/T/_MEI9VPQ38/pyarrow/lib.cpython-310-darwin.so
[41110] WARNING: file already exists but should not: /var/folders/9l/0lk2nvvs7j39q893l1k0qvcc0000gn/T/_MEI9VPQ38/torch/_C.cpython-310-darwin.so
[41110] WARNING: file already exists but should not: /var/folders/9l/0lk2nvvs7j39q893l1k0qvcc0000gn/T/_MEI9VPQ38/torch/_C_flatbuffer.cpython-310-darwin.so
[41110] WARNING: file already exists but should not: /var/folders/9l/0lk2nvvs7j39q893l1k0qvcc0000gn/T/_MEI9VPQ38/torch/_dl.cpython-310-darwin.so
libc++abi: terminating with uncaught exception of type c10::Error: Type c10::intrusive_ptr<ConvPackedParamsBase<2>> could not be converted to any of the known types.
Exception raised from operator() at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/core/jit_type.h:1735 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) + 81 (0x18a356951 in libc10.dylib)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 98 (0x18a355022 in libc10.dylib)
frame #2: c10::detail::getTypePtr_<c10::intrusive_ptr<ConvPackedParamsBase<2>, c10::detail::intrusive_target_default_null_type<ConvPackedParamsBase<2> > > >::call()::'lambda'()::operator()() const + 276 (0x1d894caa4 in libtorch_cpu.dylib)
frame #3: c10::Type::SingletonOrSharedTypePtr<c10::Type> c10::getTypePtrCopy<c10::intrusive_ptr<ConvPackedParamsBase<2>, c10::detail::intrusive_target_default_null_type<ConvPackedParamsBase<2> > > >() + 27 (0x1d894c82b in libtorch_cpu.dylib)
frame #4: c10::detail::infer_schema::(anonymous namespace)::createArgumentVector(c10::ArrayRef<c10::detail::infer_schema::ArgumentDef>) + 210 (0x1d82fb0a2 in libtorch_cpu.dylib)
frame #5: c10::detail::infer_schema::make_function_schema(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&, c10::ArrayRef<c10::detail::infer_schema::ArgumentDef>, c10::ArrayRef<c10::detail::infer_schema::ArgumentDef>) + 86 (0x1d82fae56 in libtorch_cpu.dylib)
frame #6: c10::detail::infer_schema::make_function_schema(c10::ArrayRef<c10::detail::infer_schema::ArgumentDef>, c10::ArrayRef<c10::detail::infer_schema::ArgumentDef>) + 67 (0x1d82fb893 in libtorch_cpu.dylib)
frame #7: std::__1::unique_ptr<c10::FunctionSchema, std::__1::default_delete<c10::FunctionSchema> > c10::detail::inferFunctionSchemaFromFunctor<at::Tensor (*)(at::Tensor, c10::intrusive_ptr<ConvPackedParamsBase<2>, c10::detail::intrusive_target_default_null_type<ConvPackedParamsBase<2> > > const&, double, long long)>() + 85 (0x1d89e5945 in libtorch_cpu.dylib)
frame #8: torch::CppFunction::CppFunction<at::Tensor (at::Tensor, c10::intrusive_ptr<ConvPackedParamsBase<2>, c10::detail::intrusive_target_default_null_type<ConvPackedParamsBase<2> > > const&, double, long long)>(at::Tensor (*)(at::Tensor, c10::intrusive_ptr<ConvPackedParamsBase<2>, c10::detail::intrusive_target_default_null_type<ConvPackedParamsBase<2> > > const&, double, long long), std::__1::enable_if<c10::guts::is_function_type<at::Tensor (at::Tensor, c10::intrusive_ptr<ConvPackedParamsBase<2>, c10::detail::intrusive_target_default_null_type<ConvPackedParamsBase<2> > > const&, double, long long)>::value, std::nullptr_t>::type) + 122 (0x1d89e586a in libtorch_cpu.dylib)
frame #9: at::native::(anonymous namespace)::TORCH_LIBRARY_IMPL_init_quantized_QuantizedCPU_4(torch::Library&) + 35 (0x1d89e3b93 in libtorch_cpu.dylib)
frame #10: torch::detail::TorchLibraryInit::TorchLibraryInit(torch::Library::Kind, void (*)(torch::Library&), char const*, c10::optional<c10::DispatchKey>, char const*, unsigned int) + 203 (0x1d81a762b in libtorch_cpu.dylib)
frame #11: _GLOBAL__sub_I_qconv.cpp + 178 (0x1d89e8cb2 in libtorch_cpu.dylib)
frame #12: invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 182 (0x220f2ee4f in dyld)
frame #13: invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 129 (0x220f55911 in dyld)
frame #14: invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 557 (0x220f4ce26 in dyld)
frame #15: dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 129 (0x220f1bdb3 in dyld)
frame #16: dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 179 (0x220f4cbb7 in dyld)
frame #17: dyld3::MachOAnalyzer::forEachInitializerPointerSection(Diagnostics&, void (unsigned int, unsigned int, unsigned char const*, bool&) block_pointer) const + 118 (0x220f55342 in dyld)
frame #18: dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 386 (0x220f555b4 in dyld)
frame #19: dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 144 (0x220f2ed82 in dyld)
frame #20: dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 178 (0x220f2ef0e in dyld)
frame #21: dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 149 (0x220f2eef1 in dyld)
frame #22: dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 149 (0x220f2eef1 in dyld)
frame #23: dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const + 108 (0x220f2efb2 in dyld)
frame #24: dyld4::APIs::dlopen_from(char const*, int, void*) + 592 (0x220f3de00 in dyld)
frame #25: py_dl_open + 135 (0x1457cef77 in _ctypes.cpython-310-darwin.so)
<omitting python frames>

Versions

[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.1
[pip3] pytorch-lightning==1.4.2
[pip3] torch==1.12.1
[pip3] torch-fidelity==0.3.0
[pip3] torchaudio==0.12.1
[pip3] torchmetrics==0.6.0
[pip3] torchvision==0.13.1
[conda] numpy 1.23.1 pypi_0 pypi
[conda] pytorch-lightning 1.4.2 pypi_0 pypi
[conda] torch 1.13.0.dev20220907 pypi_0 pypi
[conda] torch-fidelity 0.3.0 pypi_0 pypi
[conda] torchmetrics 0.6.0 pypi_0 pypi
[conda] torchvision 0.14.0.dev20220907 pypi_0 pypi

cc @malfet @zou3519

[CaptainStiggz]

Looks like importing from pytorch_lightning causes the same error, but could still be an import of torchvision under the hood.

[malfet]

Hmm, oncall: jit feels like a wrong tag here (as it has nothing to do with it, likely crash happens because libtorch_cpu somehow getting packaged twice by pyinstaller....

At least I can reproduce the crash while trying to package something as simple as:

#!/usr/bin/env python3
import torch
import torchvision

def main():
    print(torch.__version__)
    print(torchvision.__version__)

if __name__ == "__main__":
    main()

[CaptainStiggz]

I found the workaround from this thread to work for the time-being. After pyinstaller finishes, I can run:

rm libtorch_python.dylib
rm libtorch.dylib
rm libc10.dylib
rm libtorch_cpu.dylib

ln -s ./torch/lib/libtorch_python.dylib libtorch_python.dylib
ln -s ./torch/lib/libtorch.dylib libtorch.dylib
ln -s ./torch/lib/libc10.dylib libc10.dylib
ln -s ./torch/lib/libtorch_cpu.dylib libtorch_cpu.dylib

pyinstaller/pyinstaller-hooks-contrib#375 (comment)

This indicates it might be a pyinstaller bug, rather than a torch issue.

[malfet]

@CaptainStiggz thank you for confirming. I.e. it does not look like the bug is in PyTorch or TorchVision, but rather in a way how pyinstaller packages those libraries together. Do you mind opening an issue there and I would gladly coordinate with them if there is a metadata we can provide in the PyTorch dependent packages to avoid this double-packing issue

[seemethere]

Meta question, should we transfer this issue to pytorch/vision since this is more of an issue with vision than core pytorch?

[malfet]

I don't think it's specific to torchvision (nor to M1 to be frank)