-
Notifications
You must be signed in to change notification settings - Fork 7.2k
Description
🚀 Feature
We should relocate all the binary dependencies of torchvision on the conda recipes. That means that all torchvision conda distributions should package the extension binaries, as well as their own dependencies (i.e., libpng, libjpeg-turbo, ffmpeg), similar to what is done for wheel packaging.
Motivation
After PR #2777, we found that it is possible to have collisions between binary dependencies across some of the libraries that torchvision depends on, e.g., PyAV and torchvision's video_reader. This is very common on Windows, where the DLLs do not have an rpath, as opposed to ELF and MachO formats, which do have.
Also it is possible to have some conflicts on Mac and Linux when users try to mix FFmpeg versions between conda-forge and other sources different from the PyTorch channel ffmpeg
Pitch
The suggested relocation steps should occur as follows:
- During build time (
build
dependencies on meta.yaml), we list the binary packages that we require to build each of the torchvision extensions, i.e., libpng, libjpeg-turbo and ffmpeg (pytorch channel) - The recipe should compile torchvision as usual
- Once the compilation is done, we proceed to relocate the required binaries into the python package itself by renaming the shared libraries and updating the rpaths on Mac and Linux and using https://github.com/njsmith/machomachomangler on Windows DLLs
- We remove the already-packaged binaries from the runtime dependencies (
run
in meta.yaml).
By doing this, we're preventing any possible clash that could occur between user-installed versions of any of the aforementioned libraries and other packages that might depend on them.
Note: If users want a different version of any of the packages that we're bundling, they should recompile torchvision from scratch themselves.
Alternatives
Additional context
- On Linux we can use patchelf
- On Mac we can take a look at the delocate mechanism
- On Windows, we need to use https://github.com/njsmith/machomachomangler, since it is the only tool available to patch DLLs