Skip to content

Relocate and package binary dependencies in conda distributions #2866

@andfoy

Description

@andfoy

🚀 Feature

We should relocate all the binary dependencies of torchvision on the conda recipes. That means that all torchvision conda distributions should package the extension binaries, as well as their own dependencies (i.e., libpng, libjpeg-turbo, ffmpeg), similar to what is done for wheel packaging.

Motivation

After PR #2777, we found that it is possible to have collisions between binary dependencies across some of the libraries that torchvision depends on, e.g., PyAV and torchvision's video_reader. This is very common on Windows, where the DLLs do not have an rpath, as opposed to ELF and MachO formats, which do have.

Also it is possible to have some conflicts on Mac and Linux when users try to mix FFmpeg versions between conda-forge and other sources different from the PyTorch channel ffmpeg

Pitch

The suggested relocation steps should occur as follows:

  1. During build time (build dependencies on meta.yaml), we list the binary packages that we require to build each of the torchvision extensions, i.e., libpng, libjpeg-turbo and ffmpeg (pytorch channel)
  2. The recipe should compile torchvision as usual
  3. Once the compilation is done, we proceed to relocate the required binaries into the python package itself by renaming the shared libraries and updating the rpaths on Mac and Linux and using https://github.com/njsmith/machomachomangler on Windows DLLs
  4. We remove the already-packaged binaries from the runtime dependencies (run in meta.yaml).

By doing this, we're preventing any possible clash that could occur between user-installed versions of any of the aforementioned libraries and other packages that might depend on them.

Note: If users want a different version of any of the packages that we're bundling, they should recompile torchvision from scratch themselves.

Alternatives

Additional context

cc @fmassa @seemethere @malfet

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions