Publish DGL Package in conda-forge channel

[bgawrych]

Publish DGL Package in conda-forge channel

Abstract

The proposal recommends adding the DGL (Deep Graph Library) package to the conda-forge channel. This move will benefit the open-source community in multiple ways. Firstly, it will make it easier for users to install and manage DGL while ensuring compatibility with other libraries from the same channel. Secondly, it will enable faster updates, promote collaboration, and improve overall accessibility of the library.

Motivation

The motivation behind publishing the DGL package in the conda-forge channel is to address library dependency conflicts commonly encountered when managing dependencies from different sources. By including DGL in conda-forge, users can install and use DGL alongside other packages without encountering compatibility issues. This simplifies the management of dependencies and provides a reliable and consistent environment for users. The publication of DGL in conda-forge aims to enhance the accessibility, usability, and collaboration within the DGL community while mitigating the risks associated with library dependency conflicts.

Advantages of Publishing DGL in conda-forge

Dependency Compatibility: The publication of the DGL package in conda-forge guarantees compatibility with other libraries in the channel, which users can depend on. conda-forge uses strict testing and quality assurance procedures to ensure that packages function smoothly together. This results in a stable and dependable environment for users, free from conflicts and compatibility problems.

Package Manager Integration: conda-forge, as a well-integrated platform with widely-used package managers such as Conda and Mamba, provides a seamless integration opportunity. By publishing the DGL package in the conda-forge channel, users will gain effortless access to the library, enabling them to manage DGL seamlessly within their current environments and alongside their existing packages. This integration not only promotes collaboration but also enhances interoperability with other Python libraries, ultimately strengthening the overall ecosystem.

Cross-Platform Compatibility: conda-forge supports various operating systems, such as Windows, macOS, and Linux distributions. The publication of DGL in the conda-forge channel guarantees that users on different platforms can effortlessly access and use DGL.

Faster Updates and Bug Fixes: conda-forge provides an automated build system, ensuring that the DGL package is regularly updated to the latest stable release. Once the DGL package is published in conda-forge, the automated build process track PyPI/GitHub releases and ensures that updates, including new features, bug fixes, and performance improvements, are promptly incorporated into the package. This allows users to effortlessly keep their installations up-to-date and benefit from the latest advancements in DGL.

Links:

• conda-forge documentation
• Contributing packages
• Maintaining packages

[BarclayII]

Should we move the discussion to #1855 ?

[hmacdope]

Hey @BarclayII I have been working on getting a build going on here conda-forge/staged-recipes#22691 an extension of the now stale conda-forge/staged-recipes#18620. The build is still quite WIP.

You can see the CMake changes I have been making to get this to work on my branch https://github.com/hmacdope/dgl/tree/conda-forge3 . I would appreciate any help / feedback on the recipe as I go.

I have been building the 0.8.2 version due to some CUB/Thrust versioning issues documented in #1855 but I am guessing we will probably either vendor CUB/Thrust or somehow link against cuda-cccl-impl from CF (https://github.com/conda-forge/cuda-cccl-impl-feedstock, see discussion here conda-forge/thrust-feedstock#19). We may also need to do some work to push some of the last few dependencies in the fantastic table by @hadim onto CF.

[github-actions]

This issue has been automatically marked as stale due to lack of activity. It will be closed if no further activity occurs. Thank you

[github-actions]

This issue has been automatically marked as stale due to lack of activity. It will be closed if no further activity occurs. Thank you

[hmacdope]

Hi all, particularly (@jermainewang @BarclayII @bgawrych @hadim @mikemhenry)

There is now a conda-forge package for DGL 1.1.0! 😄 🎉 Try it out with:

conda install -c conda-forge dgl

Thanks so much to @mikemhenry for doing most of the heavy lifting.

There were significant CMake patches required to satisfy the conda setup as well as two additional smaller patches. I have attached them here.

Going forward, should you be amenable, I think we should upstream as many of these changes as possible.
so that a complicated patch setup is not required for conda releases and such that there is only one source of truth (the wonderful DGL package!).

1. conda-build.patch -> re-works CMake to allow building from deps in a conda env and changes libxsmm to header only mode
2. fix_clang_errors.patch -> small change to deal with some clang casting strictness, backport from current master
3. fix_libdgl_sparse_pytorch_location_logic.patch -> Makes the finding dgl_sparse libs more robust.

My branch hmacdope/conda-forge-1.1.0-patches has a clean copy of the code with patches applied for records sake so that you can bring this in as an upstream back-branch if you want.

My next steps were going to be:

• Generate equivalent CMake patches for 1.1.1 so we can work on a 1.1.1 build in the feedstock.
• ~~PR against main with patches 2 and 3, should be simple and easy. ~~
• PR against main for an equivalent of patch 1, which is more involved.
• Remove the requirement for patches from the feedstock.

Let me know if you have feedback on this course of action or would like to approach things differently.

Cheers,

Hugo

[mikemhenry]

Just as a note, this patch https://github.com/conda-forge/dgl-feedstock/blob/main/recipe/fix_libdgl_sparse_pytorch_location_logic.patch was kind of a hack to fix some weird issues I was seeing, with the way conda-forge does the builds, we will only ever build one version of PyTorch against dlg_sparse, so it is safe to just grab the one we built. I don't think we should upstream that one and we should make dgl.sparse it's own conda-forge package if it is meant to be stand alone

[jermainewang]

Wow, this is indeed a great amount of work! Please accept my hats off! @hmacdope @mikemhenry .

@frozenbugs Please remember to highlight this in our next release and also follow up with this to see how to incorporate those changes to our cmake.

[hmacdope]

1.1.1 is live now! Again if you want to loop in the branch its hmacdope:conda-forge-1.1.1

[Atcold]

I tried to install it. It does not work.

$ conda install -c conda-forge dgl
Collecting package metadata (current_repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - dgl

Current channels:

  - https://conda.anaconda.org/conda-forge/osx-arm64
  - https://conda.anaconda.org/conda-forge/noarch
  - https://repo.anaconda.com/pkgs/main/osx-arm64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/osx-arm64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

[mikemhenry]

@Atcold It looks like you are on osx-arm64. It is on my to-do list but since conda-forge doesn't have a way to natively build on osx-arm64 it has to be cross compiled which I am not sure if the current CMake setup is configured to handle that.

If you want to give this a spin on a M1/M2 mac, you can do this:

CONDA_SUBDIR=osx-64 conda create --name dgl-conda-forge -c conda-forge
conda activate dgl-conda-forge
conda config --env --set subdir osx-64

That will create a conda env using apple's Rosetta 2 emulation for x86-64.

[Atcold]

Oh, I see! Thanks!

I followed these instructions and I got a working environment.
Is my installation using Rosetta 2 emulation as well?

[mikemhenry]

I believe installing the wheel from the repo like that does not use emulation since there are wheels listed like this dgl-1.1.2-cp39-cp39-macosx_11_0_arm64.whl which implies arm64 native builds.

[Atcold]

Oh, so this would be an even better solution, correct?
Unfortunately it Cannot find DGL C++ sparse library.

[mikemhenry]

Sorry to be responding on both threads 🙃

"better solution" is a relative term. If you want to play around with dgl, I think pulling from conda-forge with some emulation should be fine, and then you can source other packages from conda-forge and everything should be fine ABI wise.

Emulation will be slower than native, but I image that when performance really matters, you will be running something on a linux HPC or workstation that has a CUDA gpu or something.

[github-actions]

This issue has been automatically marked as stale due to lack of activity. It will be closed if no further activity occurs. Thank you

[mikemhenry]

1.1.2 is now on conda-forge, so we can close this issue.