Changes to support PyPI packages

[peastman]

To be able to create PyPI packages, there are some issues we need to resolve which will probably involve changes to code or build scripts. This issue is to discuss possible ways of solving them.

CUDA

The biggest issue we need to consider is how to deal with CUDA. Different CUDA releases are not binary compatible with each other (or only binary compatible over a narrow range of releases), so at runtime OpenMM needs the libraries from the particular toolkit version it was built against. Each toolkit in turn requires particular driver versions, and users often can't control what driver version they have (for example on a cluster they don't administer). That means it's helpful to build OpenMM against multiple CUDA versions. We need a way for each user to get an OpenMM build and CUDA libraries that are compatible with each other and with their driver.

Conda handles this with two mechanisms: first by providing all the libraries in a cudatoolkit package, and second by allowing a package to have multiple builds that differ only in which version of that package they require. At install time, the conda client tries (not always successfully) to automatically select a version of cudatoolkit that is compatible with the user's driver, and then installs the corresponding versions of other packages.

For installing with pip, NVIDIA provides their own repository with all the redistributable libraries. You can install packages from it either by adding an extra URL to requirements.txt, or alternatively by first installing the nvidia-pyindex package. See https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#pip-wheels-installation-linux for more information.

On the other hand, pip does not support having multiple builds that differ only in what CUDA version they're built against. A given OpenMM version has to be built against a single CUDA version, which would presumably be the oldest (and therefore most compatible) one we support. And that means any other package in the same environment that also uses CUDA (e.g. PyTorch) also has to be built against the same version.

PyTorch works around this issue by providing their own repository with additional versions. For example, if you type pip install torch you get one that's built against CUDA 10.2. If you instead want a package built against CUDA 11.6 you type pip install torch --extra-index-url https://download.pytorch.org/whl/cu116. We could consider doing something similar, though it's probably more trouble than it's worth.

Another possibility discussed in #3196 is that a single OpenMM package could contain multiple versions of the plugins built against different CUDA versions. Within the plugins directory you would have libOpenMMCUDA10.2.so, libOpenMMCUDA11.0.so, etc. At runtime, the one matching the available CUDA libraries would be loaded, and all the others would fail to load since the libraries they depended on couldn't be found.

Plugins

Another issue we need to consider is handling of plugins installed by multiple packages. To ensure all dependencies between plugins are handled correctly, we need all of them to be loaded by a single call to Platform.loadPluginsFromDirectory(). With conda that happens automatically, since there's a single lib directory for each environment that's shared by all packages, so everything ends up in a single plugins folder. PyPI tries to keep things separate, with each package having its own private lib directory. For a package that installs new plugins (e.g. OpenMM-Torch), I believe we could make setup.py copy its plugins into the main plugins folder. Alternatively, we could have it just create a symlink pointing to its own folder. For example, it would create the link openmm/lib/plugins/openmm-torch that would point to openmm-torch/lib/plugins. That would require changing the behavior of loadPluginsFromDirectory() to make it check subdirectories. Yet another option is to make it edit OpenMM's version.py to add its own plugins folder.

With any of those approaches, we need to figure out how to make sure that if you update the openmm package, it doesn't lose all the plugins installed by other packages.

Versioned libraries

libOpenMM is currently built as a versioned library. That means there's a library called (for example) libOpenMM.8.0.so, and a symlink called libOpenMM.so pointing to it. That doesn't work with PyPI packages. When it installs a package, symlinks get turned into real files. In his proof of concept (#3239), @tristanic worked around this by adding a CMake flag to tell it to create unversioned libraries instead. We also should consider just getting rid of the versioning. It was never really fully implemented. libOpenMM is versioned, but other libraries like libOpenMMAmoeba aren't. And none of the plugins is versioned, and trying to version them wouldn't work at all, since if you had multiple versions installed it would load all of them instead of just picking one. And honestly, I don't think there was ever really a good reason to version them in the first place. See the discussion at #2281. The only reason was to allow distributing OpenMM as a Debian package, which we don't support and have no plans to support.

[tristanic]

An alternative solution to the plugins problem would be for OpenMM to create a file in the user space (~/.local/share/OpenMM/{version} in Linux, ~/Library/Application Support/OpenMM/{version} in MacOS, %localappdata%/OpenMM/{version} in Windows), and have OpenMM provide a method like registerPluginDirectory() to add it to a list stored there. Then just loop through all those directories when loading plugins at runtime.

…

On Fri, 30 Sep 2022 at 20:45, Peter Eastman ***@***.***> wrote: To be able to create PyPI packages, there are some issues we need to resolve which will probably involve changes to code or build scripts. This issue is to discuss possible ways of solving them. CUDA The biggest issue we need to consider is how to deal with CUDA. Different CUDA releases are not binary compatible with each other (or only binary compatible over a narrow range of releases), so at runtime OpenMM needs the libraries from the particular toolkit version it was built against. Each toolkit in turn requires particular driver versions, and users often can't control what driver version they have (for example on a cluster they don't administer). That means it's helpful to build OpenMM against multiple CUDA versions. We need a way for each user to get an OpenMM build and CUDA libraries that are compatible with each other and with their driver. Conda handles this with two mechanisms: first by providing all the libraries in a cudatoolkit package, and second by allowing a package to have multiple builds that differ only in which version of that package they require. At install time, the conda client tries (not always successfully) to automatically select a version of cudatoolkit that is compatible with the user's driver, and then installs the corresponding versions of other packages. For installing with pip, NVIDIA provides their own repository with all the redistributable libraries. You can install packages from it either by adding an extra URL to requirements.txt, or alternatively by first installing the nvidia-pyindex package. See https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#pip-wheels-installation-linux for more information. On the other hand, pip does not support having multiple builds that differ only in what CUDA version they're built against. A given OpenMM version has to be built against a single CUDA version, which would presumably be the oldest (and therefore most compatible) one we support. And that means any other package in the same environment that also uses CUDA (e.g. PyTorch) also has to be built against the same version. PyTorch works around this issue by providing their own repository with additional versions. For example, if you type pip install torch you get one that's built against CUDA 10.2. If you instead want a package built against CUDA 11.6 you type pip install torch --extra-index-url https://download.pytorch.org/whl/cu116. We could consider doing something similar, though it's probably more trouble than it's worth. Another possibility discussed in #3196 <#3196> is that a single OpenMM package could contain multiple versions of the plugins built against different CUDA versions. Within the plugins directory you would have libOpenMMCUDA10.2.so, libOpenMMCUDA11.0.so, etc. At runtime, the one matching the available CUDA libraries would be loaded, and all the others would fail to load since the libraries they depended on couldn't be found. Plugins Another issue we need to consider is handling of plugins installed by multiple packages. To ensure all dependencies between plugins are handled correctly, we need all of them to be loaded by a single call to Platform.loadPluginsFromDirectory(). With conda that happens automatically, since there's a single lib directory for each environment that's shared by all packages, so everything ends up in a single plugins folder. PyPI tries to keep things separate, with each package having its own private lib directory. For a package that installs new plugins (e.g. OpenMM-Torch), I believe we could make setup.py copy its plugins into the main plugins folder. Alternatively, we could have it just create a symlink pointing to its own folder. For example, it would create the link openmm/lib/plugins/openmm-torch that would point to openmm-torch/lib/plugins. That would require changing the behavior of loadPluginsFromDirectory() to make it check subdirectories. Yet another option is to make it edit OpenMM's version.py to add its own plugins folder. With any of those approaches, we need to figure out how to make sure that if you update the openmm package, it doesn't lose all the plugins installed by other packages. Versioned libraries libOpenMM is currently built as a versioned library. That means there's a library called (for example) libOpenMM.8.0.so, and a symlink called libOpenMM.so pointing to it. That doesn't work with PyPI packages. When it installs a package, symlinks get turned into real files. In his proof of concept (#3239 <#3239>), @tristanic <https://github.com/tristanic> worked around this by adding a CMake flag to tell it to create unversioned libraries instead. We also should consider just getting rid of the versioning. It was never really fully implemented. libOpenMM is versioned, but other libraries like libOpenMMAmoeba aren't. And none of the plugins is versioned, and trying to version them wouldn't work at all, since if you had multiple versions installed it would load all of them instead of just picking one. And honestly, I don't think there was ever really a good reason to version them in the first place. See the discussion at #2281 <#2281>. The only reason was to allow distributing OpenMM as a Debian package, which we don't support and have no plans to support. — Reply to this email directly, view it on GitHub <#3796>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFM54YEWNDJJZJULTUG6MYLWA47NFANCNFSM6AAAAAAQ2BZTUM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[peastman]

That would make the list of directories global for the user, not specific to a Python environment. Perhaps we could do something similar within the environment?

[tristanic]

Sure. You’d probably still want both options - if they do a straight “pip install” then write to a file in the OpenMM directory; if they do “pip install —user”, use the approach I suggested.

…

On Fri, 30 Sep 2022 at 22:29, Peter Eastman ***@***.***> wrote: That would make the list of directories global for the user, not specific to a Python environment. Perhaps we could do something similar within the environment? — Reply to this email directly, view it on GitHub <#3796 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFM54YH4KNXN7CS25W3TBJTWA5LTDANCNFSM6AAAAAAQ2BZTUM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[peastman]

How about this logic?

• Check for the environment variable CONDA_PREFIX. If you're in a conda environment, it will exist and point to the environment root. Put the data in ${CONDA_PREFIX}/share/openmm.
• Next check for VIRTUAL_ENV. That's the same for venv. If it exists, put the data in ${VIRTUAL_ENV}/share/openmm.
• If neither of those exists, use the OS specific path suggested above.

The check would happen both at install time (deciding where to store the information) and at runtime (deciding where to load from).

A variation on that would be to actually stick the plugins directory in that location. For example, in a conda environment it would go in ${CONDA_PREFIX}/lib/openmm/plugins and contain symlinks to the package-specific plugins folders, as discussed above.

[tristanic]

I admit I mis-read your previous comment - got it in my head that you were talking about global (all users) vs. single-user installation, rather than multi-environment. You'll probably need to consider both of those cases as well - if a package is installed with pip install it goes into the global Python space for all users; pip install --user goes to the user-specific space. You can find the designated directories for a given Python environment with:

import site
site.getsitepackages() # global site-packages directory
site.getusersitepackages() # user-specific site-packages directory

If you create a file (or subdirectory) in each of those locations, it should be entirely environment-specific (I think) and survive across OpenMM reinstallations.

[peastman]

Is there a way for setup.py to determine whether it's being installed globally or for one user?

[peastman]

Let's enumerate all the cases we want to handle. There seem to be a lot of them.

• Installing the pip package:
  • In the base environment of a copy of miniconda (or anaconda) inside the user's directory.
  • In a different environment of a copy of miniconda inside the user's directory.
  • In an environment created with venv that's inside the user's directory.
  • In a copy of CPython (downloaded from python.org) in the user's directory.
  • In a python that's installed system-wide for all users, putting it in the global site-packages directory.
  • In a python that's installed system-wide, but with the --user flag so it gets installed for one user.
• Installing the conda package:
  • In the base environment of a copy of miniconda inside the user's directory.
  • In a different environment of a copy of miniconda inside the user's directory.
• Building from source.

[mattwthompson]

pip install into global space is really rife for headaches, but I guess there isn't much you do can when users choose not to install tools into isolated environments.

Installing the conda package:
In the base environment of a copy of miniconda inside the user's directory.
In a different environment of a copy of miniconda inside the user's directory.

Are these use cases any different? Stuff should go somewhere inside of $CONDA_PREFIX whether it's being installed into an isolated environment or the base environment (yuck)

[peastman]

pip install into global space is really rife for headaches

Agreed! The situation where I've seen this come up is when the administrators on a system want to preinstall a bunch of scientific software for their users. We do have the option of saying we aren't going to support that use case.

Are these use cases any different?

In principle yes, but it's likely we can handle both with the same logic. I was just trying to be thorough to be sure we consider (and test!) all the different possible situations.

[tristanic]

Yep - pre-installation of global packages by sysadmins was the use case I was thinking of. For something like OpenMM I can imagine that ability would be desirable for lots of research groups.

[tristanic]

I heard from the ChimeraX team that their Linux builds are now done in gcc-toolset-10 (available in RedHat 8, CentOS 8, Rocky 8). According to the documentation at https://github.com/pypa/manylinux that's compliant with the manylinux2014 tag. The Singularity definition for their build environment is https://github.com/RBVI/ChimeraX/blob/develop/prereqs/linux_buildenv/rhel-8.def... it would make my life much, much easier if OpenMM were to also use gcc-toolset-10 for its Linux builds.

[peastman]

One way or another we definitely want it to be compatible with manylinux2014. Is it ok if we build with clang rather than gcc? In my experience it's a lot less buggy. We've had a number of problems with compiler bugs in the gcc builds on conda-forge.

[tristanic]

It should be OK - as long as you're building against the same standard template library implementation, then binaries compiled with clang are supposed to be compatible with those compiled with gcc. So you'd still want to install and enable the gcc-toolset-10, then it looks like clang should be able to find and use it (as long as you use a version built after this Aug 2021 pull request: https://reviews.llvm.org/D108908).

[peastman]

How about the CUDA support questions? Does anyone have thoughts on those? We should consider each option from a few different perspectives: how difficult it will be for us to implement and maintain, how easy it will be for users, and how it will impact the developers of downstream packages, especially ones that provide new plugins. (@egallicc I'd value your perspective on that last part.)

[tristanic]

I suppose one (probably unpalatable) other option would be to do something akin to Nvidia themselves... provide support for a single recent CUDA version on the main PyPI server, but maintain a more comprehensive set of builds for older CUDA releases on a separate pip server.

[peastman]

Right, that's what PyTorch does. It would probably require a lot of work for us to maintain. It might score well on the other criteria though.

With NVIDIA, the packages are only available from their server. They also publish packages with the same names on the regular PyPI, but they're stub packages whose only function is to warn you that you've installed the wrong thing. That's also an option, I suppose.

[tristanic]

Yes - I can’t remember exactly how, but the list of command-line flags is accessible programmatically.

…

On Mon, 3 Oct 2022 at 17:19, Peter Eastman ***@***.***> wrote: Is there a way for setup.py to determine whether it's being installed globally or for one user? — Reply to this email directly, view it on GitHub <#3796 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFM54YGGBNCMNJHJUPYC6L3WBMBRNANCNFSM6AAAAAAQ2BZTUM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jinmingyi1998]

Hi guys, Nvidia support install cuda runtime, cuda cudnn, cuda cublas etc. by pip install nvidia-cudaxxx-cu11 on pypi after cuda 11.7. Also PyTorch 1.13 support install cuda from pypi.

[egallicc]

How about the CUDA support questions? [...>] especially ones that provide new plugins. (@egallicc I'd value your perspective on that last part.)

Sorry, I missed this. I think it would be difficult for external plugin maintainers to keep up with the same installation complexities as the core OpenMM package. Conda-forge multiple builds feature helped us with the CUDA dependencies. We would need assistance or some kind of workflow to meet more involved installation setups. However, there are potentially many external plugins out there and it might be difficult to develop and maintain something that works for all of them.

In the long run, we could consider a pool of OpenMM-endorsed external plugins that are tested, built and distributed with OpenMM. This would address compatibility issues as well. For example, we faced bugs caused by changes to internal OpenMM APIs that were difficult to track down. A synchronized build and testing system would have probably flagged such occurrences.

[peastman]

Thanks! Are those comments meant in reference to building PyPI packages in general, or to dealing with CUDA specifically? If the latter, do you have suggestions for the best way of handling multiple CUDA versions?

[jinmingyi1998]

Thanks! Are those comments meant in reference to building PyPI packages in general, or to dealing with CUDA specifically? If the latter, do you have suggestions for the best way of handling multiple CUDA versions?

In my opinion, There is no best way to handle cuda version on user's computer. As a user, my environment use cuda cudnn docker, and i install openmm in conda env with --do-deps to avoid install cuda toolkit again.

I suggest: Openmm may dynamic link cuda libs, and dont have to concern about which version. Cuda version & drivers are users' responsibility.

Anyway, even if openmm help user to install cuda environment, there still be many problems like:

• cuda11 software <----> cuda10 GPU hardware conflict.
• latest cuda11 <----> old drivers conflict
• cuda11 <----> cuda 10 driver conflict
• duplicate cuda environment cause No Space Left

So, I sugget Openmm Just be yourself, dont think too much.

For people who dont familiar with CUDA, there may be a "Quick Start Wiki" to help them install CUDA environment

[peastman]

Cuda version & drivers are users' responsibility.

The problem is that we have to compile against a CUDA version that's compatible with what the user has. If we compile against CUDA 10, it won't work with CUDA 11 and vice versa. And we need to support a range of CUDA versions, because users may have limitations. Too old and it won't support their hardware. Too new and it won't support their driver (which they may not be able to change).

[jinmingyi1998]

The problem is that we have to compile against a CUDA version that's compatible with what the user has.

For example, there are 2 released versions openmm distribution: openmm=xxx+cuda10 openmm=xxx+cuda11, which compiled and dynamic lint to, for example, libcudart.so.10 / libcudart.so.11.

And user decide which version they should install.

User can still install cudatoolkit with conda. Just assure cuda libs can be linked.

[peastman]

PyPI doesn't support having multiple versions that differ only in what CUDA version they're compiled against.

[jinmingyi1998]

PyPI doesn't support having multiple versions that differ only in what CUDA version they're compiled against.

There still one way: release with different name like openmm-python-cpu/ openmm-python-cu10 / openmm-python-cu11

[mattwthompson]

Releasing a different package for each version of CUDA won't scale well. That's the sort of thing that build variants are for - although even that's a stretch. It doesn't sound like that's supported, or at least easy to manage.

[jinmingyi1998]

yes, that sounds not very cool. so i suggest that support pypi install begin from cuda11.7 (because nvidia support cuda pypi install from 11.7) .

As for earlier versions, don't change,just like before .

my key point is to start a new way after cuda11.7, and don't think too much about compatibility on earlier cuda

[peastman]

There still one way: release with different name like openmm-python-cpu/ openmm-python-cu10 / openmm-python-cu11

That would break dependency management. If another Python package depended on OpenMM, it would have to specify a particular one of those packages to depend on and wouldn't work with the others.

[jinmingyi1998]

it would have to specify a particular one of those packages to depend on and wouldn't work with the others.

start a new way after cuda11.7, and don't think too much about compatibility on earlier cuda.

not named with different names. just named python-openmm and requires nvidia-cuda>=11.7.

[wenyan4work]

may I ask what is the current plan for the pypi package of openmm? Also, are there any plans to distribute other openmm projects on pypi, such as nnpops and openmm-ml?

[peastman]

Once we get 8.0 out, this will be a high priority. The goal is for the full stack of package to be available on PyPI.

[mattwthompson]

#3131 appears relevant to this discussion - maybe not deprecate it after all

[peastman]

That package isn't really OpenMM.

[mattwthompson]

Right, but it's name-squatting on openmm

[jchodera]

I've been talking to @tristanic recently, who contributed #3239 to build PyPI packages for OpenMM. I'm wondering if he might be able to help with a renewed effort here.

[tristanic]

Yes - I'm afraid things have gone a little more slowly than I'd hoped, but this is still in the forefront of my mind. I was talking to a member of our software engineering team about it a few days back, and there are plans to help out - hopefully sooner rather than later.

[isuruf]

Here are the changes that I'm using for the wheels: https://github.com/isuruf/openmm-wheels/blob/main/recipe/0001-wheels.patch

I can send a PR if you like (modulo the openmm_library_path change)

[peastman]

Are those changes specific to building pip packages, or are they also meant to be used when building conda packages or building from source? If the former, it probably makes more sense to keep them as a patch.