Splitting plumed into multiple libraries

[GiovanniBussi]

I would like to change a bit the install procedure for the distributed binaries (currently: MacPorts and conda) splitting plumed into multiple libraries. Separate libraries would be independently installable. I currently see three cases where this would be an advantage.

Contributed code. I think we should continue to keep contributed code separate from developers code. However, I see also the advantage of having binaries available for contributed code. We could have something like conda install plumed to install developers code and conda install plumed-contrib to install a separate library containing only the contributed modules. Extra shared libraries could be loaded automatically and should only contain the additional code (i.e. plumed-contrib would NOT contain the core code). It also would be possible to have separate libraries for each module (e.g. conda install plumed-ves) though I am not completely sure it is worth.

Code with complex dependencies. If parts of the code depend on a difficult-to-install library they might be shipped separately. One case is for instance boost library. Notice that whereas conda provides boost libraries, one might get conflicts if he tries to install plumed with boost libraries together with another package that needs a different version of boost libraries. This is a problem for C++ libraries since they typically change their ABI at every single release, so that keeping a version compatible with multiple boost libraries would be a nightmare. Notice that boost libraries are now only used in a contributed module, so that this falls in the case above. Another possible case would be the mklib script that currently does not work on conda since it requires installing compilers in the run environment. We could have a plumed-mklib package that does so and allows using LOAD with a cpp file.

Code depending on libraries with multiple versions. This is the case for MPI. There are indeed at least two flavors for MPI (mpich and openmpi), that are not ABI compatible and thus would require a complete recompilation of the PLUMED code. I have currently a prototype to allow installing a separate plumed-mpi package, that only contains wrappers to MPI functions and that would need to be installed for the specific MPI version, whereas plumed package would be MPI neutral. This specific implementation would also allow to automatically use in PLUMED the same MPI version used in the MD host code. On a cluster one might use the conda binary and only compile the MPI extension. Notice that this is in principle also the case for lapack/blas, however I think this is not so relevant since any version of those libraries is fine.

I am already working from time to time on these issues and I think at some point I will be able to open specific pull requests. Ideally everything would be done before 2.6 beta release. Meanwhile I am happy to take feedbacks.

Giovanni

[invemichele]

I think this is a very good idea.
The conda install is very convenient, and nowadays many people look for it and expect it to work just as well as the standard installation. Adding support for MPI and contributed modules (I'm thinking of one in particular 😉) would be great!

I don't have much experience with the topic, but please let me know if there is something I can do to help with this.

[GiovanniBussi]

@invemichele thanks! I actually worked a bit on both things:

• separate packages for MPI implementation
• separate packages for contributed modules.

The two topics are virtually independent. In both cases, the idea is to install an additional library side-by-side with the plumed library (e.g. /usr/local/libplumedKernel.so and /usr/local/libplumedExtension-opes.so) and make sure that as soon as you create a plumed object all the installed extensions are loaded (with dlopen). Allowing to choose which libraries are loaded in the input file (as with LOAD) is slightly more complicated and might be not necessary. In addition, it would make it difficult to do other stuff, such as listing all available actions, generating documentation, have vim syntax, etc.

I just pushed a branch with the work I have done so far for contributed modules. I was testing it with a single additional library (called /usr/local/libplumedExtension-contrib.so) containing all the contributed modules (and depending on boost), and if I remember correctly it was working. The last commit (5487b77) refers to an attempt to create multiple packages (one per contributed module), that should probably be the right thing to do.

What is missing is:

• test if it works
• evaluate if it's the correct thing to do or if there is a simpler path
• hunt for problems (things to check are: vim syntax and python input builder, manual build, check if header files are installed correctly, find other possible problems).
• automatize the construction of the meta.yml file so as to include multiple contributed modules.

We might also decide to be happy with a single contrib extension (instead of one per contributed module). However, I am afraid that with a growing number of contributed modules it is worth to directly jump to something more flexible.

I think I don't have time to make progress on this in the nearest future, but I am happy if someone else wants to try :-)

Related to the MPI thing, that is kind of more complicated, there's certainly some extra work that I need to do in first person, so it might take some more time before I can share something.