Automatically load LAMMPS plugins if they are in folders listed in LAMMPS_PLUGIN_PATH environment variable

[akohlmey]

Summary

This PR adds automatic loading of LAMMPS plugins if they are located in a directory listed in the
LAMMPS_PLUGIN_PATH environment variable. This is only applied to files whose names end in plugin.so (on any operating system).

Author(s)

Axel Kohlmeyer, Temple U

Licensing

By submitting this pull request, I agree, that my contribution will be included in LAMMPS and redistributed under either the GNU General Public License version 2 (GPL v2) or the GNU Lesser General Public License version 2.1 (LGPL v2.1).

Backward Compatibility

N/A

Post Submission Checklist

• The feature or features in this pull request is complete
• Licensing information is complete
• Corresponding author information is complete
• The source code follows the LAMMPS formatting guidelines
• Suitable new documentation files and/or updates to the existing docs are included
• The added/updated documentation is integrated and tested with the documentation build system
• The feature has been verified to work with the conventional build system
• The feature has been verified to work with the CMake based build system

[akohlmey]

Demo:

$ ls plugins/
helloplugin.so  nve2plugin.so
$ ./lmp 
LAMMPS (17 Feb 2022)
plugin load helloplugin.so
Loading plugin: Hello world command v1.1 by Axel Kohlmeyer (akohlmey@gmail.com)
hello world
Hello, world!
plugin unload command hello
Unloading command style hello
hello world
ERROR: Unknown command: hello world (src/input.cpp:279)
Last command: hello world
$ env LAMMPS_PLUGIN_PATH=$PWD/plugins ./lmp 
LAMMPS (17 Feb 2022)
Loaded 2 plugins from /home/akohlmey/compile/lammps/build-test/plugins
hello world
Hello, world!
Total wall time: 0:00:07 

[akohlmey]

@junghans, @giacomofiorin, @cnegre, @gtribello, @ellio167, @tadmor, @yury-lysogorskiy
I have been thinking that this could be a good way to simplify maintenance and distribution for (partially) external projects and specifically modularize how LAMMPS is installed when creating binary packages.
Since there is no mechanism for dependencies between plugins, there are limitations (the implementation of plugin mechanism was meant to be simple so that it can be done easily and without(!) modification of the code in the LAMMPS package itself), but so far the implementation has been easier than expected and the behavior better than expected.

Because of their nature of being interfaces to libraries, the COLVARS, KIM, LATTE, ML-PACE, or PLUMED packages would be good targets for this approach, so this is why I am starting a discussion here (if you prefer to discuss in private/person, just email me). The key points are that building a plugin would only require one additional file (the plugin loader) and could essentially be placed anywhere to compile, it just needs access to the LAMMPS source folder. As a demo, I have tried this with the KIM package and it will be easy for me to do the equivalent to other packages.
kim-plugin.tar.gz

Specifically for binary packaging like RPMs for Fedora, DEB packages for Ubuntu, binary installer packages for Windows, conda packages, this would allow for creating add-on packages (e.g. lammps-colvars, lammps-kim, lammps-latte, lammps-pace, lammps-plumed) that can be installed separately but require the base lammps package (say lammps-core or lammps-common).
The benefit is that people that want LAMMPS without the extra functionality, don't have to download it and its dependencies with the core LAMMPS package itself, but rather only the plugin package would need to require the additional dependent package(s). At the same time, due to the automatic loading, LAMMPS will (in theory) behave exactly like it was compiled with the package included, once it finds and loads the plugin.

Since the plugin does not link to the LAMMPS library and compiling it requires the headers only, it could be compiled and installed with the external library directly and then will be "activated" only when LAMMPS is installed as well. There are plans to change how the LAMMPS sources are structured so that we will have a set of LAMMPS-API headers (e.g. for a lammps-devel or lammps-dev package) that would be sufficient to compile such a plugin and a full LAMMPS source distribution would not be needed anymore. In addition, we would place those headers specifically under LGPL license terms and thus improving the compatibility with libraries using other licenses.

Right now, the detection of plugins to auto-load depends on the presence of the environment variable LAMMPS_PLUGIN_PATH which lists one or more folders (like other "path" variables), but it would be easy to also have some canonical locations in the file system hierarchy added that would also be checked (similar to, say, the site-packages folders in Python), e.g. ${HOME}/.local/lib/lammps/plugins-20220217 and - for example - /usr/lib64/lammps/plugins-20220217 (or rather ${CMAKE_INSTALL_LIBDIR}/lammps/plugins-20220217) for the 17 February 2022 version.
Any advice on common conventions/rules for binary packaging that would determine which folders to add to the search path is highly welcome.

Please let me know what you think. Please also note that there are currently no plans to change the distribution/inclusion of the mentioned packages/libraries in LAMMPS. This is merely meant as an additional feature specifically aimed at packaging and providing an alternate route to distribute updates of your library/software without having to update LAMMPS itself.

Please also note that this can be used for debugging/testing. Dynamically loaded plugins will "overwrite" any corresponding compiled in commands/styles so it should also be possible to test updates without having to recompile LAMMPS.

@sjplimp I hope that as a person who is not so familiar with creating binary packages, I can recruit you to provide advice on what needs to be documented in the manual (for users and for developers).

[athomps]

I approve.

[giacomofiorin]

Hi @akohlmey, thanks for pointing out the new possibilities raised by the plugin mechanism. I only briefly saw the past GitHub conversations about it, but haven't tried it yet.

I looked at the documentation, and the following points are not 100% clear to me:

1. Is it possible to have a LAMMPS build with the library "X" statically linked in, while also allowing at runtime to load a more recent version of "X" from a plugin? You mention something like that in the comment, but this is a critical point.

2. If the above is possible, how would the different versions of "X" included in the LAMMPS executable vs. from the plugin be reconciled? The lammpsplugin_t struct does not have a member for the plugin's version.

3. Assuming that the above is possible, could the info printed about dynamically loaded plugins from $LAMMPS_PLUGIN_PATH be more detailed? At least for those plugins that override existing code. I am concerned about potential mishandlings of environment variables, due to people being unaware of how queue systems like Slurm propagate variables to jobs, or more simply, people just being sloppy with their shell init scripts.

In general, I think that a plugin loading mechanism is useful. But even in its absence, you and the LAMMPS team have issued stable releases with reasonable frequency. So at least for codes like Colvars that are fully integrated with the LAMMPS build system and don't have other dependencies, there isn't a major concern about getting functionality out to users.

Also, the concept of LAMMPS "packages" is already established in the users' minds, and the documentation is well organized, such that it is easier for users to find the correct information and credit both the LAMMPS team and the authors of the contributed libraries. For comparison, in a "monolithic" app like NAMD these tasks have been much more difficult (to be blunt). So I want to take the opportunity to thank you for the work done toward improving the overall experience of integrating another code with LAMMPS.

[akohlmey]

1. Is it possible to have a LAMMPS build with the library "X" statically linked in, while also allowing at runtime to load a more recent version of "X" from a plugin? You mention something like that in the comment, but this is a critical point.

Yes. It is not yet an "official" feature.

2. If the above is possible, how would the different versions of "X" included in the LAMMPS executable vs. from the plugin be reconciled?

Please keep in mind that styles are created from a map connecting a keyword with a function pointer to a factory function that will create an instance of the class connected with the keyword. The plugin file needs to provide a loader function and that function will call the registration callback provided by lammps with a data structure that has a pointer to the factory function included in the plugin and the keyword. The callback will now overwrite the entry in the map with the new factory function in the dynamically loaded object which will call out to code provided in the plugin.

The lammpsplugin_t struct does not have a member for the plugin's version.

This would be rather pointless since the map is not designed around plugins. Please remember, that in - for instance - VMD the statically loaded "plugins" are still handled as plugins and have their (static) internal registration function. We don't have that in LAMMPS, but also I deliberately tried to simplify the plugin interface and removed one level of indirection that VMD uses. Load means load. If the result crashes or eats your hard drive: tough luck 😉

3. Assuming that the above is possible, could the info printed about dynamically loaded plugins from $LAMMPS_PLUGIN_PATH be more detailed? At least for those plugins that override existing code. I am concerned about potential mishandlings of environment variables, due to people being unaware of how queue systems like Slurm propagate variables to jobs, or more simply, people just being sloppy with their shell init scripts.

Within the loader function you have access to the LAMMPS class pointer and can use it for additional output.
For the automatic loading I deliberately opted for a terse output, since the "normal" output would be rather verbose and if that would be printed every time LAMMPS is started or after every "clear" command, people would complain.
You will get the verbose output on loading and unloading when you use the plugin command directly.

For a batch/cluster environment, I would envision that either the variable would be set from an environment module or by the individual user. The first would be straightforward and there would be no user issues if the environment module is properly done, the second is a case of SEP. There are so many ways a users can shoot themselves in the foot, if somebody is willing to compile an add-on as a plugin and automatically load it, I see it as the user's responsibility to take care of consistent environment variable setting.

Let me summarize: the primary application I see for the automatic load of plugins is in "packaging", e.g. RPMs, DEBs, pip, conda, windows installer packages, environment module deployment on a cluster and so on. I see this as a chance to fight "dependency hell" as with a plugin, the LAMMPS binary itself would depend on fewer external libraries (that isn't so much an issue for colvars, but for some other packages like KIM, PLUMED, ML-HDNNP, ML-QUIP, ML-PACE, LATTE, ADIOS, NETCDF, SCAFACOS) and is easier to compile and package. For KIM and PLUMED for example, the LAMMPS specific code is very little.

The option to load a plugin over a compiled in package, I was mostly seeing as an opportunity for making life simpler for developers of a package since you won't have to recompile LAMMPS itself. This can be particularly annoying when compiling a "loaded" binary and you want to change compiler flags.

In general, I think that a plugin loading mechanism is useful. But even in its absence, you and the LAMMPS team have issued stable releases with reasonable frequency. So at least for codes like Colvars that are fully integrated with the LAMMPS build system and don't have other dependencies, there isn't a major concern about getting functionality out to users.

Also, the concept of LAMMPS "packages" is already established in the users' minds, and the documentation is well organized, such that it is easier for users to find the correct information and credit both the LAMMPS team and the authors of the contributed libraries.

That is why I started the discussion listing "packages" as options for distributing plugins and loading them automatically.