PluginsDevelopment

Plugins: Developer instructions

Before proceeding, please read the Plugins page about how one can enable plugins (this is fast), and read the customphysics.cc example carefully (this takes a bit longer). The customphysics.cc example is important, since it contains all the elements needed to extend and modify the neutron scattering physics produced by NCrystal, including custom Scatter classes and factories. However, it does so with everything hardcoded within a single C++ file, and does not answer the question of how one can enable the physics in another context (e.g. when running McStas or Geant4) or how to share the customisations with other users (including the NCrystal developers). Hence the need for a dedicated plugin mechanism and workflow which will be described on this page.

As described on the Plugins page, NCrystal plugins can either be compiled separately from NCrystal itself, loaded dynamically at runtime, or they can be compiled directly into the main NCrystal library. In order for all of this to work easily, there is a bit of boilerplate CMake, C++ and C-code which has to be written in a very specific manner. It is additionally likely that the required boiler-plate code could change slightly in the future. For that reason, it is best to develop plugins by using the ncplugin-template repository as a starting point, and preferably by using the GitHub fork mechanism. That way, NCrystal developers can easily keep track of available plugins and provide fixes as pull-requests to the plugin developers, in case future changes to core NCrystal code would otherwise make the plugins stop working. Of course, it is highly recommended to get in touch with the NCrystal developers before starting your plugin, so we are aware of the plugin you are developing and can provide feedback, etc.

It is assumed that all plugin developers have at least a basic familiarity with NCrystal, C++, and build systems. Additionally, it is an advantage to be comfortable with CMake, Python, and the command line on either Linux or OSX. If you think you lack the necessary skills, it might obviously be best if you have a colleague or supervisor nearby who can help you get started and provide guidance when/if you get stuck on something.

First step: Picking a plugin name, forking the ncplugin-template library, first commit.

Pick a name for your plugin! For consistency, try to pick a name which is unique, clear, and concise, and consists only of characters in the english alphabet (a-z, both upper and lower cased characters allowed). Prefer CamelCasing. Example names of hypothetical (not actual) plugins could be CohInelasESS (coherent-inelastic models developed at ESS), SANSJC (small angle neutron scattering models, developed by James Chadwick), LoadHKLFmtXYZ (.hkl file format support, developed at XYZ institute). The name of your plugin can of course be changed later, but it is easier to simply pick a proper name to begin with.

Once you picked a good name, you can go ahead and fork the ncplugin-template repository. The name of your forked repository should be ncplugin-<yourpluginname>, e.g. ncplugin-CohInelasESS or ncplugin-SANSJC.

The first thing you should do immediately after forking (and cloning) the plugin template is that you should edit two files in the repo: ncplugin_name.txt and README.md. The ncplugin_name.txt should contain your chosen plugin name, and the contents of the README.md file should be completely replaced with text like the following (of course replace CohInelasESS with the name of your plugin and feel free to already from the beginning add a bit more info than the minimum below about what the plugin will provide):

# The CohInelasESS plugin for NCrystal

This repository contains code for the NCrystal plugin named `CohInelasESS`.

It is currently under development and not ready for general usage.

You should immediately go ahead and commit and push the changes you made to ncplugin_name.txt and README.md to your repo, because until you do so the repo contents will be a bit confusing to everyone else. Later, when your plugin has actually seen real development, you should go back and edit the README.md file again, removing the It is currently under development... text, and instead adding a small presentation of what the plugin provides, the current state of the plugin development, and any other relevant information.

Next, you should change the description of the repo at GitHub (otherwise it will keep describing itself as an example repo). So navigate to your repo at GitHub and click the small settings ("gear") icon next to where it says "About" on the right side of the main page, and fill in something more appropriate such as "Repository containing the CohInelasESS plugin for NCrystal".

A final important thing to do at this initial stage is to verify whether or not the license described in the LICENSE file, namely the Apache 2.0 license, is indeed the license under which your plugin code is intended to be distributed. If at all possible, we strongly recommend that you stick with this license, because then it will be straight-forward to integrate all or some of your code into the core NCrystal repository at any point in the future. A good reason for wanting to change the license could for instance be that the plugin is planned to be developed using code copied from existing projects under different license terms. If that is the case, make sure you update the LICENSE file in your repository accordingly (usually you can of course still keep any newly written code under the Apache 2.0 license, just make sure it is clear which files are under a separate license).

Next steps : the actual plugin development

The plugin itself consists of C++ code in the <reporoot>/src/ and <reporoot>/include/ directories. File extensions should be .hh for header files and .cc for source files. Source (.cc) files should go in the <reporoot>/src/ directory, while header (.hh) files can reside in both of the two directories. However, only header files in the <reporoot>/include/ directory will be available for the testing/validation code in the <reporoot>/testcode/ sub directory (the layout of the testcode/ subdirectory will be discussed in the next section below).

No matter the purpose of the plugin being developed, all C++ files in the plugin should include the header file #include "NCrystal/NCPluginBoilerplate.hh" which among other things defines the namespace macro NCPluginNamespace (and alias NCP::), into which all code should be encapsulated in order to avoid clashes between different plugins (it also includes NCrystal/NCrystal.hh and sets up the namespace alias NC=NCrystal). The file <reporoot>/src/NCPluginBoilerplate.cc is special. It provides implementation of certain boilerplate functions (by defining NCPLUGIN_BOILERPLATE_CC at the top of the file), and it provides an implementation of the function void NCP::registerPlugin(), in which the plugin must execute whatever code is needed to activate itself (usually this means registering a factory with NCrystal).

To make it easy to get started, the template repository contains a silly example in which a plugin replaces the incoherent-elastic scattering physics of NCrystal with a dummy model. The model, implemented in the files NCPhysicsModel.hh and NCPhysicsModel.cc, can be activated by a custom section in NCMAT data. A factory implementation in NCPluginFactory.cc takes care of wrapping this physics model with the appropriate NCrystal interface class (a specialisation of the NCrystal::ScatterIsotropic class), and of combining the new physics model with existing physics models in NCrystal which are not replaced by the plugin. The factory is registered with NCrystal in the void NCP::registerPlugin() function.

In principle, that is all it takes: Modify NCPluginFactory.cc code to change the logic of how to combine your new physics model with existing models in NCrystal, and modify NCPhysicsModel.hh/.cc to implement your model instead of the dummy one. Of course, do not let the NCPhysicsModel.hh/.cc become huge and unmaintainable - if your model is complicated, you should probably break it down and add new helper classes and utilities in additional files in <reporoot>/src/ and <reporoot>/include/.

If you decide you wish to provide any NCMAT files to users of your plugin, you can create the directory <reporoot>/data and put them there. To avoid clashes with files from different plugins, their name should start with ncplugin-<yourpluginname>_. Thus, if you have an .ncmat file describing a supermaterial and your plugin name is SANSJC, your file might be named ncplugin-SANSJC_supermaterial.ncmat. It might be worthwhile to note that the nctool command (formerly known as ncrystal_inspectfile) has --plugins, --browse and --extract options, which might come in handy when debugging availability of a plugin and its files.

In addition to these files which are directly needed for the plugin itself, development of a new physics model is in practice likely to additionally involve a lot of scripts and code used to debug, benchmark, test, and validate the performance. Possible infrastructure for doing so is provided and discussed in the next section.

The testcode/ sub directory.

The <reporoot>/testcode/ directory contains example infrastructure which can be used to facilitate debugging, benchmarking, validation, and testing of the new plugin. In principle the files in this sub directory are not part of the plugin itself, and developers are free to reorganise the contents of the testcode/ subdirectory (or even remove it completely, but then the add_subdirectory(testcode) line should also be removed from the <reporoot>/CMakeLists.txt file).

The testcode/ directory contains a CMake sub-project which adds:

• A compiled C++ library (in the testcode/src and testcode/include directories) with additional common testing utilities. It also exports a few dedicated symbols in the NCForPython.cc file, which will be used to build Python hooks (see next item).
• A python module (in the testcode/python subdirecotry) which contain common testing utilities. Specifically it also provides a python class MyPhysicsModel which provides access to the physics model developed in the plugin. It does so by accessing the C-functions defined in the NCForPython.cc file via the python ctypes module (to keep the bindings light-weight, feel free to use e.g. pybind11 instead for this connection if you wish and know how to).
• C++ applications in testcode/app_xxx directories. Each directory must contain at least a main.cc file, and after build+install, the command will be available to run via the name ncplugin_xxx.
• Python scripts in testcode/scripts. A script named yyy will be available to run via the name ncplugin_yyy after build+install.
• Data files (e.g. .ncmat files) in the testcode/data sub directory. After build+install, these are accessible to NCrystal by prefixing ncplugin/ to their names (e.g. load them via NCrystal in python: info = NCrystal.createInfo("ncplugin/custom.ncmat")).

The testcode/utils directory contains some convenience scripts which can be used to quickly build+install both plugin and testing code while developing. People wishing to deal with NCrystal installation and CMake invocations themselves can simply ignore those, otherwise read the next section.

testcode/utils convenience scripts.

First of all the testcode/utils/installncrystal.x script can be used to fetch and build NCrystal itself. Most users should simply invoke it with no arguments, which will try to install the latest NCrystal code from GitHub:

$> ./testcode/utils/installncrystal.x

To uninstall at any point, simply remove the testcode/utils/cache directory (which is obviously in .gitignore). To install other versions of NCrystal, consult the instructions via:

$> ./testcode/utils/installncrystal.x --help

Now, having installed NCrystal, you want to get started working. So source the bootstrap.sh script like this (and do this again whenever you open a new terminal and want to start working on your plugin):

$> . ./testcode/utils/bootstrap.sh

As described, this gives you a command ncpluginbuild. Simply run it to (re)build your plugin and testing code:

$> ncpluginbuild

If all goes well, you can now start running your scripts and applications (ncpluginbuild should have printed a list for you of which commands you now have available):

Whenever you modify C++ code or add/remove python or data files, you must run the ncpluginbuild command once again. Python and data files are installed via symlinks, so you do not need to rerun ncpluginbuild if all you did was to edit such files.

You can read a bit more about the ncpluginbuild command by typing:

$> ncpluginbuild --help

Prerequisites

It is beyond the scope of the NCrystal developers to provide specific instructions for how to prepare your system. It should include a self-consistent setup of:

• git
• CMake (at least version 3.15.3 for plugin development)
• C++/C compiler (preferably modern GCC or Clang).
• Python3 (at least version 3.6). Should probably include numpy+matplotlib modules (if nothing else, try e.g. "python3 -mpip install numpy").
• A bash shell in /bin/bash

On most newer Linux distributions, the above requirements are easily satisfied by installing appropriate system packages (CentOS7 is a bit dated, so usage here will require tricks such as using software collections, etc.). On OSX, HomeBrew should be able to provide an appropriate environment.