Commits on Sep 13, 2021

1. Engine: implement functionality to import completed CalcJobs …

   When people start using AiiDA they typically already have many
   calculation jobs completed without the use of AiiDA and they wish to
   import these somehow, such that they can be included in the provenance
   graph along with the future calculations they will run through AiiDA.
   
   This concept was originally implemented for the `PwCalculation` in the
   `aiida-quantumespresso` plugin and worked, but the approach required a
   separate `CalcJob` implementation for each existing `CalcJob` class that
   one might want to import.
   
   Here we implement a generic mechanism directly in `aiida-core` that will
   allow any `CalcJob` implementation to import already completed jobs. The
   calculation job is launched just as one would launch a normal one through
   AiiDA, except one additional input is passed: a `RemoteData` instance
   under the name `remote_folder` that contains the output files of the
   completed calculation. The naming is chosen on purpose to be the same as
   the `RemoteData` that is normally created by the engine during a normal
   calculation job run.
   
   When the engine detects this input, instead of going through the normal
   sequence of transport tasks, it simply performs the presubmit and then
   goes straight to the "retrieve" step. Here the engine will retrieve the
   files from the provided `RemoteData` as if they had just been produced
   during an actual run. In this way, the process is executed almost
   exactly in the same way as a normal run, except the job itself is not
   actually executed.

   

   sphuber committed Sep 13, 2021
   Configuration menu

   • View commit details
   • Copy full SHA for fee0902
   • Browse repository at this point

   Copy the full SHA

   fee0902 View commit details

   Browse the repository at this point in the history

2. Engine: add CalcJobImporter class and associated entry point group …

   The `CalcJobImporter` class is added, which defines a single abstract
   staticmethod `parse_remote_data`. The idea is that plugins can define an
   importer for a `CalcJob` implementation and implement this method. The
   method takes a `RemoteData` node that points to a path on the associated
   computer that contains the input and output files of a calculation that
   has been run outside of AiiDA, but by an executable that is normally run
   with this particular `CalcJob`.
   
   The `parse_remote_data` implementation should read the input files found
   in the remote data and parse their content into the input nodes that
   when used to launch the calculation job, would result in similar input
   files. These inputs, including the `RemoteData` as the `remote_folder`
   input, can then be used to run an instance of this particular `CalcJob`.
   The engine will recognize the `remote_folder` input, signalling an
   import job, and instead of running a normal job that creates the input
   files on the remote before submitting it to the scheduler, it passes
   straight to the retrieve step. This will retrieve the files from the
   `RemoteData` as if it would have been created by the job itself. If a
   parsers was defined in the inputs, the contents are parsed and the
   returned output nodes are attached.
   
   The `CalcJobImporter` can be loaded through its entry point name using
   the `CalcJobImporterFactory`, just like the entry points of all other
   entry point groups have their associated factory. As a shortcut, the
   `CalcJob` class, provides the `get_importer` class method which will
   attempt to load a `CalcJobImporter` class with the exact same entry
   point. Alternatively, the caller can specify the desired entry point
   name should it not correspond to that of the `CalcJob` class.
   
   To test the functionality, a `CalcJobImporter` is implemented for the
   `ArithmeticAddCalculation` class.

   

   sphuber committed Sep 13, 2021
   Configuration menu

   • View commit details
   • Copy full SHA for 8910074
   • Browse repository at this point

   Copy the full SHA

   8910074 View commit details

   Browse the repository at this point in the history