Initial draft proposal, version 0.0.2
Version 0.0.1 was supposed to start a discussion – but did not. Looking back, it was rather underdeveloped, and we should probably continue to ignore it. The idea behind this version is to start agreeing on at least the overall structure of the model we want to build. The key point is about the minimum number of objects and crosslinks between them that we need in order to accommodate our data. Class names and details can change, classes can be split and merged, and we may want to add more complex data structures (like unit cells or reflection shell data) but I think we shall need at least this much structure.
For a relevant use case, I am thinking about a workflow-driven multi-sweep crystallography experiment. For such experiments it is crucial that data input and processing result can be given in terms of a set of selected sweeps. Organisation in terms of single sweeps is obviously inadequate for the purpose, and organisation in terms of Samples not only precludes straightforward representation of multi-sample processing results, but also lacks a way of distinguishing which of the multiple sweeps, mesh scans, or experiments performed on a sample actually are (or should be) processed together.
We should remember that the model is meant to specify an interface between programs and LIMS systems. People are free to use any internal organisation they find practical, as long as they are able to read and re-export data according to the model.
Abstract classes defining the model structure and the links between them. For those unused to UML note the cardinalities: ‘1’ (mandatory), ‘0..1’ (optional), and ‘*’ (0 to many)
- These are abstract classes - actual classes will be subtypes. The Job class is used both for describing actual experiments and for processing runs (calculations), with Input and Result collecting the input resp. output for one Job. Everything we think of as data are subtypes of the Data class, which might be Sweep, Mesh scan, Line scan, Model, Map, MTZ file etc. ... The InputData class serves only to hold parameters that describe how a particular Data object fits in as input to a particular Job. Note that only the Data class will have links to files.
- The model is intended as an API, i.e. for describing messages, so in practice each message would contain copies of the relevant objects as necessary. The need for proper objects and crosslinks is more urgent for LIMS and storage, where you do not want a lot of maybe identical data structures cluttering up your databases. I would propose that each of the six abstract classes shown in the diagram should have an UUID attribute, so it is possible to track the history, origin and identity of the information.
- All classes should have a slot for program-specific name-spaced data.
- Any Job (experiment or processing) will have some parameters or results that refer to the entire Job rather than to any individual input or output file. Accordingly, each Job is connected to one Input and one Result object that can hold these parameters, as well as grouping multiple data sets. Note that Results are only connected to one Job, and that the same Input in practice will rarely be used more than once.
- Each Data object is produced by a single Job (of whatever type), and so can hold parameters that are only defined relative to that Job, like the distinction between characterisation sweeps and acquisition sweeps, or overall maps and early-minus-late maps.
- Data objects can serve as input for many different Jobs, so we need the InputData class to sit between the Data and Input classes and specify parameters that describe how the data fit into this particular Job. There may not be that many attributes here, but we might need e.g. a role (characterisation versus acquisition, input versus reference mtz files, …), information on included/excluded image ranges, possibly a weight, or program-specific parameters. Note that InputData are specific to a given Input object, and so the link between the two is mandatory.
- We should limit the number of different subtypes by using the same classes in different contexts. E.g. a Sweep Data type could be used as part of a diffraction plan (partially populated and without a file Url), as part of a message to the acquisition queue (with a Url but no data file as yet), to store the data collection result, and as part of the input for a processing program (linked to an InputData object). InputData objects could similarly be used as part of a processing plan.
- As currently drafted, the model assumes that each Job (experimental or processing) is connected to (at most) one single Sample. For single-crystal experiments, the sample would be equivalent to the crystal. This could be extended to multi-crystal experiments later, as long as the composition, history, expected crystal form etc. is the same for all the crystals in the same well / chip / vial. Where it is appropriate, we can link a Sample also to Result objects or to processing Input. Processing jobs are not in principle limited to using data from a single sample, but there is no explicit support for processing multi-sample data in the model, except by following inter-object links back to the Sample record. The alternative of connecting the Sample record to the Data object would give full support for multi-sample experiments and processing but would severely complicate the more common case of doing an experiment or processing on a single sample and wanting to find the relevant Sample record.
The ICAT model objects do not fit exactly with what we need to do here, so storing these data in ICAT would require a certain amount of crafting. One problem is that ICAT links the sample to the Dataset rather than the DataCollection. A more basic problem is that ICAT does not have crosslinks between Datasets but only between Datafiles, and that the ICAT Datafile strictly describes a file object, with location, checksum, etc. Considering that our Data object can correspond to a file, to a set of mini-cbf files, to only part of the data inside an HDF5 file, or to no file at all, this does not suggest an obvious mapping. The diagram below shows one possible way that the model presented might be stored in ICAT.
Possible ICAT implementation of the draft model. InputData and Data are handled by a combination of a Dataset and a Datafile with a 1:1 link between them. To reproduce the draft model would require a hack that somehow connected each DataCollection with a single, optional Sample.