Initial draft proposal, version 0.0.3
This comes before we have had time to discuss version 0.0.2. The changes are not that big, so they could be discussed together. Since version 0.0.2 Kate Smith and May Sharpe at the SLS have indicated a new use case that would require some changes to the model. The explanations from version 0.0.2 remain relevant, so you should read that first – I have not repeated them here. Furthermore I have come up with what I think is a better set of class names for version 0.0.3.
We are not going to incorporate SSX for now but we will eventually, and anyway this case points to a more general issue.
In SSX (or XFEL) an experiment would amount to measuring an entire chip, with thousands of tiny crystals yielding a single image each. The first processing step could then be a matter of combining the images from several experiments (chips), and selecting a set of images from across the chips that would then be used for further processing. In other contexts you might be dividing your data not just in ‘use’ and ‘discard’, but potentially in multiple groups, e.g. for different crystal forms. One way or the other, the output of that first processing stage would be a Dataset that is essentially a grouping of other Datasets. This led to two changes in the model:
-
While you might have to have a separate Dataset object for each image – in order to be able to track them individually – it is not attractive to also require a separate DataInput object for each of these Datasets, when all the input parameters are going to be the same for every image. Therefore the link between DataInput and Dataset has been changed, so it is many-to-many instead of many-to-one.
-
In order to support a dataset that is basically a grouping of other Datasets, it is necessary to add a mechanism for storing more or less arbitrary links between Datasets.
The general situation remains the same as in version 0.0.2. All the classes in the diagram are abstract, and there will be specific classes for different kinds of experiments (MX, SSX, …), different kinds of processing, different types of datasets etc., with a schema to ensure that connected objects are of compatible type. All objects should have space for parameters (‘metadata’) with a slot for program-specific data, and a UUID to identify the object. We should aim to use the same schemas (classes) in multiple contexts, e.g. a SweepData dataset could be used for both diffraction plan, experimental setup, experimental results, and processing input.
Abstract classes defining the model structure and the links between them. For those unused to UML note the cardinalities: ‘1’ (mandatory), ‘0..1’ (optional), and ‘*’ (0 to many)
The AbstractJob could be either an actual experiment or a processing job. Each can have a single JobInput object describing the input, and a single Result object describing the output. There are various options for splitting or combining classes here, but that is a matter of modelling technique rather than substance. All actual data live in some type of Dataset, which can be the output of a single AbstractJob. For input into AbstractJobs the DataInput object serves to connect one or more Datasets to a specific JobInput, and holds parameters that refer to that combination, rather than to any of the objects by itself.
The fromLINKNAME – toLINKNAME link to Dataset is a way of diagramming the situation where it is possible to make general many-to-many links between (different types of) Dataset. In the (hypothetical) case of a ReflectionData Dataset, you might have a many-to-many link with role names mergedDatasets and unmergedDatasets.
The links to Sample, like the links from Dataset to DataInput and to itself are crosslinks, that cannot be properly represented in a tree structure like JSON. Depending on context the API could treat them either by making duplicate records (e.g. input to a processing program would contain all necessary JobInput, DataInput and Dataset records), or by adding the UUID of the referenced object.
As in version 0.0.2, mapping to ICAT suffers from the problem that ICAT 1) has fewer classes, 2) does not support links between different Datasets; 3) connects to the Sample at the level of the ICAT Dataset rather than DataCollection. The diagram below shows one way one might use ICAT to store the data structure shown above. The solution relies on representing both Dataset and DataInput by a combination of ICAT Dataset and Datafile objects, and using dummy Datafile objects combined with the RelatedDataFile class to store the crosslinks. Links to the Sample would have to be handled in an ad-hoc manner as well.
There may not be any plan to use SciCat directly as the framework for building a LIMS, but it would still be useful to see how our structure could be mapped to SciCat. SciCat, as opposed to ICAT, does support links between Datasets. There is no equivalent to the Job or DataCollection classes in ICAT, so we combine the AbstractJob, JobInput and Result into a single special-purpose Dataset, here marked Result. The RelationshipClass supports all required links between Datasets. The DataInput is again a dummy Dataset that does not hold data of its own, but serves to connect a Result Dataset to its input in such a way that the role the input Dataset relative to this particular Result can be specified.
