Draft proposal, version 0.1.0

This version, which has been jumped from 0.0.5 to 0.1.0 is no longer a discussion starter, but an actual proposal. Once people have reacted to it, and it has been harmonised with the recent proposal for modelling Sample shipping, it should be converted to a JSON schema. Please note that the precise tag and class names are not set in stone, and proposals to replace them with more descriptive names are very welcome.

Scope

The point of MXLIMS is to make a series of JSON schemas that describe an API for the data you need to pass around. That means communications from users shipping samples, to beamline control systems, to LIMS systems, to data storage, querying and viewing results. The actual data (images, reflection sets, maps, models. ...) live in files that are not included but only referenced from MXLIMS. The metadata we cover are all described in the schemas.

The initial scope of MXLIMS is macromolecular crystallography, starting with MX. That would cover everything from describing the samples (and experiment plans) you send to synchrotrons, to acquisition, processing, and specifying, querying, and accessing processed data. The work may eventually be extended to further (neighbouring) techniques, and to downstream processing.

Although the main purpose of this work is to make a model for an API, it is necessary to consider how these data structures can be stored in a LIMS, retrieved, and how to keep the model manageable as it expands and changes. This requires a basic structure to the model that can be mapped to a maintainable database. We are aiming for a system similar to MongoDB, with a minimal number of database tables that take care of inter-object links, while the bulk of the information is kept in metadata dictionaries inside these tables. The implementation of LIMS systems is beyond the scope of MXLIMS, and indeed purely a matter for the implementors, but the core structure defines a minimum level of complexity needed to properly reflect the data.

A major problem with ISPyB was that a precise and detailed data model for a large and changing scientific area rapidly becomes unwieldy. The number of tables proliferate, and it becomes impossible to make modifications without breaking existing code. The MongoDB approach helps in that you can version the metadata and support multiple versions in parallel. To further avoid these problems we are trying to limit the number of different schemas, and to keep them generic. Also we are allowing for additional keyword:value data in each object, either in namespaced extensions where each program or site can define its own schemas, or in a schema-free uncontrolled extensions property. This should at least make it possible for each site to cater for its needs without misusing the well-defined parts of the data model.

Core Model

The core model is shown in the diagram below. It should be emphasised that the core model consists of abstract classes, and is not supposed to replace the specific schemas, but only to harmonise them. There are essentially four things, or abstract classes in the model (Jobs, Datasets, LogisticalSamples, and SamplePeparations). These abstract classes and the links between them form the skeleton of the model. Anything that can be referenced to from the outside must be a subclass of one of these four abstract classes, and all inter-object links should follow the core model. Each actual schema further defines the data for a given subtype like a Plate, Mounted Crystal, MX experiment, reflection file, ...) and other schemas (e.g. for CrystalForm) can be nested inside these schemas.

Core Model This diagram shows the core of the model with the four core abstract classes, and the links between them. In JSON documents links could typically be implemented by simply including the target schemas, but the links are optional – i.e. you would always be free to not include the linked-to objects. Links with arrows are one-way navigable (i.e. a Dataset would never contain the Jobs that used it as input), others are two-way. Note that the type of the links can be set in the individual schemas, to control (up to a point, at least) e.g. what types of Datasets can be used as input for a given kind of processing, or what kind of LogisticalSamples can be contained within a puck.

The type field indicates the kind the subtype, i.e. which schema describes the object. This field will only be explicitly populated where the object type cannot be determined from context (as it can, for instance, in a Pydantic data structure). The derived_from_id field serves only for provenance tracking and will be stored as such also in JSON data.

When implemented as a JSON document, the links are translated as contents, and the two-way links could go either way. A Job document could contain a sample field with LogisticalSample JSON, and lists of datasets for results, templates, input_data, and reference_data. However, you could also have a top-level Dataset, containing a source link with a Job, which in turn contained a ‘sample element, which contained a preparation element and so forth. In a JSON document you would always be free to omit linked-to elements (otherwise you could end up with huge networks of JSON objects.

The modelling here is heavily influenced by the need to support properly the acquisition and processing of multi-sweep experiments. In a model where you assume 1 Sample : 1 Sweep : 1 processing result there is some sense in basing your modelling and your viewers on purely Datasets with attached Metadata. But with multi-sweep experiments you need to be able to record and view information that applies to the experiment as such (e.g. the strategy being followed) that cannot be attached to a single Sweep. Similarly multi-sweep processing, potentially producing several Datasets, cannot be accurately displayed if the only object you have available is the Dataset. Nor can you hang everything off Samples, once you admit the possibility of processing data jointly from multiple Samples. The introduction of the Job object gives you a place to hang information about the Experiments of the Calculations you are performing – which is likely what the users would be interested in – without having to hang it off Datasets. Arguably viewers should be organised as list of Jobs.

The proposed combination of objects and links should be enough to allow full provenance tracking. The Job class connects the experiment/calculation, and its various kinds of inputs to the output Datasets, and the LogisticalSample that the Job was performed on (for an Experiment).

Dataset

Dataset		Abstract ‘Superschema’ for datasets
uuid	UUID	Globally unique identifier.
source	Job	Optional. return link for Job.results
role	str	Role of Dataset relative to the source Job. E.g. result, characterisation, centring, intermediate
sample	LogisticalSample	Optional link. return link for LogisticalSample.datasets
derived_from_id	UUID	Optional. UUID for Dataset from which this Dataset was derived
extensions	dict	Unconstrained, schema-free additional parameters. Avoid where possible.
_extensions	NAMESPACE_Dataset	Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL’)

The Dataset is the superclass for any type of datasets, images, reflection files, models, maps, etc. The actual data files are stored separately, but the Dataset must contain the file pointers. Since the same data could be stored in a number of complex ways, e.g. as individual images, as a single multi-image files, or as part of a larger HDF5 data structure, the file specification must support any of these, including specifying subsets of larger data structures. A Dataset is characterised by its role, which defines it relative to the experiment or calculation that created it. For jobs that also create intermediate or secondary results, the Dataset(s) that are the intended output should have role of result.

The (optional) source link leads from the Dataset to the Job that created it, for tracking purposes. The derived_from_id provides an alternative provenance. It may be that the input to a calculation uses only part of a Dataset, e.g. after filtering out unwanted images. It may also be that you need to attach parameters to calculation input that belong neither to the Dataset as such, not to the calculation, but to the combination of the two, e.g. a weight parameter. In either case you can create a duplicate Dataset record with the data contents and parameters you need, and use the derived_from_id for provenance tracking. In general one would not duplicate the actual data files, but only the Dataset record pointing to them.

Job

Job		Abstract ‘Superschema’ for experiments and processing jobs
uuid	UUID	Globally unique identifier.
sample	LogisticalSample	Optional link. return link for LogisticalSample.jobs
templates	list(Dataset)	Templates with parameters for output datasets – e.g. diffraction plan, processing plan
input_data	list(Dataset)	Input data sets (pre-existing)
reference_data	list(Dataset)	Reference Datasets
results	list(Dataset)	Datasets produced by Job
extensions	dict	Unconstrained, schema-free additional parameters. Avoid where possible.
_extensions	NAMESPACE_Job	Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL’)

Depending on the type, the Job could specify either an actual experiment, or a processing job or calculation. Since a Job (or experiment) can have multiple input and output data sets, the Job, rather than the Dataset, is the natural unit for the user to look at in overviews. The data produced by a Job are found in the results link. Job input can be specified as input_data (to be processed), reference_data, and templates. Templates contain the same kind of Dataset records as the results (avoiding the need for separate ‘diffraction plan’ schemas), but contain input parameters and are used to handle e.g. diffraction plans and processing plans. Templates would often be sparsely populated, and should not contain any links to actual files.

LogisticalSample

LogisticalSample		Abstract ‘Superschema’ For logistical samples (from Dewars and Plates to drops or crystal-containing regions)
uuid	UUID	Globally unique identifier.
preparation	SamplePreparation	optional link – The Sample preparation that applies to this LogisticalSample and all its contents
container	LogisticalSample	optional link – LogisticalSample containing this one
contents	list(LogisticalSample)	optional link – LogisticalSample contained in this one
jobs	list(Job)	Jobs (templates, planned, initiated, or completed) for LogisticalSample.
datasets	list(Dataset)	Datasets (templates, planned, initiated, or completed) for LogisticalSample.
extensions	dict	Unconstrained, schema-free additional parameters. Avoid where possible.
_extensions	NAMESPACE_LogisticalSample	Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL’)

The LogisticalSample describes the kind of samples you ship, giving a nested series of containers from Shipment at the top down to crystals (mounted or in-situ) at the bottom of the hierarchy. The LogisticalSample can be linked to either Jobs or Datasets at any level. This is overkill, necessary to make the class generic, but depending on the experiment type you can have meaningful links to experiments not only from individual crystals, but also from drops, SSX or XFEL grids or streams, or multicrystal loops, and in some experiment types (e.g. preparations of cubic lipid phase) you do not identify the individual crystals until partway through the experiment.

Note that diffraction plans or processing plans are specified by adding Job objects (with template Datasets) that contain the parameters you want to use for acquisition (including characterisation or centring if desired) or processing.

SamplePreparation

SamplePreparation		Abstract ‘Superschema’ for Sample preparations (describing sample content)
uuid	UUID	Globally unique identifier.
samples	List(LogisticalSample)	LogisticalSamples with contents from this SamplePreparation)
extensions	dict	Unconstrained, schema-free additional parameters. Avoid where possible.
_extensions	NAMESPACE_SamplePreparation	Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL’)

The SamplePreparation describes the actual content of the Sample, as well as the identifiers that point back to the act of preparing it. The Sample could refer to multiple crystals and in some cases (e.g. SSX) even to multiple grids or streams.