Draft proposal, version 0.1.1
Version 0.1.1 incorporates feedback from the MXLIMS meeting on May 7 2024, as well as from Global Phasing, and includes a draft for crystallographic processing. It is intended for discussion at the MXLIMS meeting on May 14. Once the feedback from this meeting ahs been incorporated, the result should be converted to JSON schemas and checked in.
The point of MXLIMS is to make a series of JSON schemas that describe an API for the data you need to pass around. That means communications from users shipping samples, to beamline control systems, to LIMS systems, to data storage, querying and viewing results. The actual data (images, reflection sets, maps, models. ...) live in files that are not included but only referenced from MXLIMS. The metadata we cover are all described in the schemas.
The initial scope of MXLIMS is macromolecular crystallography, starting with MX. That would cover everything from describing the samples (and experiment plans) you send to synchrotrons, to acquisition, processing, and specifying, querying, and accessing processed data. The work may eventually be extended to further (neighbouring) techniques, and to downstream processing.
The main purpose of this work is to make a model for an API, that serves to transfer data as messages between programs, for sample shipping etc. This requires detailed schemas that allow precise validation of input, particularly form external sources like sample shipping. On the other hand it is necessary to consider how these data structures can be stored in a LIMS, retrieved, and how to keep the model manageable as it expands and changes. This requires a basic structure to the model that can be mapped to a maintainable database. We are aiming for a system similar to MongoDB, with a minimal number of database tables that take care of inter-object links, while the bulk of the information is kept in metadata dictionaries inside these tables. The implementation of LIMS systems is beyond the scope of MXLIMS, and indeed purely a matter for the implementors, but the core structure defines a minimum level of complexity needed to properly reflect the data.
A major problem with ISPyB was that a precise and detailed data model for a large and changing scientific area rapidly becomes unwieldy. The number of tables proliferate, and it becomes impossible to make modifications without breaking existing code. The MongoDB approach helps in that you can version the metadata and support multiple versions in parallel. To further avoid these problems, we must try to limit the number of different schemas, and to keep them generic. This is sometimes in contrast to the need for schema validation mentioned in the preceding paragraph, but both objectives must be kept in mind. Also we are allowing for additional keyword:value data in each object, either in namespaced extensions where each program or site can define its own schemas, or in a schema-free uncontrolled extensions property. This should at least make it possible for each site to cater for its needs without misusing the well-defined parts of the data model.
The core model is shown in the diagram below. It should be emphasised that the core model consists of abstract classes, and is not supposed to replace the specific schemas, but only to harmonise them. There are essentially four things, or abstract classes in the model (Jobs, Datasets, LogisticalSamples, and PreparedSamples). These abstract classes and the links between them form the skeleton of the model. Anything that can be referenced from the outside must be a subclass of one of these four abstract classes, and all inter-object links should follow the core model. Each actual schema further defines the data for a given subtype like a Plate, Mounted Crystal, MX experiment, reflection file, ...) and other schemas (e.g. for UnitCell) can be nested inside these schemas.
Core Model This diagram shows the core of the model with the four core abstract classes, and the links between them. In JSON documents links could typically be implemented by simply including the target schemas, but the links are optional – i.e. you would always be free to not include the linked-to objects. Links with arrows are one-way navigable (i.e. a Dataset would never contain the Jobs that used it as input), others are two-way. Note that the type of the links can be set in the individual schemas, to control (up to a point, at least) e.g. what types of Datasets can be used as input for a given kind of processing, or what kind of LogisticalSamples can be contained within a puck.
The type field indicates the kind the subtype, i.e. which schema describes the object. This field will only be explicitly populated where the object type cannot be determined from context (as it can, for instance, in a Pydantic data structure). The derived_from_id field serves only for provenance tracking and will be stored as such also in JSON data.
When implemented as a JSON document, the links are translated as contents, and the two-way links could go either way. A Job document could contain a sample field with LogisticalSample JSON, and lists of datasets for results, templates, input_data, and reference_data. However, you could also have a top-level Dataset, containing a source link with a Job, which in turn contained a ‘sample element, which contained a preparation element and so forth. In a JSON document you would always be free to omit linked-to elements (otherwise you would end up with huge networks of JSON objects.
The modelling here is heavily influenced by the need to support properly the acquisition and processing of multi-sweep experiments. In a model where you assume 1 Sample : 1 Sweep : 1 processing result, there is some sense in basing your modelling and your viewers on purely Datasets with attached Metadata. But with multi-sweep experiments you need to be able to record and view information that applies to the experiment as such (e.g. the strategy being followed) that cannot be attached to a single Sweep. Similarly multi-sweep processing, potentially producing several Datasets, cannot be accurately displayed if the only object you have available is the Dataset. Nor can you hang everything off Samples, once you admit the possibility of processing data jointly from multiple Samples. The introduction of the Job object gives you a place off which to hang information about the Experiments or the Calculations you are performing – which is likely what the users would be interested in – without having to hang it off Datasets. Arguably, viewers should be organised as list of Jobs.
The proposed combination of objects and links should be enough to allow full provenance tracking. The Job class connects the experiment/calculation and its various kinds of inputs to the output Datasets, and the LogisticalSample that the Job was performed on (for an Experiment). The Dataset.derived_from_id allows you to make modified Datasets (e,.g, for image selection) while keeping track of the provenance.
| Dataset | Abstract ‘Superschema’ for datasets | |
|---|---|---|
| uuid | UUID | Globally unique identifier. |
| source | Job | Optional. Job that created the Dataset. return link for Job.results |
| role | str | Role of Dataset relative to the source Job. E.g. Result, Intermediate, Characterisation, Centring. Intended for filtering/selection of Datasets. |
| sample | LogisticalSample | Optional link. Sample relevant to Dataset, overrides Job.sample. return link for LogisticalSample.datasets |
| derived_from_id | UUID | Optional. UUID for Dataset from which this Dataset was derived. For modified Datasets without a source
|
| extensions | dict | Unconstrained, schema-free additional parameters. Avoid where possible. |
| namespace_extensions | NAMESPACE_Dataset | Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL') |
The Dataset is the superclass for any type of datasets, images, reflection files, models, maps, etc. The actual data files are stored separately, but the Dataset must contain the file pointers. Since the same data could be stored in a number of complex ways, e.g. as individual images, as a single multi-image files, or as part of a larger HDF5 data structure, the file specification must support any of these, including specifying subsets of larger data structures. A Dataset is characterised by its role, which describes it relative to the experiment or calculation that created it. For jobs that also create intermediate or secondary results, the Dataset(s) that are the intended output should have role of result.
The (optional) source link leads from the Dataset to the Job that created it, for tracking purposes. The derived_from_id provides an alternative provenance. It may be that the input to a calculation uses only part of a Dataset, e.g. after filtering out unwanted images. It may also be that you need to attach parameters to calculation input that belong neither to the Dataset as such, not to the calculation, but to the combination of the two, e.g. a weight parameter. In either case you can create a duplicate Dataset record with the data contents and parameters you need and use the derived_from_id for provenance tracking. In general, one would not duplicate the actual data files, but only the Dataset record pointing to them.
| Job | Abstract ‘Superschema’ for experiments and processing jobs | |
|---|---|---|
| uuid | UUID | Globally unique identifier. |
| sample | LogisticalSample | Optional link. return link for LogisticalSample.jobs |
| templates | list(Dataset) | Templates with parameters for output datasets – e.g. diffraction plan, processing plan |
| input_data | list(Dataset) | Input data sets (pre-existing) |
| reference_data | list(Dataset) | Reference Datasets |
| results | list(Dataset) | Datasets produced by Job |
| start_time | Timestamp | Actual starting time for Experiment/Calculation |
| end_time | Timestamp | Actual finishing time for Experiment/Calculation |
| job_status | Enum | Status of job. E.g. Template, Ready, Running, Completed, Failed, Aborted
|
| extensions | dict | Unconstrained, schema-free additional parameters. Avoid where possible. |
| namespace_extensions | NAMESPACE_Job | Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL') |
Depending on the type, the Job could specify either an actual experiment, or a processing job or a calculation. Since a Job (or experiment) can have multiple input and output data sets, the Job, rather than the Dataset or Sample, is the natural unit for the user to look at in overviews. The data produced by a Job are found in the results link. Job input can be specified as input_data (to be processed), reference_data, and templates. Templates contain the same kind of Dataset records as the results (avoiding the need for separate ‘diffraction plan’ schemas) but contain input parameters and are used to handle e.g. diffraction plans and processing plans. Templates would often be sparsely populated and should not contain any links to actual files. SA regards teh job_status, ‘Template’ signifies a Job that is not ready for running, intended either for later filling in, or only to serve as a template to generate other jobs.
| LogisticalSample | Abstract ‘Superschema’ For logistical samples (from Dewars and Plates to drops, pins or crystals | |
|---|---|---|
| uuid | UUID | Globally unique identifier. |
| preparation | PreparedSample | optional link – The sample preparation that applies to this LogisticalSample and all its contents |
| container | LogisticalSample | optional link – LogisticalSample containing this one |
| contents | list(LogisticalSample) | optional link – LogisticalSample contained in this one |
| jobs | list(Job) | Jobs (templates, planned, initiated, or completed) for LogisticalSample. |
| datasets | list(Dataset) | Datasets (templates, planned, initiated, or completed) for LogisticalSample. |
| extensions | dict | Unconstrained, schema-free additional parameters. Avoid where possible. |
| namespace_extensions | NAMESPACE_LogisticalSample | Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL') |
The LogisticalSample describes the kind of samples you ship, giving a nested series of containers from Shipment at the top down to crystals (mounted or in-situ) at the bottom of the hierarchy. The LogisticalSample can be linked to either Jobs or Datasets at any level. This is overkill, necessary to make the class generic, but depending on the experiment type you can have meaningful links to experiments not only from individual crystals, but also from drops, SSX or XFEL grids or streams, or multi-crystal loops; and in some experiment types (e.g. preparations of cubic lipid phase) you do not identify the individual crystals until partway through the experiment.
Note that diffraction plans or processing plans are specified by adding Job objects (with template Datasets) that contain the parameters you want to use for acquisition (including characterisation or centring if desired) or processing.
| PreparedSample | Abstract ‘Superschema’ for Sample preparations (describing sample content) | |
|---|---|---|
| uuid | UUID | Globally unique identifier. |
| samples | List(LogisticalSample) | Logistical Samples with contents from this PreparedSample) |
| extensions | dict | Unconstrained, schema-free additional parameters. Avoid where possible. |
| namespace_extensions | NAMESPACE_SamplePreparation | Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL') |
The PreparedSample describes the actual content of the Sample, as well as the identifiers that point back to the act of preparing it. The Sample could refer to multiple crystals, and in some cases (e.g. SSX) even to multiple grids or streams.
This class is a type of Job; the schema contains information and results that applies to the experiment as a whole regardless of the number of sweeps. The templates (‘diffraction plan’) could contain either generic Sweep records (such as ‘Characterisation’ and ‘Acquisition’) or a complete list of Dataset metadata records, such as could be submitted to an acquisition queue.
The expected_resolution is the resolution that the user expects and wants to measure to; it is used to set up the experiment (including the detector distance). ‘required resolution’ (‘measure only if better than...’) is not catered for in the model as it stands.