Draft proposal, version 0.1.0
<title></title>
<meta name="generator" content="LibreOffice 24.2.1.2 (Linux)"/>
<style type="text/css">
@page { size: 21cm 29.7cm; margin: 2cm }
h3 { color: #000000; text-align: left; orphans: 2; widows: 2; margin-top: 0.25cm; margin-bottom: 0.21cm; direction: ltr; background: transparent; page-break-after: avoid }
h3.western { font-family: "Liberation Sans", serif; font-size: 14pt; so-language: en-GB; font-weight: bold }
h3.cjk { font-family: "FandolFang R"; font-size: 14pt; so-language: zh-CN; font-weight: bold }
h3.ctl { font-family: "Droid Sans Devanagari"; font-size: 14pt; so-language: hi-IN; font-weight: bold }
h2 { color: #000000; text-align: left; orphans: 2; widows: 2; margin-top: 0.35cm; margin-bottom: 0.21cm; direction: ltr; background: transparent; page-break-after: avoid }
h2.western { font-family: "Liberation Sans", serif; font-size: 16pt; so-language: en-GB; font-weight: bold }
h2.cjk { font-family: "FandolFang R"; font-size: 16pt; so-language: zh-CN; font-weight: bold }
h2.ctl { font-family: "Droid Sans Devanagari"; font-size: 16pt; so-language: hi-IN; font-weight: bold }
This version, which has been jumped from 0.0.5 to 0.1.0 is no longer a discussion starter, but an actual proposal. Once people have reacted to it, and it has been harmonised with the recent proposal for modelling Sample shipping, it should be converted to a JSON schema. Please note that the precise tag and class names are not set in stone, and proposals to replace them with more descriptive names are very welcome.
The point of MXLIMS is to make a series of JSON schemas that describe an API for the data you need to pass around. That means communications from users shipping samples, to beamline control systems, to LIMS systems, to data storage, querying and viewing results. The actual data (images, reflection sets, maps, models. ...) live in files that are not included but only referenced from MXLIMS. The metadata we cover are all described in the schemas.
The initial scope of MXLIMS is macromolecular crystallography, starting with MX. That would cover everything from describing the samples (and experiment plans) you send to synchrotrons, to acquisition, processing, and specifying, querying, and accessing processed data. The work may eventually be extended to further (neighbouring) techniques, and to downstream processing.
Although the main purpose of this work is to make a model for an API, it is necessary to consider how these data structures can be stored in a LIMS, retrieved, and how to keep the model manageable as it expands and changes. This requires a basic structure to the model that can be mapped to a maintainable database. We are aiming for a system similar to MongoDB, with a minimal number of database tables that take care of inter-object links, while the bulk of the information is kept in metadata dictionaries inside these tables. The implementation of LIMS systems is beyond the scope of MXLIMS, and indeed purely a matter for the implementors, but the core structure defines a minimum level of complexity needed to properly reflect the data.
A major problem with ISPyB was that a precise and detailed data model
for a large and changing scientific area rapidly becomes unwieldy.
The number of tables proliferate, and it becomes impossible to make
modifications without breaking existing code. The MongoDB approach
helps in that you can version the metadata and support multiple
versions in parallel. To further avoid these problems we are trying
to limit the number of different schemas, and to keep them generic.
Also we are allowing for additional keyword:value data in each
object, either in namespaced extensions where each program or site
can define its own schemas, or in a schema-free uncontrolled
extensions property. This should at least make it possible for each
site to cater for its needs without misusing the well-defined parts
of the data model.
The core model is shown in the
diagram below. It should be emphasised that the core model consists
of abstract classes, and is not supposed to replace the specific
schemas, but only to harmonise them. There are essentially four
things, or abstract classes in the model (Jobs, Datasets,
LogisticalSamples, and SamplePeparations). These abstract classes and
the links between them form the skeleton of the model. Anything that
can be referenced to from the outside must be a subclass of one of
these four abstract classes, and all inter-object links should follow
the core model. Each actual schema further defines the data for a
given subtype like a Plate, Mounted Crystal, MX experiment,
reflection file, ...) and other schemas (e.g. for CrystalForm) can be nested inside these schemas.
Core Model This diagram shows the core of the model with the four core abstract classes, and the links between them. In JSON documents links could typically be implemented by simply including the target schemas, but the links are optional – i.e. you would always be free to not include the linked-to objects. Links with arrows are one-way navigable (i.e. a Dataset would never contain the Jobs that used it as input), others are two-way. Note that the type of the links can be set in the individual schemas, to control (up to a point, at least) e.g. what types of Datasets can be used as input for a given kind of processing, or what kind of LogisticalSamples can be contained within a puck.
The type field indicates the kind
the subtype, i.e. which schema describes the object. This field will
only be explicitly populated where the object type cannot be
determined from context (as it can, for instance, in a Pydantic data
structure). The derived_from_id field serves only for provenance
tracking and will be stored as such also in JSON data.
When implemented as a JSON document,
the links are translated as contents, and the two-way links could go
either way. A Job document could contain a sample field with
LogisticalSample JSON, and lists of datasets for results, templates,
input_data, and reference_data. However, you could also have a
top-level Dataset, containing a source link with a Job, which in
turn contained a ‘sample element, which contained a preparation
element and so forth. In a JSON document you would always be free to
omit linked-to elements (otherwise you could end up with huge
networks of JSON objects.
The modelling here is heavily influenced by the need to support properly the acquisition and processing of multi-sweep experiments. In a model where you assume 1 Sample : 1 Sweep : 1 processing result there is some sense in basing your modelling and your viewers on purely Datasets with attached Metadata. But with multi-sweep experiments you need to be able to record and view information that applies to the experiment as such (e.g. the strategy being followed) that cannot be attached to a single Sweep. Similarly multi-sweep processing, potentially producing several Datasets, cannot be accurately displayed if the only object you have available is the Dataset. Nor can you hang everything off Samples, once you admit the possibility of processing data jointly from multiple Samples. The introduction of the Job object gives you a place to hang information about the Experiments of the Calculations you are performing – which is likely what the users would be interested in – without having to hang it off Datasets. Arguably viewers should be organised as list of Jobs.
The proposed combination of objects and links should be enough to allow full provenance tracking. The Job class connects the experiment/calculation, and its various kinds of inputs to the output Datasets, and the LogisticalSample that the Job was performed on (for an Experiment).
| Dataset | Abstract ‘Superschema’ for datasets | |
|---|---|---|
| uuid | UUID | Globally unique identifier. |
| source | Job | Optional. return link for Job.results |
| role | str | Role of Dataset relative to the source Job. E.g. result, characterisation, centring, intermediate
|
| sample | LogisticalSample | Optional link. return link for LogisticalSample.datasets |
| derived_from_id | UUID | Optional. UUID for Dataset from which this Dataset was derived |
| extensions | dict | Unconstrained, schema-free additional parameters. Avoid where possible. |
| _extensions | _Dataset | Extensions governed by registered schema starting with (e.g. ‘GPHL’) |
This part of the model is still incomplete, and there are a number of outstanding questions that need answering. Bullet points that start with NBNB contain points for discussion.
The model for this part includes CrystallographyProcessing (a Job), ReflectionSet (a Dataset), ReflectionShell, and some low-level data structures (ObservedCriterion, QualityFactor, Ellipsoid, and SortedEllipsoid). The input is based on the mmCIF categories reflns and reflns_shell (https://mmcif.wwpdb.org/dictionaries/mmcif_pdbx_v50.dic/Categories/) and the input specification to MRFANA. The idea is that values should be mapped directly to the relevant mmCIF fields, since these 1) are precisely defined, 2) will eventually be used for storing the data in the PDB. Using MRFANA input parameters to specify cutoffs etc. should make it possible to specify comparable data treatment for different programs and synchrotrons, since MRFANA is open-source and specifically designed to provide reproducible quality factors in different software environments.
Unlike MX experiments this Job actually does require input datasets, and this schema shows the need for multiple types of input data, to wit ‘input_data’ and ‘reference_data’. Other types of processing job might require more links at this level. When passed around as JSON documents input_data, reference_data and result would simply be included in the document – note that the schemas contain only the metadata, whereas the actual data live in the pointed-to files. Within a LIMS system these links would presumably be handled as foreign keys between tables, but that is up to individual implementations.
�