Draft proposal, version 0.1.0
This version, which has been jumped from 0.0.5 to 0.1.0 is no longer a discussion starter, but an actual proposal. Once people have reacted to it, and it has been harmonised with the recent proposal for modelling Sample shipping, it should be converted to a JSON schema. Please note that the precise tag and class names are not set in stone, and proposals to replace them with more descriptive names are very welcome.
The point of MXLIMS is to make a series of JSON schemas that describe an API for the data you need to pass around. That means communications from users shipping samples, to beamline control systems, to LIMS systems, to data storage, querying and viewing results. The actual data (images, reflection sets, maps, models. ...) live in files that are not included but only referenced from MXLIMS. The metadata we cover are all described in the schemas.
The initial scope of MXLIMS is macromolecular crystallography, starting with MX. That would cover everything from describing the samples (and experiment plans) you send to synchrotrons, to acquisition, processing, and specifying, querying, and accessing processed data. The work may eventually be extended to further (neighbouring) techniques, and to downstream processing.
Although the main purpose of this work is to make a model for an API, it is necessary to consider how these data structures can be stored in a LIMS, retrieved, and how to keep the model manageable as it expands and changes. This requires a basic structure to the model that can be mapped to a maintainable database. We are aiming for a system similar to MongoDB, with a minimal number of database tables that take care of inter-object links, while the bulk of the information is kept in metadata dictionaries inside these tables. The implementation of LIMS systems is beyond the scope of MXLIMS, and indeed purely a matter for the implementors, but the core structure defines a minimum level of complexity needed to properly reflect the data.
A major problem with ISPyB was that a precise and detailed data model
for a large and changing scientific area rapidly becomes unwieldy.
The number of tables proliferate, and it becomes impossible to make
modifications without breaking existing code. The MongoDB approach
helps in that you can version the metadata and support multiple
versions in parallel. To further avoid these problems we are trying
to limit the number of different schemas, and to keep them generic.
Also we are allowing for additional keyword:value data in each
object, either in namespaced extensions where each program or site
can define its own schemas, or in a schema-free uncontrolled
extensions property. This should at least make it possible for each
site to cater for its needs without misusing the well-defined parts
of the data model.
The core model is shown in the
diagram below. It should be emphasised that the core model consists
of abstract classes, and is not supposed to replace the specific
schemas, but only to harmonise them. There are essentially four
things, or abstract classes in the model (Jobs, Datasets,
LogisticalSamples, and SamplePeparations). These abstract classes and
the links between them form the skeleton of the model. Anything that
can be referenced to from the outside must be a subclass of one of
these four abstract classes, and all inter-object links should follow
the core model. Each actual schema further defines the data for a
given subtype like a Plate, Mounted Crystal, MX experiment,
reflection file, ...) and other schemas (e.g. for CrystalForm) can be nested inside these schemas.
Core Model This diagram shows the core of the model with the four core abstract classes, and the links between them. In JSON documents links could typically be implemented by simply including the target schemas, but the links are optional – i.e. you would always be free to not include the linked-to objects. Links with arrows are one-way navigable (i.e. a Dataset would never contain the Jobs that used it as input), others are two-way. Note that the type of the links can be set in the individual schemas, to control (up to a point, at least) e.g. what types of Datasets can be used as input for a given kind of processing, or what kind of LogisticalSamples can be contained within a puck.
The type field indicates the kind
the subtype, i.e. which schema describes the object. This field will
only be explicitly populated where the object type cannot be
determined from context (as it can, for instance, in a Pydantic data
structure). The derived_from_id field serves only for provenance
tracking and will be stored as such also in JSON data.
When implemented as a JSON document,
the links are translated as contents, and the two-way links could go
either way. A Job document could contain a sample field with
LogisticalSample JSON, and lists of datasets for results, templates,
input_data, and reference_data. However, you could also have a
top-level Dataset, containing a source link with a Job, which in
turn contained a ‘sample element, which contained a preparation
element and so forth. In a JSON document you would always be free to
omit linked-to elements (otherwise you could end up with huge
networks of JSON objects.
The modelling here is heavily influenced by the need to support properly the acquisition and processing of multi-sweep experiments. In a model where you assume 1 Sample : 1 Sweep : 1 processing result there is some sense in basing your modelling and your viewers on purely Datasets with attached Metadata. But with multi-sweep experiments you need to be able to record and view information that applies to the experiment as such (e.g. the strategy being followed) that cannot be attached to a single Sweep. Similarly multi-sweep processing, potentially producing several Datasets, cannot be accurately displayed if the only object you have available is the Dataset. Nor can you hang everything off Samples, once you admit the possibility of processing data jointly from multiple Samples. The introduction of the Job object gives you a place to hang information about the Experiments of the Calculations you are performing – which is likely what the users would be interested in – without having to hang it off Datasets. Arguably viewers should be organised as list of Jobs.
The proposed combination of objects and links should be enough to allow full provenance tracking. The Job class connects the experiment/calculation, and its various kinds of inputs to the output Datasets, and the LogisticalSample that the Job was performed on (for an Experiment).
| Dataset | Abstract ‘Superschema’ for datasets | |
|---|---|---|
| uuid | UUID | Globally unique identifier. |
| source | Job | Optional. return link for Job.results |
| role | str | Role of Dataset relative to the source Job. E.g. result, characterisation, centring, intermediate
|
| sample | LogisticalSample | Optional link. return link for LogisticalSample.datasets |
| derived_from_id | UUID | Optional. UUID for Dataset from which this Dataset was derived |
| extensions | dict | Unconstrained, schema-free additional parameters. Avoid where possible. |
| namespace_extensions | NAMESPACE_Dataset | Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL’) |
The Dataset is the superclass for any type of datasets, images,
reflection files, models, maps, etc. The actual data files are stored
separately, but the Dataset must contain the file pointers. Since the
same data could be stored in a number of complex ways, e.g. as
individual images, as a single multi-image files, or as part of a
larger HDF5 data structure, the file specification must support any
of these, including specifying subsets of larger data structures. A
Dataset is characterised by its role, which defines it relative to
the experiment or calculation that created it. For jobs that also
create intermediate or secondary results, the Dataset(s) that are the
intended output should have role of result.
The (optional) source link leads
from the Dataset to the Job that created it, for tracking purposes.
The derived_from_id provides an alternative provenance. It may be
that the input to a calculation uses only part of a Dataset, e.g.
after filtering out unwanted images. It may also be that you need to
attach parameters to calculation input that belong neither to the
Dataset as such, not to the calculation, but to the combination of
the two, e.g. a weight parameter. In either case you can create a
duplicate Dataset record with the data contents and parameters you
need, and use the derived_from_id for provenance tracking. In general
one would not duplicate the actual data files, but only the Dataset
record pointing to them.
| Job | Abstract ‘Superschema’ for experiments and processing jobs | |
|---|---|---|
| uuid | UUID | Globally unique identifier. |
| sample | LogisticalSample | Optional link. return link for LogisticalSample.jobs |
| templates | list(Dataset) | Templates with parameters for output datasets – e.g. diffraction plan, processing plan |
| input_data | list(Dataset) | Input data sets (pre-existing) |
| reference_data | list(Dataset) | Reference Datasets |
| results | list(Dataset) | Datasets produced by Job |
| extensions | dict | Unconstrained, schema-free additional parameters. Avoid where possible. |
| namespace_extensions | NAMESPACE_Job | Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL’) |
Depending on the type, the Job could specify either an actual experiment, or a processing job or calculation. Since a Job (or experiment) can have multiple input and output data sets, the Job, rather than the Dataset, is the natural unit for the user to look at in overviews. The data produced by a Job are found in the results link. Job input can be specified as input_data (to be processed), reference_data, and templates. Templates contain the same kind of Dataset records as the results (avoiding the need for separate ‘diffraction plan’ schemas), but contain input parameters and are used to handle e.g. diffraction plans and processing plans. Templates would often be sparsely populated, and should not contain any links to actual files.
| LogisticalSample | Abstract ‘Superschema’ For logistical samples (from Dewars and Plates to drops or crystal-containing regions) | |
|---|---|---|
| uuid | UUID | Globally unique identifier. |
| preparation | SamplePreparation | optional link – The Sample preparation that applies to this LogisticalSample and all its contents |
| container | LogisticalSample | optional link – LogisticalSample containing this one |
| contents | list(LogisticalSample) | optional link – LogisticalSample contained in this one |
| jobs | list(Job) | Jobs (templates, planned, initiated, or completed) for LogisticalSample. |
| datasets | list(Dataset) | Datasets (templates, planned, initiated, or completed) for LogisticalSample. |
| extensions | dict | Unconstrained, schema-free additional parameters. Avoid where possible. |
| namespace_extensions | NAMESPACE_LogisticalSample | Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL’) |
The LogisticalSample describes the kind of samples you ship, giving a nested series of containers from Shipment at the top down to crystals (mounted or in-situ) at the bottom of the hierarchy. The LogisticalSample can be linked to either Jobs or Datasets at any level. This is overkill, necessary to make the class generic, but depending on the experiment type you can have meaningful links to experiments not only from individual crystals, but also from drops, SSX or XFEL grids or streams, or multicrystal loops, and in some experiment types (e.g. preparations of cubic lipid phase) you do not identify the individual crystals until partway through the experiment.
Note that diffraction plans or processing plans are specified by adding Job objects (with template Datasets) that contain the parameters you want to use for acquisition (including characterisation or centring if desired) or processing.
| SamplePreparation | Abstract ‘Superschema’ for Sample preparations (describing sample content) | |
|---|---|---|
| uuid | UUID | Globally unique identifier. |
| samples | List(LogisticalSample) | LogisticalSamples with contents from this SamplePreparation) |
| extensions | dict | Unconstrained, schema-free additional parameters. Avoid where possible. |
| namespace_extensions | NAMESPACE_SamplePreparation | Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL’) |
The SamplePreparation describes the actual content of the Sample, as well as the identifiers that point back to the act of preparing it. The Sample could refer to multiple crystals and in some cases (e.g. SSX) even to multiple grids or streams.
This class is a type of Job; the schema contains information and results that applies to the experiment as a whole, regardless of the number of sweeps. The templates (‘diffraction plan’) could contain either generic Sweep records (such as ‘Characterisation’ and ‘Acquisition’) or a complete list of Dataset metadata records, such as could be submitted to an acquisition queue. The target_resolution is the resolution that the user expects and wants to measure to, and does not have a direct relation to the detector distance. ‘required resolution’ (‘measure only if better than...’) is not catered for in the model as it stands.
| CrystallographyExperiment | a type of Job | |
|---|---|---|
| Object-level attributes | ||
| uuid | UUID | Globally unique identifier. |
| sample | LogisticalSample | Optional link. return link for LogisticalSample.jobs |
| templates | list(Dataset) | Templates with parameters for output datasets – e.g. diffraction plan, processing plan |
| input_data | list(Dataset) | Input data sets (pre-existing) |
| reference_data | list(Dataset) | Reference Datasets |
| results | list(Dataset) | Datasets produced by Job |
| extensions | dict | Unconstrained, schema-free additional parameters. Avoid where possible. |
| namespace_extensions | NAMESPACE_CrystallographyExperiment | Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL) |
| Input: | ||
| experiment_kind | str | Native, MAD, SAD, ... |
| experiment_strategy | str | MXPressE, GPhL_quick, ... |
| target_resolution | float | |
| target_completeness | float | |
| target_multiplicity | float | |
| dose_budget | float | |
| characterisation_strategy | str | |
| recentring_strategy | str | |
| snapshot_count | ||
| wedge_width | float | Input for interleaving strategy, actual scans are given explicitly in the sweep |
| Output: | ||
| start_time | Timestamp | Time job started |
| completion_time | Timestamp | Time job completed |
| job_status | Enum | Job status (e.g. Template, Ready, Pending, Running, Failed, Completed) |
| measured_flux | float | |
| radiation_dose | float | Total radiation dose over experiment |
| crystal_form | CrystalForm | As determined during characterisation |
| Sweep | a type of Dataset | |
|---|---|---|
| Object-level attributes | ||
| uuid | UUID | Globally unique identifier. |
| source | Job | Optional. return link for Job.results |
| role | str | Role of Dataset relative to the source Job. E.g. result, acquisition, characterisation, centring, intermediate` |
| sample | LogisticalSample | Optional link. return link for LogisticalSample.datasets |
| derived_from_id | UUID | Optional. UUID for Dataset from which this Dataset was derived |
| extensions | dict | Unconstrained, schema-free additional parameters. Avoid where possible. |
| namespace_extensions | NAMESPACE_Sweep | Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL’) |
| General parameters: | ||
| annotation | str | |
| Acquisition: | ||
| exposure_time | float | |
| image_width | float | width of scan. |
| energy | float | |
| transmission | % | |
| Detector parameters: | ||
| detector_distance | float | NB, we do NOT store resolution. The detector distance is clearly defined, the resolution is not. |
| detector_binning_mode | Enum | |
| detector_roi_mode | Enum | |
| beam_position | (float, float) | Position on detector, determines detector offset. |
| beam_size | (float, float) | |
| beam_shape | Enum | |
| Scan and position parameters: | ||
| axis_positions_start | dict | Starting position of all axes, rotations or translations, including detector distance, by name. |
| scan_axis | Enum | Name of main scanned axis |
| overlap | float | Overlap between successive images. May be negative if images are not contiguous. Is this still necessary?? |
| axis_positions_end | dict | Positions at end of sweep for all axes that are scanned. NB scans may be acquired out of order, so this determines the limits of the sweep, not the temporal start and end points. |
| num_triggers | int | Instructions for how to set up detector and acquisition – do not modify meaning of other parameters. |
| num_images_per_trigger | int | Ibid. |
| scans | list[Scan] | Subdivisions of sweep – NB need not be contiguous, or in order, or add up to entire sweep. |
| File parameters: | NB. This assumes that path+file_type+prefix+run_number+image_no allows you to get an image. What more would we need? | |
| file_type | Enum | E.g. ,cbf, .cbf.tgz, .hdf5, ... |
| prefix | str | |
| run_number | int | |
| path | str |
The Sweep is a type of Dataset. The record is based on MX, but should work also for multi-crystal, and even SSX or XFEL experiments. In the latter two cases a lot of additional information would be needed.
Note that the Sweep specifies a single, continuous sweep range, with equidistant images given by image_width, and all starting motor positions in axis_position_start. axis_positions_end contain the end point of the sweep, and must have at least the value for the scan_axis; sweeps changing more than one motor (e.g. helical scan) can be represented by adding more values to axis_positions_end. The default number of images can be calculated from the sweep range and image_width. The actual number of images, the image numbering, and the order of acquisition (including interleaving) follows from the list of Scans. The role should be set to ‘result’ for those sweeps that are deemed to be the desired result of the experiment; in templates you would prefer to use acquisition for the Dataset that gives the acquisition parameters.
The Scan describes a continuously acquired set of images that forms a subset of the Sweep of which they form part. The ordinal gives the acquisition order of sweeps across an entire multisweep experiment; this allows you to describe out-of-order acquisition and interleaving.
| Scan | Subdivision of CollectionSweep | |
|---|---|---|
| scan_position_start | float | Value of scan_axis for first image |
| first_image_no | int | Image number to use for first image |
| num_images | int | |
| ordinal | int | acquisition order within experiment (not just within sweep) |
The LogisticalSample is just the abstract superclass, and the individual schemas will be as described in Ed Daniel’s Sample shipping proposal. The main difference is that the link to contents would be called contents at all levels, so that Shipment.contents’ replace the proposed Shipment.dewars and Shipment.plates. As well, of course, as there being some additional links at all levels, as described in the core model.
SamplePreparation (can anyone suggest a better name, btw?) is just a match to Ed Daniel’s Sample schema. There are a couple of proposed changes:
• protein and ligands would be replaced by the more general ‘macromolecule’ and ‘component’.
• This proposal allows for more than one crystal form; might that not be a requirement sometimes?
• The radiation_sensitivity is a sample parameter that is needed for (some) strategy programs.
| SamplePreparation | a type of SamplePreparation. Matches Sample
|
|
|---|---|---|
| Object-level attributes | ||
| uuid | UUID | Globally unique identifier. |
| samples | list(LogisticalSample) | LogisticalSamples whose contents are from this SamplePreparation |
| extensions | dict | Unconstrained, schema-free additional parameters. Avoid where possible. |
| namespace_extensions | NAMESPACE_SamplePreparation | Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL) |
| General: | ||
| crystal_forms | List[CrystalForm] | List of possible crystal forms |
| radiation_sensitivity | float | Radiation sensitivity relative to standard crystal |
| Content: | ||
| macromolecule | Macromolecule | Macromolecule being investigated |
| components | list(Component) | other components used in the Sample liquid (e.g. binders, substrates, metal ions, additives, ...) |
| identifiers | List[Identifier] | Identifiers of this sample |
The Crystal is a subtype of LogisticalSample, a combination of the Mounted crystal and the In-Situ crystal classes, combining the specific properties for each (positionInPin, imageRegion and plateRegion and making them optional. This is just a case of the general principle of preferring fewer schemas with optional fields to many, precise schemas. On similar lines the ImageRegion and PlateRegion (with separate links) have been replaces by a more general, multi-purpose ‘Region’ in parallel with an optional ‘image’ property.
| Crystal | A type of LogisticalSample. Matches MountedCrystal and In-situCrystal
|
|
|---|---|---|
| Object-level attributes | ||
| uuid | UUID | Globally unique identifier. |
| preparation | SamplePreparation | optional link – The Sample preparation taht applies to this LogisticalSample and all its contents |
| container | LogisticalSample | optional link – LogisticalSample containing this one |
| contents | empty list | contents must always be empty for this case |
| jobs | list(Job) | Jobs (templates, planned, initiated, or completed) for LogisticalSample. |
| datasets | list(Dataset) | Datasets (templates, planned, initiated, or completed) for LogisticalSample. |
| extensions | dict | Unconstrained, schema-free additional parameters. Avoid where possible. |
| namespace_extensions | NAMESPACE_Crystal | Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL') |
| Content: | ||
| image | CrystalImage | |
| region | Region | optional Region. Matches In-situCrystal.imageRegion or .plateRegion
|
| position | int | optional position. Matches MountedCrystal.position
|
Macromolecule and Component are just different
names for the Protein and Ligand schemas proposed by Ed Daniel.
| Macromolecule | Matches Protein. Incomplete |
|
|---|---|---|
| acronym | str | Matches proteinAcronym
|
| identifiers | List[Identifier] | Identifiers of this sample |
| Component | Matches Ligand. Incomplete. |
|
|---|---|---|
| identifiers | List[Identifier] | Identifiers of this sample |
The Region schema combines the schemas for Image Region, Plate Region, Region, Point, Line, Circle, Rectangle, and
Polygon, whereas CrystalImage is simply a renaming for DropImage.
As I understand it, you can get multiple crystals, with images, also
on pins with CrystalDirect technology.
| Region | Matches Region
|
|
|---|---|---|
| regionType | Enum | Matches Point, Circle, Line, Rectangle, Polygon
|
| units | Enum | Units used for defined points |
| points | list((float, float)) | From 1 to n points required to define region |
| radius | float | Radius. Used only for circle. |
| CrystalImage | Exact match to Drop Image. Not further specified here |
|---|
| CrystalForm | Matches Crystal Form
|
|
|---|---|---|
| a | float | |
| b | float | |
| c | float | |
| alpha | float | |
| beta | float | |
| gamma | float | |
| space_group_name | str | Space group name (unlike the SG number, this supports non-standard settings) |
| crystal_class_names | List[str] | Arithmetic crystal class names; to specify partial information or ambiguous symmetry |
This part of the model is still incomplete, and there are a number of outstanding questions that need answering. The details will be added as soon as possible
The model for this part includes CrystallographyProcessing (a Job), ReflectionSet (a Dataset), ReflectionShell, and some low-level data structures (ObservedCriterion, QualityFactor, Ellipsoid, and SortedEllipsoid). The input is based on the mmCIF categories reflns and reflns_shell (https://mmcif.wwpdb.org/dictionaries/mmcif_pdbx_v50.dic/Categories/) and the input specification to MRFANA. The idea is that values should be mapped directly to the relevant mmCIF fields, since these 1) are precisely defined, 2) will eventually be used for storing the data in the PDB. Using MRFANA input parameters to specify cutoffs etc. should make it possible to specify comparable data treatment for different programs and synchrotrons, since MRFANA is open-source and specifically designed to provide reproducible quality factors in different software environments.
Unlike MX experiments this Job actually does require input datasets, and this schema shows the need for multiple types of input data, to wit ‘input_data’ and ‘reference_data’. Other types of processing job might require more links at this level. When passed around as JSON documents input_data, reference_data and result would simply be included in the document – note that the schemas contain only the metadata, whereas the actual data live in the pointed-to files. Within a LIMS system these links would presumably be handled as foreign keys between tables, but that is up to individual implementations.
MORE TO COME
Implementing this model (also) within a LIMS system would presumably require some kind of table structure. As discussions have been going until now, the preferred approach would be to have a minimum of tables and keep most of the data in metadata dictionaries, as done e.g. in MongoDB. Below one proposal for how the present model might be implemented in a limited number of tables. The diagram shows the foreign keys and intra-LIMS links that would be needed. Clearly the implementation would be up to the LIMS system programmers, but this should at least show one possibility.
The Job.results link is a simple one-to-many link, but the links between the Job and Dataset that describe Job input are the most complicated part of the model. Since the input_data, templates and reference_data links are all many-to-many, the JobInputLink class is required to represent them. The JobInputLink.role attribute would be either ‘input_data’, ‘reference_data’ or ‘templates’in these cases. Alternatively Job.templates could be handled simply by not giving them a UUID and including them in the metadata – which would mean that each template could be used only once.. If necessary this structure could be used to handle additional kinds of links between Job and Dataset.
