Draft proposal, version 0.1.1

Version 0.1.1 incorporates feedback from the MXLIMS meeting on May 7 2024, as well as from Global Phasing, and includes a draft for crystallographic processing. It is intended for discussion at the MXLIMS meeting on May 14. Once the feedback from this meeting has been incorporated, the result should be converted to JSON schemas and checked in.

Scope

The point of MXLIMS is to make a series of JSON schemas that describe an API for the data you need to pass around. That means communications from users shipping samples, to beamline control systems, to LIMS systems, to data storage, querying and viewing results. The actual data (images, reflection sets, maps, models. ...) live in files that are not included but only referenced from MXLIMS. The metadata we cover are all described in the schemas.

The initial scope of MXLIMS is macromolecular crystallography, starting with MX. That would cover everything from describing the samples (and experiment plans) you send to synchrotrons, to acquisition, processing, and specifying, querying, and accessing processed data. The work may eventually be extended to further (neighbouring) techniques, and to downstream processing.

The main purpose of this work is to make a model for an API, that serves to transfer data as messages between programs, for sample shipping etc. This requires detailed schemas that allow precise validation of input, particularly form external sources like sample shipping. On the other hand it is necessary to consider how these data structures can be stored in a LIMS, retrieved, and how to keep the model manageable as it expands and changes. This requires a basic structure to the model that can be mapped to a maintainable database. We are aiming for a system similar to MongoDB, with a minimal number of database tables that take care of inter-object links, while the bulk of the information is kept in metadata dictionaries inside these tables. The implementation of LIMS systems is beyond the scope of MXLIMS, and indeed purely a matter for the implementors, but the core structure defines a minimum level of complexity needed to properly reflect the data.

A major problem with ISPyB was that a precise and detailed data model for a large and changing scientific area rapidly becomes unwieldy. The number of tables proliferate, and it becomes impossible to make modifications without breaking existing code. The MongoDB approach helps in that you can version the metadata and support multiple versions in parallel. To further avoid these problems, we must try to limit the number of different schemas, and to keep them generic. This is sometimes in contrast to the need for schema validation mentioned in the preceding paragraph, but both objectives must be kept in mind. Also we are allowing for additional keyword:value data in each object, either in namespaced extensions where each program or site can define its own schemas, or in a schema-free uncontrolled extensions property. This should at least make it possible for each site to cater for its needs without misusing the well-defined parts of the data model.

Core Model

The core model is shown in the diagram below. It should be emphasised that the core model consists of abstract classes, and is not supposed to replace the specific schemas, but only to harmonise them. There are essentially four things, or abstract classes in the model (Jobs, Datasets, LogisticalSamples, and PreparedSamples). These abstract classes and the links between them form the skeleton of the model. Anything that can be referenced from the outside must be a subclass of one of these four abstract classes, and all inter-object links should follow the core model. Each actual schema further defines the data for a given subtype like a Plate, Mounted Crystal, MX experiment, reflection file, ...) and other schemas (e.g. for UnitCell) can be nested inside these schemas.

Core Model This diagram shows the core of the model with the four core abstract classes, and the links between them. In JSON documents links could typically be implemented by simply including the target schemas, but the links are optional – i.e. you would always be free to not include the linked-to objects. Links with arrows are one-way navigable (i.e. a Dataset would never contain the Jobs that used it as input), others are two-way. Note that the type of the links can be set in the individual schemas, to control (up to a point, at least) e.g. what types of Datasets can be used as input for a given kind of processing, or what kind of LogisticalSamples can be contained within a puck. The type field indicates the kind the subtype, i.e. which schema describes the object. This field will only be explicitly populated where the object type cannot be determined from context (as it can, for instance, in a Pydantic data structure). The derived_from_id field serves only for provenance tracking and will be stored as such also in JSON data.

When implemented as a JSON document, the links are translated as contents, and the two-way links could go either way. A Job document could contain a sample field with LogisticalSample JSON, and lists of datasets for results, templates, input_data, and reference_data. However, you could also have a top-level Dataset, containing a source link with a Job, which in turn contained a ‘sample element, which contained a preparation element and so forth. In a JSON document you would always be free to omit linked-to elements (otherwise you would end up with huge networks of JSON objects.

The modelling here is heavily influenced by the need to support properly the acquisition and processing of multi-sweep experiments. In a model where you assume 1 Sample : 1 Sweep : 1 processing result, there is some sense in basing your modelling and your viewers on purely Datasets with attached Metadata. But with multi-sweep experiments you need to be able to record and view information that applies to the experiment as such (e.g. the strategy being followed) that cannot be attached to a single Sweep. Similarly multi-sweep processing, potentially producing several Datasets, cannot be accurately displayed if the only object you have available is the Dataset. Nor can you hang everything off Samples, once you admit the possibility of processing data jointly from multiple Samples. The introduction of the Job object gives you a place off which to hang information about the Experiments or the Calculations you are performing – which is likely what the users would be interested in – without having to hang it off Datasets. Arguably, viewers should be organised as list of Jobs.

The proposed combination of objects and links should be enough to allow full provenance tracking. The Job class connects the experiment/calculation and its various kinds of inputs to the output Datasets, and the LogisticalSample that the Job was performed on (for an Experiment). The Dataset.derived_from_id allows you to make modified Datasets (e,.g, for image selection) while keeping track of the provenance.

Dataset

Dataset		Abstract ‘Superschema’ for datasets
uuid	UUID	Globally unique identifier.
source	Job	Optional. Job that created the Dataset. return link for Job.results
role	str	Role of Dataset relative to the source Job. E.g. Result, Intermediate, Characterisation, Centring. Intended for filtering/selection of Datasets.
sample	LogisticalSample	Optional link. Sample relevant to Dataset, overrides Job.sample. return link for LogisticalSample.datasets
derived_from_id	UUID	Optional. UUID for Dataset from which this Dataset was derived. For modified Datasets without a source
extensions	dict	Unconstrained, schema-free additional parameters. Avoid where possible.
namespace_extensions	NAMESPACE_Dataset	Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL')

The Dataset is the superclass for any type of datasets, images, reflection files, models, maps, etc. The actual data files are stored separately, but the Dataset must contain the file pointers. Since the same data could be stored in a number of complex ways, e.g. as individual images, as a single multi-image files, or as part of a larger HDF5 data structure, the file specification must support any of these, including specifying subsets of larger data structures. A Dataset is characterised by its role, which describes it relative to the experiment or calculation that created it. For jobs that also create intermediate or secondary results, the Dataset(s) that are the intended output should have role of result.

The (optional) source link leads from the Dataset to the Job that created it, for tracking purposes. The derived_from_id provides an alternative provenance. It may be that the input to a calculation uses only part of a Dataset, e.g. after filtering out unwanted images. It may also be that you need to attach parameters to calculation input that belong neither to the Dataset as such, not to the calculation, but to the combination of the two, e.g. a weight parameter. In either case you can create a duplicate Dataset record with the data contents and parameters you need and use the derived_from_id for provenance tracking. In general, one would not duplicate the actual data files, but only the Dataset record pointing to them.

Job

Job		Abstract ‘Superschema’ for experiments and processing jobs
uuid	UUID	Globally unique identifier.
sample	LogisticalSample	Optional link. return link for LogisticalSample.jobs
templates	list(Dataset)	Templates with parameters for output datasets – e.g. diffraction plan, processing plan
input_data	list(Dataset)	Input data sets (pre-existing)
reference_data	list(Dataset)	Reference Datasets
results	list(Dataset)	Datasets produced by Job
start_time	Timestamp	Actual starting time for Experiment/Calculation
end_time	Timestamp	Actual finishing time for Experiment/Calculation
job_status	Enum	Status of job. E.g. Template, Ready, Running, Completed, Failed, Aborted
extensions	dict	Unconstrained, schema-free additional parameters. Avoid where possible.
namespace_extensions	NAMESPACE_Job	Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL')

Depending on the type, the Job could specify either an actual experiment, or a processing job or a calculation. Since a Job (or experiment) can have multiple input and output data sets, the Job, rather than the Dataset or Sample, is the natural unit for the user to look at in overviews. The data produced by a Job are found in the results link. Job input can be specified as input_data (to be processed), reference_data, and templates. Templates contain the same kind of Dataset records as the results (avoiding the need for separate ‘diffraction plan’ schemas) but contain input parameters and are used to handle e.g. diffraction plans and processing plans. Templates would often be sparsely populated and should not contain any links to actual files. SA regards teh job_status, ‘Template’ signifies a Job that is not ready for running, intended either for later filling in, or only to serve as a template to generate other jobs.

LogisticalSample

LogisticalSample		Abstract ‘Superschema’ For logistical samples (from Dewars and Plates to drops, pins or crystals
uuid	UUID	Globally unique identifier.
preparation	PreparedSample	optional link – The sample preparation that applies to this LogisticalSample and all its contents
container	LogisticalSample	optional link – LogisticalSample containing this one
contents	list(LogisticalSample)	optional link – LogisticalSample contained in this one
jobs	list(Job)	Jobs (templates, planned, initiated, or completed) for LogisticalSample.
datasets	list(Dataset)	Datasets (templates, planned, initiated, or completed) for LogisticalSample.
extensions	dict	Unconstrained, schema-free additional parameters. Avoid where possible.
namespace_extensions	NAMESPACE_LogisticalSample	Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL')

The LogisticalSample describes the kind of samples you ship, giving a nested series of containers from Shipment at the top down to crystals (mounted or in-situ) at the bottom of the hierarchy. The LogisticalSample can be linked to either Jobs or Datasets at any level. This is overkill, necessary to make the class generic, but depending on the experiment type you can have meaningful links to experiments not only from individual crystals, but also from drops, SSX or XFEL grids or streams, or multi-crystal loops; and in some experiment types (e.g. preparations of cubic lipid phase) you do not identify the individual crystals until partway through the experiment.

Note that diffraction plans or processing plans are specified by adding Job objects (with template Datasets) that contain the parameters you want to use for acquisition (including characterisation or centring if desired) or processing.

PreparedSample

PreparedSample		Abstract ‘Superschema’ for Sample preparations (describing sample content)
uuid	UUID	Globally unique identifier.
samples	List(LogisticalSample)	Logistical Samples with contents from this PreparedSample)
extensions	dict	Unconstrained, schema-free additional parameters. Avoid where possible.
namespace_extensions	NAMESPACE_SamplePreparation	Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL')

The PreparedSample describes the actual content of the Sample, as well as the identifiers that point back to the act of preparing it. The Sample could refer to multiple crystals, and in some cases (e.g. SSX) even to multiple grids or streams.

MX Acquisition

CrystallographyExperiment

This class is a type of Job; the schema contains information and results that applies to the experiment as a whole regardless of the number of sweeps. The templates (‘diffraction plan’) could contain either generic Sweep records (such as ‘Characterisation’ and ‘Acquisition’) or a complete list of Dataset metadata records, such as could be submitted to an acquisition queue.

The expected_resolution is the resolution that the user expects and wants to measure to; it is used to set up the experiment (including the detector distance). ‘required resolution’ (‘measure only if better than...’) is not catered for in the model as it stands.

CrystallographicExperiment		a type of Job
Object-level attributes
uuid	UUID	Globally unique identifier.
sample	LogisticalSample	Optional link. return link for LogisticalSample.jobs
templates	list(Dataset)	Templates with parameters for output datasets – e.g. diffraction plan, processing plan
input_data	list(Dataset)	Input data sets (pre-existing)
reference_data	list(Dataset)	Reference Datasets
results	list(Dataset)	Datasets produced by Job. Could be CollectionSweeps or mesh scans, images, ...
start_time	Timestamp	Actual starting time for Experiment/Calculation
end_time	Timestamp	Actual finishing time for Experiment/Calculation
job_status	Enum	Status of job. E.g. Template, Ready, Running, Completed, Failed, Aborted
extensions	dict	Unconstrained, schema-free additional parameters. Avoid where possible.
namespace_extensions	NAMESPACE_Job	Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL')
Input:
experiment_strategy	str	MXPressE, GPhL_Native_quick, GPhL_SAD_full, GPhL_2wvl_quick, ...
expected_resolution	float	The resolution expected in the experiment – for positioning the detector and setting up the experiment
target_completeness	float
target_multiplicity	float
dose_budget	float	MGy
snapshot_count
wedge_width	float	Input for interleaving strategy, actual scans are given explicitly in the CollectionSweep
Output:
measured_flux	float
radiation_dose	float	Total radiation dose over experiment
unit_cell	UnitCell	As determined during characterisation
space_group_name	Enum	Space group, as determined during characterisation. Names include alternative settings.

GPHL_CrystallographyExperiment

This is an example of an extension to CrystallographyExperiment, with attributes that are relevant to Global Phasing workflow experiments, but maybe not (yet) sufficiently general for the main schema.

GPHL_CrystallographicExperiment		Global-Phasing-specific extension to CrystallographicExperiment
characterisation_strategy	str	Characterisation strategy name – matches configured list
recentring_strategy	str	Recentring strategy name – matches configured list
crystal_class_names	list(Enum)	Output: list of possible arithmetic crystal class names, as determined during characterisation.

CollectionSweep

The CollectionSweep is a type of Dataset. The record is based on MX, but should work also for multi-crystal, and even SSX or XFEL experiments. In the latter two cases a lot of additional information would be needed.

CollectionSweep		a type of Dataset
Object-level attributes
uuid	UUID	Globally unique identifier.
source	Job	Optional. Job that created the Dataset. return link for Job.results
role	str	Role of CollectionSweep relative to the source Job. E.g. Result, Characterisation, Acquisition. Intended for filtering/selection of CollectionSweep.
sample	LogisticalSample	Optional link. Sample relevant to CollectionSweep, overrides Job.sample. return link for LogisticalSample.datasets
derived_from_id	UUID	Optional. UUID for CollectionSweep from which this CollectionSweep was derived. For modified Datasets without a source
extensions	dict	Unconstrained, schema-free additional parameters. Avoid where possible.
namespace_extensions	NAMESPACE_CollectionSweep	Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL')
General parameters:
annotation	str
Acquisition:
exposure_time	float
image_width	float	width of a single image, along scan_axis.
energy	float
transmission	%
Detector parameters:
detector_type	Enum	Use mmCIF Items/_diffrn_detector.type.html for enumeration
detector_distance	float	NB, we do NOT store resolution. The detector distance is clearly defined, the resolution is not.
detector_binning_mode	Enum
detector_roi_mode	Enum
beam_position	(float, float)	Position on detector, determines detector offset.
beam_size	(float, float)
beam_shape	Enum
Scan and position parameters:
axis_positions_start	dict	Starting position of all axes, rotations or translations, including detector distance, by name.
scan_axis	Enum	Name of main scanned axis, Enum should be standard names, e.g. Omega, Kappa, Phi, Chi, TwoTheta, SampleX, SampleY, SampleZ, DetectorX, DetectorY
overlap	float	Overlap between successive images. May be negative if images are not contiguous. Is this still necessary??
axis_positions_end	dict	Positions at end of sweep for all axes that are scanned. NB scans may be acquired out of order, so this determines the limits of the sweep, not the temporal start and end points.
num_triggers	int	Instructions for how to set up detector and acquisition – do not modify meaning of other parameters.
num_images_per_trigger	int	Ibid.
scans	list[Scan]	Subdivisions of CollectionSweep – NB need not be contiguous, or in order, or add up to entire sweep.
File parameters:		NB. This assumes that path+file_type+prefix+image_no allows you to get an image. Anything more we would need?
file_type	Enum	Enum should be: mini-cbf, imgCIF, FullCBF, HDF5, MarCCD
prefix	str	input parameter – used to build up the file name template
filename_template	str	output parameter, includes prefix, suffix, run number, and a slot where image number can be filled in.
path	str	Path to look for image file names

Note that the CollectionSweep specifies a single, continuous sweep range, with equidistant images given by image_width, and all starting motor positions in axis_position_start. axis_positions_end contain the end point of the sweep, and must have at least the value for the scan_axis; sweeps changing more than one motor (e.g. helical scan) can be represented by adding more values to axis_positions_end. The default number of images can be calculated from the sweep range and image_width. The actual number of images, the image numbering, and the order of acquisition (including interleaving) follows from the list of Scans. The role should be set to ‘Result’ for those sweeps that are deemed to be the desired result of the experiment; in templates you would prefer to use Acquisition for the Dataset that gives the acquisition parameters.

Scan

The Scan describes a continuously acquired set of images that forms a subset of the CollectionSweep of which they form part. The ordinal gives the acquisition order of sweeps across an entire multi-sweep experiment; this allows you to describe out-of-order acquisition and interleaving.

Scan		Subdivision of CollectionSweep
scan_position_start	float	Value of scan_axis for first image
first_image_no	Int	Image number to use for first image
num_images	Int
ordinal	Int	acquisition order within experiment (not just within sweep)

Samples

LogisticalSample

The LogisticalSample is just the abstract superclass, and the individual schemas will be as described in Ed Daniel’s Sample shipping proposal. The main difference is that the link to contents would be called contents at all levels, so that Shipment.contents’ replace the proposed Shipment.dewars and Shipment.plates. As well, of course, as there being some additional links at all levels, as described in the core model.

CrystallographicSample

PreparedSample is just a match to Ed Daniel’s Sample schema. There are a couple of proposed changes:

• `protein` and `ligands` would be replaced by the more general ‘macromolecule’ and ‘component’.
• The radiation_sensitivity is a sample parameter that is needed for (some) strategy programs. 

CrystallographicSample		a type of PreapredSample. Matches Sample
Object-level attributes
uuid	UUID	Globally unique identifier.
samples	list(LogisticalSample)	LogisticalSamples whose contents are from this SamplePreparation
extensions	dict	Unconstrained, schema-free additional parameters. Avoid where possible.
namespace_extensions	NAMESPACE_SamplePreparation	Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL')
General:
unit_cell	UnitCell	NB do we need alternative crystal forms?
space_group_name	Enum	Name of expected space group. Names include alternative settings.
radiation_sensitivity	float	Radiation sensitivity relative to standard crystal
Content:
macromolecule	Macromolecule	Macromolecule being investigated
components	list(Component)	other components used in the Sample liquid (e.g. binders, substrates, metal ions, additives, ...)
identifiers	List[Identifier]	Identifiers of this sample

Other schemas

Sample Contents

Macromolecule and Component are just different names for the Protein and Ligand schemas proposed by Ed Daniel.

Macromolecule		Matches Protein. Incomplete
acronym	str	Matches proteinAcronym
identifiers	List[Identifier]	Identifiers of this sample

Component		Matches Ligand. Incomplete.
identifiers	List[Identifier]	Identifiers of this sample

UnitCell
a	float
b	float
c	float
alpha	float
beta	float
gamma	float

MX Processing

The model for this part includes CrystallographicProcessing (a Job), ReflectionSet (a Dataset), ReflectionStatistics, and some low-level data structures (QualityFactor and Tensor). The input is based on the mmCIF categories reflns and reflns_shell (https://mmcif.wwpdb.org/dictionaries/mmcif_pdbx_v50.dic/Categories/) and the input specification to MRFANA. The idea is that values should be mapped directly to the relevant mmCIF fields, since these 1) are precisely defined, 2) will eventually be used for storing the data in the PDB, but the precise mapping is still to some extent to be determined. Using MRFANA input parameters to specify cutoffs etc. should make it possible to specify comparable data treatment for different programs and synchrotrons, since MRFANA is open-source and specifically designed to provide reproducible quality factors in different software environments.

CrystallographicProcessing

Unlike MX experiments, this Job actually does require input datasets, and this schema shows the need for multiple types of input data, to wit ‘input_data’ and ‘reference_data’. Other types of processing job might require more links at this level. When passed around as JSON documents input_data, reference_data and result would simply be included in the document – note that the schemas contain only the metadata, whereas the actual data live in the pointed-to files. Within a LIMS system these links would presumably be handled as foreign keys between tables, but that is up to individual implementations.

GPHL_CrystallographicProcessing is an example of a namespaced specific extension.

CrystallographicProcessing		a type of Job
Object-level attributes
uuid	UUID	Globally unique identifier.
sample	LogisticalSample	Optional link. return link for LogisticalSample.jobs
templates	list(Dataset)	Templates with parameters for output datasets – e.g. processing plan
input_data	list(Dataset)	Input data sets (pre-existing)
reference_data	list(Dataset)	Reference Datasets
results	list(Dataset)	Datasets produced by Job
start_time	Timestamp	Actual starting time for Experiment/Calculation
end_time	Timestamp	Actual finishing time for Experiment/Calculation
job_status	Enum	Status of job. E.g. Template, Ready, Running, Completed, Failed, Aborted
extensions	dict	Unconstrained, schema-free additional parameters. Avoid where possible.
namespace_extensions	NAMESPACE_Job	Extensions governed by registered schema starting with NAMESPACE (e.g. ‘GPHL')
Input parameters
unit_cell	UnitCell	Expected unit cell
space_group_name	Enum	Name of expected space group. Names include alternative settings.

GPHL_CrystallographicProcessing		Global-Phasing-specific extension to CrystallographicProcessing
crystal_class_names	list(Enum)	List of possible arithmetic crystal class names.

ReflectionSet

The ReflectionSet represents a set or processed, possibly merged or scaled reflections, as might be stored within an MTZ file or as mmCIF.refln.

ReflectionSet		a type of Dataset
Object-level attributes
uuid	UUID	Globally unique identifier.
source	Job	Optional. Job that created the ReflectionSet. return link for Job.results
role	str	Role of ReflectionSet relative to the source Job. E.g. Result, Intermediate. Intended for filtering/selection of Datasets.
sample	LogisticalSample	Optional link. Sample relevant to Dataset, overrides Job.sample. return link for LogisticalSample.datasets
derived_from_id	UUID	Optional. UUID for ReflectionSet from which this ReflectionSet was derived. For modified ReflectionSets without a source
extensions	dict	Unconstrained, schema-free additional parameters. Avoid where possible.
_extensions	_Dataset	Extensions governed by registered schema starting with (e.g. ‘GPHL')
General parameters:
anisotropic_diffraction	bool	Is diffraction limit analysis anisotropic (yes/no)
resolution_rings_detected	list((float,float))	Resolution rings detected as originating from ice, powder diffraction etc.
resolution_rings_excluded	list((float, float))	Resolution ranges excluded from calculation
unit_cell	UnitCell	Crystal unit cell determined
space_group_name	Enum	Name of expected space group. Names include alternative settings.
operational_resolution	float	Operational resolution, matching observed_criteria
Parameters matching mmCIF
B_iso_Wilson_estimate	float	reflns.B_iso_Wilson_estimate
h_index_range	(int,int)	lowest and highest h index in data – matches reflns.limit_h_*
k_index_range	(int,int)	lowest and highest k index in data – matches reflns.limit_k_*
l_index_range	(int,int)	lowest and highest l index in data – matches reflns.limit_l_*
num_reflections	int	Total number of reflections – matches WHICH mmCIF parameter?
num_unique_reflections	int	Total number of unique reflections – matches WHICH mmCIF parameter?
aniso_B_tensor	Tensor	reflns.pdbx_aniso_B_tensor
diffraction_limits_estimated	Tensor	reflns.pdbx_anisodiffraction_limit unit in A. Ellipsoid of observable reflections, regardless whether all have actually been observed.
overall_refln_statistics	ReflectionStatistics
refln_shells	list(ReflectionStatistics)
observed_criterion_sigma_F	float	Criterion for when a reflection counts as observed, as a multiple of sigma(F) – matches mmCIF reflns.observed_criterion_sigma_F
observed_criterion_sigma_I	float	Criterion for when a reflection counts as observed, as a multiple of sigma(I) – matches mmCIF reflns.observed_criterion_sigma_I
signal_type	Enum	local <I/sigmaI>’, ‘local wCC_half'; matches reflns.pdbx_signal_type. Criterion for observability, as used in mmCIF revln.pdbx_signal_status
signal_cutoff	float	Limiting value for signal calculation; matches reflns.pdbx_observed_signal_threshold. Cutoff for observability, as used in mmCIF refln.pdbx_signal_status
MRFANA input parameters		# NB the various cutoff criteria are taken from observed_criteria and observed_criteria_per_run
resolution_cutoffs	list(QualityFactor)	MRFANA resolution cutoff criteria
binning_mode	Enum	Binning mode. Enum is: equal_volume, equal_number, dstar_equidistant, dstar2_equidistant
number_bins	int	Number of equal volume bins (> 0)
refln_per_bin	int	Number of reflections per bin (> 0)
refln_per_bin_per_sweep	int	Number of reflections per bin per run
File parameters:		NB. This assumes a single file with a fixed name. May require expansion: Compressed nested directory? Multiple files? Multi-dataset files?
file_type	Enum
url
filename

• NOTE This is modelled so that attributes have been omitted in ReflectionSet whenever they could reasonably be put in the overall_refln_statistics slot instead. This is the case for reflns attributes d_resolution..., CChalf, pdbx_R..., pdbx_chi_squared, pdbx_redundancy..., number.... Will this work, and for which attributes?

• Note that there is no ‘resolution’ property; this is deliberate. The recommended procedure is to use operational resolution instead. If there is a need for a number approximating ‘the resolution of the data’, it is recommended to pick either the best resolution of the diffraction_limits_estimated tensor or the high-resolution limit of the overall_refln_statistics, whichever number is worse.

• The current draft still needs to be filled in relative to which fields map to exactly which mmCIF tags, and to how to represent the many possible completeness values (mmCIF pdbx_percent_possible... tags)

Other Schemas

ReflectionStatistics
resolution_limits	(float, float)	lower, higher resolution limit fo shell – matches d_res_high, d_res_low
number_observations	int	total number of observations, – matches WHICH mmCIF parameter?
number_unique_observations	int	total number of unique observations – matches WHICH mmCIF parameter?
quality_factors	list(QualityFactor)
completeness	%	# matches percent_posible_all (but what about ,,,_obs?)
chi_squared	float	Chi-squared statistic for resolution shell, matches pdbx_chi_squared
number_rejected_reflns	int	Number of rejected reflections, matches pdbx_rejects
redundancy	int	Redundancy of data collected in this shell – matches pdbx_redundancy
redundancy_anomalous	int	Redundancy of anomalous data collected in this shell, matches pdbx_redundancy_anomalous

QualityFactor
type	Enum	R(merge), R(meas), R(pim), I/SigI, CC(1/2), CC(ano), SigAno, Completeness, Redundancy
value	float

Tensor
eigenvalues	(float, float, float)	Tensor eigenvalues
eigenvectors	(float, float, float)	Tensor eigenvectors (unit vectors), in same order as eigenvalues

Object Model

Implementing this model (also) within a LIMS system would presumably require some kind of table structure. As discussions have been going until now, the preferred approach would be to have a minimum of tables and keep most of the data in metadata dictionaries, as done e.g. in MongoDB. Below one proposal for how the present model might be implemented in a limited number of tables. The diagram shows the foreign keys and intra-LIMS links that would be needed. Clearly the implementation would be up to the LIMS system programmers, but this should at least show one possibility.

The Job.results link is a simple one-to-many link, but the links between the Job and Dataset that describe Job input are the most complicated part of the model. Since the input_data, templates and reference_data links are all many-to-many, the JobInputLink class is required to represent them. The JobInputLink.role attribute would be either ‘input_data’, ‘reference_data’ or ‘templates’ in these cases. Alternatively Job.templates could be handled simply by not giving them a UUID and including them in the metadata – which would mean that each template could be used only once. If necessary, this structure could be used to handle additional kinds of links between Job and Dataset.

Points raised

Personal information

Some kind of information must be attached to a sample shipment to allow it to be connected to synchrotron LIMS systems, but any kind of personally identifiable information raises complex legal issues that differ from country to country. E.g. at ANSTO any such information would require clearing with the legal department, and was therefore best avoided. There is also the problem that there relevant information was likely to differ so much between synchrotrons that it would be difficult to make a general model. It is therefore proposed that information relevant for access control, contact persons etc. should be kept out of the main model, and should be stored as namespaced extensions specific to each synchrotron. This allows each site to specify its own requirements.

Diffraction plans

The idea is to use the same schemas for diffraction plan, for passing information to the execution queue, and for storing the description of the experiment once completed. The same would be the case for processing plans. The LogisticalSample would contain one or more Jobs, which in turn would contain Dataset descriptors – though without actual data attached. These Dataset descriptors could be attached to other Jobs as templates, and would contain input parameters as needed. In practice it is expected that the template Datasets shipped with samples might be quite thinly populated, with many parameters being set by workflow or from beamline defaults. For, e.g., a workflow-driven multi-sweep acquisition you would specify the strategy to follow, and a single Dataset with acquisition parameters (plus possibly one with characterisation parameters). The strategy would then determine the character and number of sweeps, and the parameters of those sweeps would be set from the single template, from beamline defaults, or by the workflow program.