Add MassSpectrometryConfiguration and Configuration classes #130

kheal · 2024-04-12T19:28:35Z

This PR addresses microbiomedata#1913.

As currently modeled, each MassSpectrometry instance (currently applicable to about 2000 process samples), correspond to only 4 configurations in which the mass spectrometry instrument was actually used. We propose to have a MassSpectrometryConfiguration class that houses the configuration information which would 1) greatly reduce repeating data on MassSpectrometry instances and 2) allow for workflows to be configured based on MassSpectrometryConfiguration instance and 3) allow users to query samples that were run with similar configurations for downstream comparisons.

This would move several slots from MassSpectrometry to MassSpectrometryConfiguration and associate the configuration through a new has_configuration slot (see below for diagram).

Note that all slots inherited from PlannedProcess (like start_date, has_input) will remain on MassSpectrometry

This same configuration abstraction pertains to ChromatographicSeparationProcess (see microbiomedata#1920), so we've also created an abstract Configuration parent class.

and has_configuration slot

turbomam · 2024-04-12T19:44:17Z

Presumably we will need a configuration_set collection. It would be good to include that in this PR, along with an example data file that instantiates a Database and includes a DataGeneration and a Configuration

turbomam · 2024-04-12T19:45:16Z

src/schema/nmdc.yaml

+  Configuration:
+    abstract: true
+    class_uri: nmdc:Configuration
+    description: A platform that enables the user to configure the appearance, actions, and other usage preferences on a process.


In what way is a configuration a "platform"?

Generally we were thinking a computer platform. We struggled with making this (and associated) descriptions and are open to other suggestions. Maybe "A framework that enables...."?

I've updated the definition of the Configuration class to "A programmable set of parameters that define the actions of a process and is shared among multiple instances of the process.". I think that addresses some of the outstanding issues regarding understanding when we would use this class to hold configuration information.

In the case of MassSpectrometryConfiguration, this would be 1:1 with a method file that we use to run the mass spectrometer.

turbomam · 2024-04-12T19:46:01Z

src/schema/nmdc.yaml

@@ -686,6 +704,12 @@ slots:
    range: FluidHandling
    description: how a processed sample is introduced into a mass spectrometer, for example liquid chromatography or direct infusion through a syringe

+  has_configuration:
+    any_of:


Why would the range be Configuration in some cases and DataObject in other cases?

We should go to great lengths to avoid any_of ranges. They are complex to reason over and diagram. That may change in the future.

@brynnz22 and I were thinking ahead to accommodate workflow configuration files (DataObjects) that will be associated with a has_configuration slot on WorkflowExcecution class. See microbiomedata#1912 .

I would like to make sure it's very clear when we

embed configuration-like data in an instance of a PlannedProcess

associate configuration-like data with a process via a DataObject, which is a pointer to a configuration file (but doesn't actually manifest the settings in the instance)

embed configuration-like data in instances of the new Configuration class

A review of those configuration embedding/linking options may shine some additional light on the idea of a configurations as platforms

Sorry I do not quite understand this.

I would like to make sure it's very clear we we

embed configuration-like data in an instance of a PlannedProcess

associate configuration-like data with a process via a DataObject, which is a pointer to a configuration file (but doesn't actually manifest the settings in the instance)

embed configuration-like data in instances of the new Configuration class

Are there actions we can take in the PR to meet this?

Since this PR adds a new way of capturing configuration, it is required that this PR includes documentation (or additional schema constraints) on on how it relates to existing ways of capturing configuration information.

kheal · 2024-04-12T20:10:42Z

Presumably we will need a configuration_set collection. It would be good to include that in this PR, along with an example data file that instantiates a Database and includes a DataGeneration and a Configuration

We have a few following issues related to configurations (microbiomedata#1912 and microbiomedata#1920), and we'll plan to make a configuration_set with one of those, would that work?

turbomam · 2024-04-12T22:04:52Z

@cmungall @sierra-moxon and I just started discussing this today.

This PR has a lot of good features, like a small size, inclusion of examples and the abstract Configuration parent class. @cmungall is in favor of merging small PRs like this quickly, but this one will need some refactoring and or better documentation.

We can help more if

links to real-world data are provided, like some of the 2000 process samples
modeling is kept as simple as possible. any_of ranges (unions), rules and slot_usages should be kept to an absolute minimum. Use of those features can have dramatic impact on the ability to use LinkML-related tools, which may not be readily apparent in the existing tests.
the bodies of issues or PRs include a justification for the change and a holistic review of existing modeling in the schema. The justification is pretty good here, but there are other issues from this squad that are just a title.

turbomam · 2024-04-12T22:11:51Z

If it's currently necessary for processes to indicate that they were configured by either a Configuration instance or a DataObject instance, @cmungall suggested using two different slots, like has_configuration and configuration_data_object. If they aren't really both needed now, let's just use one slot and one range.

turbomam · 2024-04-12T22:12:48Z

We will also have to think about the life cycle of Configuration instances in MongoDB. Who gets to add them? How? Who is responsible for ensuring that we don't have duplicates?

kheal · 2024-04-12T23:22:45Z

links to real-world data are provided, like some of the 2000 process samples

I can work on this using the runtime API, but just as a simple demonstrative, all the Natural Organic Matter process samples that have resulted in data objects currently available in the data portal (aka FT ICR-MS analysis results as shown below) would share a single MassSpectrometryConfiguration object. In the data portal, I'm seeing this as 2,549 DataObjects each originating from their own DataGeneration instance.

also make a better definition for Configuration and related slots

kheal · 2024-04-18T22:07:13Z

We will also have to think about the life cycle of Configuration instances in MongoDB. Who gets to add them? How? Who is responsible for ensuring that we don't have duplicates?

I think this is outside of the scope of the modeling at this moment, but MassSpectrometryConfigurations should be easy to generate from DMS (actually easier than generating MassSpectrometry instances, at the moment). Naming a Configuration instance would help with dealing with duplicates and deciding if we need to add an additional I've added an inheritance of NamedThing to Configuration to facilitate this.

expand comments on Configuration and similar

kheal · 2024-05-01T18:59:54Z

I've made the following updates after PRs #142 and #141

Used updated slot names and update examples to multivalued (after Edit slots and usages to accommodate multiple detection configurations for single MassSpectrometry instance #141)
Made abstract Configuration class inherit from InformationObject class.

@turbomam, this is ready for re-review.

@SamuelPurvine, please take a look, especially the example file https://github.com/microbiomedata/berkeley-schema-fy24/blob/mass_spec_config/src/data/valid/Database-mass-spectrometry.yaml which holds two example DataGeneration (specifically MassSpectrometry) instances with their respective MassSpectrometryConfigurations. In practice, we'll likely load many more MassSpectrometry instances paired with very few MassSpectrometryConfigurations.

kheal · 2024-05-08T18:31:59Z

@SamuelPurvine gave me his review via teams, I've incorporated these changes into the https://github.com/microbiomedata/berkeley-schema-fy24/blob/mass_spec_config/src/data/valid/Database-mass-spectrometry.yaml.

turbomam · 2024-05-13T14:27:12Z

src/data/valid/MassSpectrometry-NOM.yaml

+has_output:
+  - nmdc:dobj-00-9n9n9n
+has_configuration: nmdc:mscon-99-oW43DzG0
+eluent_introduction: nmdc:cspro-99-hello00


do you know if we have any example data file that populate an eluant introduction instance? Is that still going to be relevant after you do some chromatography configuration modeling?

Yes, this will still be relevant. I will provide an extend these MassSpectrometry examples alongside the parallel work I've been doing for the ChromatographicSeparationConfiguration work.

turbomam · 2024-05-13T14:29:09Z

src/schema/nmdc.yaml

@@ -727,6 +754,10 @@ slots:
    range: FluidHandling
    description: how a processed sample is introduced into a mass spectrometer, for example liquid chromatography or direct infusion through a syringe

+  has_configuration:
+    range: Configuration
+    description: a set of parameters that define the actions of the process


I am going to merge this but make a PR afterwards, with you as an approver, to change this description. As a matter of principle, we can't use near-identical descriptions for has_configuration and and Configuration

Sounds good, thanks. I could definitely use support in proper/ontologically correct descriptions.

kheal added 7 commits April 11, 2024 15:36

Add draft class for MassSpectrometryConfiguration

f2c2283

Add slot description for has_configuration

54218f8

Merge branch 'main' into 'mass_spec_config'

e643926

Modify example MassSpectrometry data files

89e8f29

Fix MassSpectrometry example data

a601242

Create abstract Configuration class

3aa04de

and has_configuration slot

Modify configuration slot and class descriptions

d635e94

kheal self-assigned this Apr 12, 2024

kheal requested review from turbomam and removed request for turbomam April 12, 2024 19:33

kheal marked this pull request as ready for review April 12, 2024 19:38

kheal requested a review from turbomam April 12, 2024 19:38

turbomam reviewed Apr 12, 2024

View reviewed changes

kheal mentioned this pull request Apr 12, 2024

Add ChromatographicSeparationConfiguration class #131

Closed

Limit range of has_configuration slot

6ff6f83

kheal mentioned this pull request Apr 15, 2024

Add permissible values to FileTypeEnum and DataCategoryEnum to accomodate workflow configuration DataObjects microbiomedata/nmdc-schema#1912

Closed

kheal added 3 commits April 17, 2024 10:32

Merge branch 'main' into 'mass_spec_config'

243caa6

Make Configuration a NamedThing, require name for Configuration

4830c6b

also make a better definition for Configuration and related slots

Revise description of Configuration class and associated slots

cc17e64

kheal added 4 commits April 19, 2024 10:04

Add Database example and configuration_set

cd2dc34

Add configuration_set Database example

f44d676

expand comments on Configuration and similar

Fix typo

7a45052

Merge 'main' into 'mass_spec_config'

ea32bd9

kheal added 2 commits April 19, 2024 10:26

Change comments to notes

2f9fcb9

Update definition of has_configuration slot

f213ed6

kheal mentioned this pull request Apr 25, 2024

Edit slots and usages to accommodate multiple detection configurations for single MassSpectrometry instance #141

Merged

kheal added 3 commits May 1, 2024 10:38

Merge branch 'main' into mass_spec_config

793229c

Modify slots on MassSpecConfig after merging in main

b1398b8

Add inheritance to Configuration abc

c578538

kheal requested review from turbomam and SamuelPurvine May 1, 2024 18:59

kheal added 4 commits May 7, 2024 14:10

Remove alternate identifiers from MassSpectrometry example

45afe15

Revise Database mass spectrometry example

8df4343

Improve database mass spec example

84f4cf8

Revise database mass spectrometry yaml

f6d5010

turbomam reviewed May 13, 2024

View reviewed changes

turbomam approved these changes May 13, 2024

View reviewed changes

turbomam merged commit 36b1916 into main May 13, 2024
2 checks passed

kheal deleted the mass_spec_config branch May 14, 2024 14:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add MassSpectrometryConfiguration and Configuration classes #130

Add MassSpectrometryConfiguration and Configuration classes #130

kheal commented Apr 12, 2024 •

edited

Loading

turbomam commented Apr 12, 2024 •

edited

Loading

turbomam Apr 12, 2024

kheal Apr 12, 2024

kheal Apr 18, 2024 •

edited

Loading

kheal Apr 18, 2024

turbomam Apr 12, 2024

turbomam Apr 12, 2024

kheal Apr 12, 2024

turbomam Apr 12, 2024 •

edited

Loading

turbomam Apr 12, 2024

kheal Apr 12, 2024

turbomam Apr 12, 2024

kheal commented Apr 12, 2024

turbomam commented Apr 12, 2024

turbomam commented Apr 12, 2024

turbomam commented Apr 12, 2024

kheal commented Apr 12, 2024

kheal commented Apr 18, 2024

kheal commented May 1, 2024

kheal commented May 8, 2024

turbomam May 13, 2024

kheal May 13, 2024

turbomam May 13, 2024

kheal May 13, 2024

Add MassSpectrometryConfiguration and Configuration classes #130

Add MassSpectrometryConfiguration and Configuration classes #130

Conversation

kheal commented Apr 12, 2024 • edited Loading

turbomam commented Apr 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kheal Apr 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

turbomam Apr 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kheal commented Apr 12, 2024

turbomam commented Apr 12, 2024

turbomam commented Apr 12, 2024

turbomam commented Apr 12, 2024

kheal commented Apr 12, 2024

kheal commented Apr 18, 2024

kheal commented May 1, 2024

kheal commented May 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kheal commented Apr 12, 2024 •

edited

Loading

turbomam commented Apr 12, 2024 •

edited

Loading

kheal Apr 18, 2024 •

edited

Loading

turbomam Apr 12, 2024 •

edited

Loading