value_as_concept: Stricter validations for better OMOP mappings

[AndrewThien]

Problem:

Users would like to have the value_as_concept_id column (which is/will be useful for the long-term vision, where value can be standardised as concept, instead of number/string) in OMOP table observation and measurement populated long time ago (mentioned in the issue: Health-Informatics-UoN/carrot-mapper#619 and Health-Informatics-UoN/carrot-mapper#1008, as well as in Carrot Community meeting).
But we couldn't solve this due to many reasons:

• The current version of the JSON mapping rule didn't have connections between Field (which is currently mappings for all children values, this needs re-thinking) and Value mappings
• UoN didn't have a 100% control over the end ETL tool (Carrot CDM in the past)
• No actual QA/validations on field/value mappings that help to establish the value_as_concept_id or pair mappings (one value can receive both measurement_concept_id and value_as_concept_id, for example)

Solution:

Since the infrastructure to tackle this issue is now established:

• We have Carrot Transform, in-house, and we are talking a grip of it
• We are building/publishing V2 of mapping rules from Mapper (Version 2 of Mapping rules JSON carrot-mapper#1117) and ingesting them in Transform (Support for Mapping rules JSON V2 from Mapper carrot-transform#93)

We have the opportunity to actually solve this long-standing issue.

We have been thinking about a set of validations that can support/solve the issue in Mapper, then populate the field value_as_concept_id in the table observation and measurement in Transform, and I would like to document/propose them here for all parties to have a good understanding/discussion/agreement about what will be done/applied in the future.

Rethink the purpose of the field level mappings

Field level mappings have been used as mappings for all of children values. For example, I have a field Gender and values F/M in that field. If I add a concept Gender at the field level, then values F and M each will have a record with same observation_concept_id = 4135376 in the OMOP table observation. Seems to be reasonable...
However, the problem will come when a user then adds more concepts at the value level for M and F. For example, I can add Male to M, the value M will have another record in the table person with gender_concept_id = 8507. M now have a record in observation and in person. Is that needed...?

So, we are proposing to rethink the purpose of field mapping:

• As far as I am concerned, only two tables in OMOP have records stored as attribute-value pairs, i.e., observation and measurement. So, the field mapping's main purpose in the future will be providing the attribute for a record/its children values, i.e., observation_concept_id, measurement_concept_id, not a convenient way to map all children values anymore.
• Another purpose of field mapping will be indicating the fallback regarding value_as_concept / value_as_number / value_as_string. This will be detailed below.

Future validations set:

New validation 1:

• Since only observation and measurement have attribute-value pairs of records (there may be more, please add your comments), we only allow adding concepts at the field level if users want to make the values/concepts to be stored at the mentioned table. In other words, we will block field mapping.
• To unlock this feature, users would need to go Edit field, then choose either Measurement/Observation in the dropdown.
• The UI/UX design for this can be discussed later, but the general idea is: To have field mapping, users need to let Mapper know in which "special way" (details below) the mapping rules in this field should be treated, by indicating the target table (either Measurement or Observation).

For MANUAL mappings:

New validation 2: If the target table is Observation:

Mapping mode	Condition	Result
only add concepts at field level	- Concepts have to be in the valid domains (1) - Disallow multiple mappings for one field (2)	- All children's values have the same observation_concept_id (id of the added concept) - All children's values have their value as value_as_number/string based on the datatype
only add concepts at value level	Disallowed (3)	NA
add concept at field and value level	- Concepts have to be in the valid domains (1) - Disallow multiple mappings (2)	- All children's values have the same observation_concept_id (id of the added concept at the field level) - The value having its own concept added will have this concept's id in the column value_as_concept_id - The value doesn't have any concept added will have their value as value_as_number/string based on the datatype

New validation 3: If the target table is Measurement:

Mapping mode	Condition	Result
only add concepts at field level	- Concepts have to be in the Measurement domain - Disallow multiple mappings for one field (2)	- All children's values have the same measurement_concept_id (id of the added concept) - All children's values have their value as value_as_number
only add concepts at value level	Disallowed (3)	NA
add concept at field and value level	- Concepts at the field level have to be Measurement domain - Concepts at the value level have to be Meas value domain - Disallow multiple mappings (2)	- All children's values have the same measurement_concept_id (id of the added concept at the field level) - The value having its own concept added will have this concept's id in the column value_as_concept_id - The value doesn't have any concept added will have their value as value_as_number

For AUTO mappings:

New validation 4:

• If one field has auto-mapping concepts tied at value level (V concepts - built from source vocab indicated by Data dictionary), then disallow field mapping for that field. Because the measurement_concept_id or observation_concept_id can be determined by the Maps to relationship of the coded value already. For other tables, rather than measurement and observation, <table>_concept_id can be determined by the Maps to relationship normally (and we don't allow field mapping for these tables anyway)
• In case a coded value has both Maps to and Maps to value relationship:
  • We will use the domain of the concept_2 in Maps_to to determine the target table and to have the <table>_concept_id
  • We will put the concept_2 of Maps to value as value_as_concept_id for the record of the coded value

Questions for auto-mapping:

1. There are some concepts having Maps to and Maps to value rels. and the concept_id_2 of the rel. Maps to that have the domain not Measurement or Observation, for example: https://athena.ohdsi.org/search-terms/terms/37615975. What should we do in this case? Cause tables, other than Measurement or Observation, don't have the field value_as_concept_id.
2. There are some concepts having two Maps to (concept_id_2 of these rels are in different domains) and one Maps to value rels, for example: https://athena.ohdsi.org/search-terms/terms/45568009. What should we do in this case?

Additional notes:
(1): Valid domains are the domains that are not Condition, Procedure, Drug, Device, Specimen, or Measurement domains
(2): To guarantee each record/pair have one attribute and one value, avoiding the mixed-and-matched scenario. For the scenario of field mapping only, I wondered if we should allow multiple mappings for one field? Opinions here would help.
(3): observation_concept_id is required but value_as_* is not required. So it doesn't make sense to allow only value mappings for this table

All of the content/notes/ideas in this discussion were written based on my OMOP CDM readings and little experience working with OMOP.

All opinions/comments are welcome!