Schemas for contained resources are all jumbled together - can we do better?

[mikix]

Right now, all contained resources are just equal members of the contained array.

And so when we write to an output target that expects a schema (like Delta Lake), the ETL ends up creating a Frankenstein schema of the union of all contained resources it finds in the parent resource.

In practice, this doesn't seem to be a high priority issue for two reasons:

• Most EHRs don't stuff many different kinds of contained resources in the same parent resource. i.e. MedicationRequest might contain Medication but is unlikely to also hold Patient
• Most FHIR fields with the same name but on different resource don't have different types (do they ever?)

Still, this is an odd situation that could potentially cause trouble in the future.

Brainstormed solutions:

• Modify incoming data and separate out the contained resources into separate lists like containedMedications. But that feels like a whole can of worms as well. Doing it in a way that still writes out valid FHIR would be tricky, because while we can stuff our new arrays into an extension, the internal ids would still expect to be found in the main contained list. Maybe a modifierExtension would let us avoid that.
• Write out new tables for contained resources, like medicationrequest_contained_medication or similar. Then you'd have to change the referencing ID in a way that consumers would have to know which table to go look up. And you'd have to handle the contained resource changing IDs / being dropped. Hmm. This all kind of sucks.
• Move contained resources into the external table where they belong. That might breaks a few current assumptions of the ETL currently (i.e. that running task A will only modify a set of known tables) and would require inventing an ID scheme that wouldn't conflict with any existing IDs. And you'd still have to handle contained IDs changing / being dropped.

Something to think about.