Referential Integrity

[codyebberson]

Referential Integrity

This document describes current state and future plans for "Referential Integrity" in Medplum FHIR Server. Referential integrity is an optional feature that enforces valid FHIR References.

Given the deeply nested structure of FHIR resources, this is a non-trivial problem to implement fully and within reasonable performance constraints.

What is Referential Integrity?

• a property of data such that all references are valid ("broken links")
• a property of a system such that it ensures that all references are to data that actually exists

In the FHIR specification:

• Update: "Servers are permitted to reject update interactions because of integrity concerns or other business rules"
• Delete: "If the server refuses to delete a resource because of reasons specific to that resource, such as referential integrity, it should return the 409 Conflict status code. Note that the servers MAY choose to enforce business rules regarding deletion of resources that are being referenced by other resources, but they also MAY NOT do so."
• "Both client and server systems SHOULD clearly document how transaction integrity is handled, in the documentation in their CapabilityStatement."

Why not enforce Referential Integrity?

• Enforcing Referential integrity is a natural and necessary feature of Enterprise Information Systems
  • Primary sources of information
• But:
  • Security / Access Control: client may not have access
  • Secondary sinks (CDR) may get partial records
  • Asynchronous processes - things may arrive out of order
  • Legacy Data - Broken business processes (paper)
  • Archive trails

Goals

The goal of the Medplum server is to provide the features necessary to power complex workflows and complex organizations. To that end, we want to provide optional referential integrity features as necessary for customers.

Features

We have identified the following sub projects on the path toward full implementation:

• Enforce on write (create or update)
• Enforce on delete
• Rewrite display strings

Enforce on Write

On all write operations (create, update, patch, etc), reference targets will be checked. For example, if a patient contains a reference to managing organization Organization/FOO but FOO is not a valid ID for an organization on the server, then the operation will be blocked.

Status: Deployed

Enforce on Delete

On delete operation, the server will check for any inbound references. Resources can only be deleted if there are no other resources with references to the candidate resource for deletion.

A distinction can be made between "all references" and "indexed references":

• "All references" refers to the full set of all FHIR references, including inside non-indexed properties and extensions.
• "Indexed references" refers only to FHIR references that appear in SearchParameter expressions.

Our review of publicly available FHIR server documentation indicates that most other servers only check "Indexed references".

For example:

• A Patient resource referenced by ServiceRequest.subject would be checked, because the "subject" search parameter includes "subject" in the expression.
• A Location resource referenced by ServiceRequest.locationReference would not be checked, because there are no SearchParameters that include locationReference in the expression.

To implement this in a performant manner, additional preprocessing may be required at write-time. For example, an additional table for references indexed by target may be required.

Status: Not started

Rewrite Display Strings

FHIR Reference objects have an optional "display" property, which can be convenient in the UI layer. Ideally, the display property would automatically update to reflect changes in the target resource.

For example, if a ServiceRequest.subject refers to a Patient, and the Patient name changes, the ServiceRequest.subject.display property will not be updated to reflect the change.

This is unspecified behavior, and includes opinionated business logic (i.e., "what string should be used for the display?"). Medplum includes a "createReference()" utility which attempts to intelligently generate a display string, but advanced users may want to use their own logic.

Status: Not started

References:

• FHIR References: https://www.hl7.org/fhir/references.html
• Grahame Grieve Referential Integrity: https://www.devdays.com/wp-content/uploads/2021/12/DD21US_20210607_Grahame_Grieve_Referential_Integrity.pdf
• Smile CDR Referential Integrity: https://smilecdr.com/docs/fhir_repository/creating_data.html#validating-references-and-referential-integrity
• Google FHIR Referential Integrity: https://cloud.google.com/healthcare-api/docs/concepts/fhir-referential-integrity
• AWS Labs Referential Integrity: Maintain referential integrity for creates awslabs/fhir-works-on-aws-persistence-ddb#19