v10.0.0

This release adds terminology in SQL, which returns FHIR value sets and concept maps as relations that Spark SQL can join to, extends local terminology mode to large concept maps and SNOMED CT simple maps, and aligns several FHIRPath functions with their specifications, including breaking changes to memberOf and getReferenceKey().

Breaking changes

• memberOf now follows the FHIR definition #2854. It returns empty where the input has more than one item, and accepts code and string input, inferring the system from the value set. Where the value set cannot be resolved or the terminology server rejects the request, membership is unknown: empty in FHIRPath and null from the member_of UDF.
• getReferenceKey() now returns empty for absolute URLs, contained references and URNs, which can never equal a [type]/[id] resource key #2808. Only relative references, with an optional /_history/[version] suffix, produce a key. Contributed by @developer-rpai in #2812.
• The local terminology store format version is now 2 #2815. A store written by 9.9.0 that holds ConceptMaps is migrated in place when it is first opened in local mode #2858. This requires write access to the store, and the migrated store can no longer be read by 9.9.0.

New features

Terminology in SQL

PathlingContext can now return the content of a value set or concept map as a DataFrame with the relations defined by the SQL on FHIR Terminology in SQL page, so a query written against that specification runs unchanged once each relation is registered as a view. Content is obtained from either a terminology server or the local terminology store.

• Java and Scala: valueSet, valueSetFromJson, conceptMap and conceptMapFromJson.
• Python: value_set, value_set_from_json, concept_map and concept_map_from_json.
• R: pathling_value_set, pathling_value_set_from_json, pathling_concept_map and pathling_concept_map_from_json.

This builds on value set expansion #2772 and concept map reading #2783, now part of the terminology service. See the terminology in SQL documentation for details.

In local mode, a SNOMED CT implicit concept map can be read only for the four association reference sets (POSSIBLY EQUIVALENT TO, REPLACED BY, SAME AS and ALTERNATIVE). A URL naming any other reference set, such as a simple map that translate accepts, is rejected with a message giving that reason #2859.

Local terminology mode

• Large ConceptMaps are now imported and translated within bounded memory, with one row per mapping in a new concept_mapping table and maps loaded on first use #2815.
• Local translate now supports SNOMED CT simple map reference sets, with the target system and equivalence that Ontoserver reports for the ICD-O, CTV3 and ARTG maps #2779. SNOMED CT versions imported by 9.9.0 did not store simple map targets and must be re-imported to translate through them; a warning naming each affected version is logged until they are.
• FHIR packages are verified against the checksum published by the package registry on import, and the import is refused on a mismatch. The source SHA-256, package identity and verification status are recorded in the store manifest #2744.
• FHIR terminology packages are imported with one write per table, so importing a package of many small resources such as hl7.fhir.r4.core is much faster #2849.

Other improvements

• Added the startsWith(), endsWith() and contains() FHIRPath string functions #2380. Contributed by @chgl in #2637.
• PathlingContext.create() in Python accepts a spark_conf dictionary, matching the --spark-conf option of the command line interface #2669. Contributed by @developer-rpai in #2754.

Bug fixes

• getReferenceKey() now strips the /_history version, so versioned references join #2797.
• select(), nested forEach and repeat no longer lose the whole result when one item's projection is empty #2801.
• Navigation to an element beyond the encoded nesting depth, such as a deeply nested Questionnaire.item, now returns empty rather than failing with FIELD_NOT_FOUND, so that functions, forEach and columns behave as they would for absent data #2625. select() no longer counts an empty projection as an item, and typed empty results now have the same column types as decimal and base64Binary values.
• toString() on a Coding now returns the Coding literal #2802.
• Conversion functions now treat only a FHIR Quantity with a value, a UCUM or calendar duration code and no comparator as a System.Quantity #2803.
• The indexer no longer counts negative indexes from the end, and a computed index such as count() - 1 no longer fails #2805.
• Equality and ordering involving a Quantity with a comparator now return empty #2806.
• In local terminology mode, a SNOMED CT concept in or not-in value set filter now names reference sets, as defined by FHIR, and comma-separated filter values are read as a code list #2851.
• In local terminology mode, a SNOMED CT edition URI without a version, such as http://snomed.info/sct/900000000000207008, now resolves to the latest stored version of that edition. This applies to implicit value sets and concept maps, VCL and Coding.version #2782. Experimental http://snomed.info/xsct URIs are now also resolved as SNOMED CT content.
• Local translate now returns every target of a SNOMED CT association #2777, and reports the relationship each reference set defines rather than equal #2778.
• Writing Parquet or NDJSON to a directory that already holds files no longer discards the write. Files are now named <type>.<partId>-<writeId>.<ext> #2823.
• The ViewDefinition profile is no longer added to meta.profile when encoding #2799.
• The command line interface's --verbose flag now surfaces Pathling's library logging #2691. Contributed by @developer-rpai in #2756.

Dependencies

• Upgraded Delta Lake to 4.0.1, fixing CONVERT TO DELTA by path on Spark 4.0.2 #2824.
• Upgraded Jackson to 2.22.3 to address published vulnerabilities.

Full Changelog: v9.9.0...v10.0.0