Verify TERMINOLOGY_SERVER_URL wiring on the 3 deployed test servers after next release

[smunini]

Context

PR #262 removes the built-in default terminology server (#217). After that lands, an unset terminology URL is not a fallback — it is a hard failure: %terminologies, memberOf(), subsumes() and the :in / :not-in search modifiers all error out (the modifiers return 501 Not Implemented).

The three deployed test servers had no terminology variable set at all. The only FHIRPATH_TERMINOLOGY_SERVER in ci.yml was on the Test FHIRPath job (a CI-only env var that never reached a deploy target).

This was fixed in the deploy job by adding a single TERMINOLOGY_SERVER_URL: https://hts.heliossoftware.com and threading it into each service's SERVICE_ENV. Note there is no single variable name that covers all three — each binary reads its own, and sof/hfs propagate theirs into the embedded FHIRPath engine:

Deploy	Variable the binary reads	Propagates to
fhirpath.heliossoftware.com	FHIRPATH_TERMINOLOGY_SERVER	—
sof.heliossoftware.com	SOF_TERMINOLOGY_SERVER	FHIRPATH_TERMINOLOGY_SERVER
hfs.heliossoftware.com	HFS_TERMINOLOGY_SERVER	FHIRPATH_TERMINOLOGY_SERVER

Setting HFS_TERMINOLOGY_SERVER on all three would be a silent no-op on two of them.

Why this needs a manual check after the next release

The deploy job rewrites each systemd unit, so in principle this applies itself on the next tagged release. It still needs eyes on it because:

1. The change has never executed — the deploy job only runs on refs/tags/v*, so it is unverified until a real release.
2. A wrong or missing value fails at runtime, not at deploy time. The deploy goes green either way.
3. hts deploys in the same matrix with fail-fast: false, so all four servers deploy in parallel — and the hts deploy resets hts.db, triggering a bootstrap re-import on startup. The three clients may briefly point at an HTS that is mid-restart.

TODO after the next release

• Confirm each systemd unit actually carries its variable (systemctl show -p Environment <service>)
• Run the validation script below against all three hosts
• If HTS was mid-bootstrap during the parallel deploy, re-run after it settles
• Decide whether the parallel-deploy race deserves a needs: ordering or a post-deploy HTS health gate

Validation script

Endpoints below match the actual routes: POST / on fhirpath-server (crates/fhirpath/src/server.rs:269), POST /$viewdefinition-run on sof-server (crates/sof/src/server.rs:358). HTS serves operations at the root of its base URL — no /r4 or /r5 segment.

#!/usr/bin/env bash
# Verify terminology wiring on the three deployed test servers after a release.
# Exits non-zero if any check fails.
set -uo pipefail

HTS=${HTS:-https://hts.heliossoftware.com}
FHIRPATH=${FHIRPATH:-https://fhirpath.heliossoftware.com}
SOF=${SOF:-https://sof.heliossoftware.com}
HFS=${HFS:-https://hfs.heliossoftware.com}
fail=0

check() { # name, condition-output, expected-substring
  if printf '%s' "$2" | grep -q "$3"; then
    echo "  PASS  $1"
  else
    echo "  FAIL  $1"
    echo "        expected to find: $3"
    echo "        got: $(printf '%s' "$2" | head -c 300)"
    fail=1
  fi
}

echo "== 0. HTS reachable (everything below depends on it) =="
meta=$(curl -sS --max-time 20 -H 'Accept: application/fhir+json' "$HTS/metadata")
check "HTS /metadata returns a CapabilityStatement" "$meta" 'CapabilityStatement'

vc=$(curl -sS --max-time 20 -H 'Accept: application/fhir+json' \
  "$HTS/CodeSystem/\$validate-code?url=http://hl7.org/fhir/administrative-gender&code=male")
check "HTS \$validate-code resolves at root (no /r4 segment)" "$vc" '"valueString":"Male"'

echo "== 1. fhirpath-server: memberOf() via FHIRPATH_TERMINOLOGY_SERVER =="
# memberOf() has no local fallback; without a configured server this errors.
fp=$(curl -sS --max-time 30 -X POST "$FHIRPATH/" \
  -H 'Content-Type: application/fhir+json' \
  -d '{
    "resourceType": "Parameters",
    "parameter": [
      {"name": "expression", "valueString": "Patient.gender.memberOf('\''http://hl7.org/fhir/ValueSet/administrative-gender'\'')"},
      {"name": "resource", "resource": {"resourceType": "Patient", "id": "tx-check", "gender": "male"}}
    ]
  }')
check "memberOf() evaluates to true" "$fp" '"valueBoolean":true'
check "memberOf() did not report a missing tx server" \
      "$(printf '%s' "$fp" | grep -c 'FHIRPATH_TERMINOLOGY_SERVER' || true)" '^0$'

echo "== 2. sof-server: terminology inside a ViewDefinition =="
sof=$(curl -sS --max-time 30 -X POST "$SOF/\$viewdefinition-run" \
  -H 'Content-Type: application/fhir+json' \
  -d '{
    "resourceType": "Parameters",
    "parameter": [
      {"name": "viewResource", "resource": {
        "resourceType": "ViewDefinition", "status": "active", "resource": "Patient",
        "select": [{"column": [
          {"name": "id",        "path": "getResourceKey()", "type": "string"},
          {"name": "is_gender", "path": "gender.memberOf('\''http://hl7.org/fhir/ValueSet/administrative-gender'\'')", "type": "boolean"}
        ]}]
      }},
      {"name": "resource", "resource": {"resourceType": "Patient", "id": "tx-check", "gender": "male"}}
    ]
  }')
check "ViewDefinition memberOf() column resolves" "$sof" 'is_gender'
check "sof did not report a missing tx server" \
      "$(printf '%s' "$sof" | grep -c 'TERMINOLOGY_SERVER' || true)" '^0$'

echo "== 3. hfs: :in search modifier (501 when tx server is unset) =="
code=$(curl -sS -o /tmp/hfs-in.json -w '%{http_code}' --max-time 30 \
  -H 'Accept: application/fhir+json' \
  "$HFS/Patient?gender:in=http://hl7.org/fhir/ValueSet/administrative-gender")
if [ "$code" = "501" ]; then
  echo "  FAIL  :in returned 501 -- HFS_TERMINOLOGY_SERVER is NOT set on hfs"
  fail=1
elif [ "$code" = "200" ]; then
  echo "  PASS  :in modifier served (HTTP 200)"
else
  echo "  WARN  :in returned HTTP $code -- inspect /tmp/hfs-in.json"
  head -c 300 /tmp/hfs-in.json; echo
fi

echo
[ "$fail" -eq 0 ] && echo "All terminology checks passed." || echo "One or more checks FAILED."
exit "$fail"

Also worth confirming on the boxes

# Each unit should show its own variable name, pointing at HTS.
ssh <fhirpath-host> 'systemctl show -p Environment fhirpath-server'   # FHIRPATH_TERMINOLOGY_SERVER=...
ssh <sof-host>      'systemctl show -p Environment sof-server'        # SOF_TERMINOLOGY_SERVER=...
ssh <hfs-host>      'systemctl show -p Environment hfs'               # HFS_TERMINOLOGY_SERVER=...

Caveats on the script

• The :in check treats a non-200/501 as a warning, not a failure — an empty test dataset or auth on hfs can produce other codes that are unrelated to terminology wiring. Read the body before concluding.
• It exercises memberOf only. $expand, $subsumes and $translate are not covered and have not been validated against the hosted HTS — $translate in particular is the path CI: FHIRPath tx conformance tests skip silently instead of failing when HTS is unreachable #287/HTS: ConceptMap/$translate rejects the R5-spec 'sourceCoding' parameter (accepts only code+system) #288 touched, so it is worth a manual look.
• HTS advertises fhirVersion: 6.0.0 while the three clients are built R4-first. The Parameters wire shape is stable across versions for these fields, so this is expected to be fine, but it is the first place to look if an operation deserializes badly.