tx.fhir.org serves status=active for VSAC value sets VSAC has retired (mirror missing in-place retirements)

[jmandel]

Summary

tx.fhir.org serves status=active for VSAC value-set versions that the authoritative VSAC FHIR API (cts.nlm.nih.gov) reports as retired — including the widely-used clinical sets Smoking Status and Tobacco Use and the US-Realm core Detailed Ethnicity set. For each, the same revision (identical meta.versionId) is active on tx but retired on VSAC, so this is not "tx holds an older revision" — it is the mirror not observing VSAC's in-place retirements.

Independently verified live (2026-06-06)

Each row: identical OID + version + meta.versionId on both servers; only status differs.

OID	version	versionId	tx.fhir.org	VSAC (cts.nlm.nih.gov) — authoritative
2.16.840.1.114222.4.11.877 — Detailed Ethnicity	20251123	22	active	retired (lastUpdated 2026-04-13)
2.16.840.1.113883.11.20.9.38 — Smoking Status	20240606	25	active	retired (2026-04-29)
2.16.840.1.113883.11.20.9.41 — Tobacco Use	20240606	36	active	retired (2026-04-29)
2.16.840.1.113762.1.4.1170.58 — Goals/preferences for medical treatment	20250403	5	active	retired (2026-05-29)

Repro

# tx (no auth) -> status=active
curl -s -G -H 'Accept: application/fhir+json' \
  --data-urlencode 'url=http://cts.nlm.nih.gov/fhir/ValueSet/2.16.840.1.114222.4.11.877' \
  --data-urlencode 'version=20251123' \
  'https://tx.fhir.org/r4/ValueSet' \
 | jq '.entry[0].resource | {version, status, vId: .meta.versionId, lastUpdated: .meta.lastUpdated}'
# => {"version":"20251123","status":"active","vId":"22","lastUpdated":"2025-11-23T18:00:57.000-05:00"}

# authoritative VSAC (Basic auth: username "apikey", password = your UMLS/VSAC API key) -> status=retired
curl -s -G -H 'Accept: application/fhir+json' -u "apikey:$VSAC_API_KEY" \
  --data-urlencode 'url=http://cts.nlm.nih.gov/fhir/ValueSet/2.16.840.1.114222.4.11.877' \
  --data-urlencode 'version=20251123' \
  'https://cts.nlm.nih.gov/fhir/ValueSet' \
 | jq '.entry[0].resource | {version, status, vId: .meta.versionId, lastUpdated: .meta.lastUpdated}'
# => {"version":"20251123","status":"retired","vId":"22","lastUpdated":"2026-04-13T20:10:21.000-04:00"}

Expected vs actual

• Expected: the mirror reflects the authoritative source's current status. Because meta.versionId is identical on both sides (22, 25, 36, 5), tx is holding the same revision VSAC has since retired in place, so tx should report retired.
• Actual: tx reports active, between ~1 month (Goals/preferences) and ~7 weeks (Detailed Ethnicity) after VSAC's retirement.

Why it differs (source + lag mechanism)

• Source citation (not an assumption): the authoritative source itself — VSAC's own FHIR API at cts.nlm.nih.gov — returns retired for these exact revisions (verified above, same versionId).
• VSAC frequently retires a version in place: it flips status: active → retired on the same version. For OID 877 it also bumped meta.lastUpdated (2025-11-23 → 2026-04-13), so even a timestamp-aware mirror should have caught it — yet tx has not, ~7 weeks later. tx even ingested OID 877's successor 20260410 (so it touched this OID after the retirement), but left 20251123 flagged active.
• Per FHIR PublicationStatus, retired means the resource "should no longer be used" — serving such a value set as active misrepresents its lifecycle to consumers (https://hl7.org/fhir/valueset-publication-status.html).

Scope / honesty

This is a data-freshness / mirror-sync observation (FHIR does not mandate a refresh interval), filed because the affected sets are high-traffic. The 4 rows above were independently re-verified today against both servers with a live VSAC API key. (An earlier 5th candidate, OID 2.16.840.1.113762.1.4.1260.427, was dropped: tx serves it at a different versionId than VSAC's retired revision, so it is not a clean same-revision discrepancy.)

---

Re-confirmed live against https://tx.fhir.org on 2026-06-06; server self-reports FHIRsmith 0.9.6.

[grahamegrieve]

Referred to VSAC for discussion

[jmandel]

Update: VSAC offers a cheap incremental-sync mechanism — and it catches these exact value sets

A fair question on this issue is whether keeping a VSAC mirror current requires re-scanning the whole corpus daily. It does not — VSAC's FHIR API supports the standard _lastUpdated delta-sync, and I verified it returns these specific retired value sets.

The corpus + the mechanism (verified live against cts.nlm.nih.gov/fhir, 2026-06-06)

• Total ValueSets on VSAC: 18,127.
• _lastUpdated is a supported ValueSet search parameter (per VSAC's own CapabilityStatement) — usable both as a filter (_lastUpdated=gtYYYY-MM-DD) and as _sort=-_lastUpdated.
• The daily delta is tiny: 6 value sets changed in the last ~3 days, 295 in ~2 weeks, 1,107 since 2026-04-12. So a mirror pulls only the delta with GET /ValueSet?_lastUpdated=gt<last-sync>&_sort=-_lastUpdated&_count=N — never the full 18k.
• (There is no FHIR _history change-feed — /ValueSet/_history → HTTP 400 — but _lastUpdated search is sufficient and is the intended mechanism.)

Proof the incremental poll catches the four stale value sets

Each retirement bumped meta.lastUpdated, so a _lastUpdated-windowed query returns the version as retired; a control window starting after the retirement correctly excludes it:

OID (name)	version	_lastUpdated=gt2026-04-12 (spanning window)	control _lastUpdated=gt2026-06-04
2.16.840.1.114222.4.11.877 (Detailed Ethnicity)	20251123	HIT — status=retired, lastUpdated 2026-04-13	excluded (0)
2.16.840.1.113883.11.20.9.38 (Smoking Status)	20240606	HIT — status=retired, lastUpdated 2026-04-29	excluded (0)
2.16.840.1.113883.11.20.9.41 (Tobacco Use)	20240606	HIT — status=retired, lastUpdated 2026-04-29	excluded (0)
2.16.840.1.113762.1.4.1170.58 (Goals/preferences for medical treatment)	20250403	HIT — status=retired, lastUpdated 2026-05-29	excluded (0)

Repro (windowed delta query returning the retired version):

curl -s -u "apikey:$VSAC_API_KEY" -H 'Accept: application/fhir+json' \
 'https://cts.nlm.nih.gov/fhir/ValueSet?url=http://cts.nlm.nih.gov/fhir/ValueSet/2.16.840.1.114222.4.11.877&version=20251123&_lastUpdated=gt2026-04-12&_elements=version,status,meta'
# => 1 entry, status "retired", meta.lastUpdated 2026-04-13

Correction to the lag-mechanism note above

All four of these retirements did bump meta.lastUpdated (into 2026), so a mirror keyed on _lastUpdated would observe them. Please disregard the earlier hedge that a lastUpdated-keyed sync "can miss in-place retirements" — that is not the case for any of these four. The takeaway is therefore stronger, not weaker: an inexpensive incremental _lastUpdated poll suffices to stay current, yet tx is serving these as active 7+ weeks (OID 877) to over a year (others) after VSAC retired them.

[jmandel]

Sliding-window check: _lastUpdated behaves as a true incremental cursor

Confirming the delta-sync cursor works monotonically — a tighter _lastUpdated lower bound returns fewer results, widening it returns more. Result counts via GET /ValueSet?_lastUpdated=gt<cutoff>&_count=0 (reading Bundle.total), run live against cts.nlm.nih.gov/fhir at 2026-06-06T22:13Z:

window (_lastUpdated > now − …)	ValueSets changed
5 seconds	0
1 hour	0
1 day	0
1 week	206
1 month	585
3 months	1,505
1 year	5,893
(no filter — entire corpus)	18127

So the cursor is real: a few-seconds/hour/day window returns 0 (nothing changed that recently — the most recent VSAC change at run time was 2026-06-04), and the count grows smoothly as the window widens. A mirror polling daily (or even weekly) with _lastUpdated=gt<last-sync> pulls only the delta — single-digit to low-hundreds of value sets — never the full 18127. This is the cheap, standard mechanism tx would use to stay current.

[jmandel]

Thanks — you're right that the _lastUpdated scan (_scanLastUpdated, phase 1b) is there. I traced it end-to-end and the query mechanism is sound; the staleness looks like a coverage gap, not a missing query. What I verified against live VSAC:

• VSAC does bump meta.lastUpdated on a retire-in-place: 877→2026-04-13, 9.38/9.41→2026-04-29, 1170.58→2026-05-29 — so they're visible to _lastUpdated.
• GET /ValueSet?_lastUpdated=ge2026-04-12 returns all four as status=retired; /res/ValueSet/… returns 200; the search bundle carries meta.lastUpdated for the cursor.
• batchUpsertValueSets hashes JSON.stringify(vs), which includes status — so if these were fetched, the upsert would fire. So nothing downstream is dropping them; they're just never re-fetched.

Two spots in vsacRefresh (from the published main) explain that:

1. Phase 1a never re-checks an already-known version. The full list (/ValueSet?_count=1000&_elements=id,url,version,status) enqueues only when !this.valueSetMap.has(vs.url+"|"+vs.version). A version that flips active→retired in place is already in the map, so it's never re-fetched/re-hashed here — unless a brand-new sibling version also appears (which re-pulls the whole url via phase 2). 877 is a perfect example: its successor 20260410 appeared ~04-10, three days before 20251123 was retired on 04-13, so by retirement time both versions were already in the map → phase 1a skipped the url entirely.

2. Phase 1b only covers a rolling window. _scanLastUpdated defaults to a 10-day look-back on a cold start, otherwise rides a watermark (bundle.meta.lastUpdated). A retirement that happened before 1b's window — because 1b first ran after it, or a cycle was missed/failed — is never reconciled afterward. That matches the observed lags (53 days to >1 year).

So in-place retirements only get picked up if phase 1b's window happens to span them; otherwise nothing ever revisits an existing url|version.

A near-free fix: phase 1a already has the current status in hand (it's in the _elements=id,url,version,status projection) but only checks map-existence. Comparing the fetched status against the stored status and enqueuing on mismatch would catch every retire-in-place with zero extra requests. And on a cold start / when the gap since the last successful run exceeds the look-back, widening the 1b _lastUpdated window to cover that gap (or a one-time backfill) would reconcile the historical ones once.

(Caveat: this is from tracing main, not the deployed 0.9.6 — happy to be corrected on specifics. Corpus size, for reference: VSAC currently has 18,127 distinct value sets [9,463 active + 8,664 retired], ~44k counting all historical versions — so an incremental _lastUpdated poll's daily delta is tiny, single-digit to low-hundreds.)

[grahamegrieve]

I don't understand why the 10 day window on cold start or the watermark is a problem - on a cold start, everything is loaded at once, so 10 days is 9 days too long. And there after, it's keeping track of time

[jmandel]

Following up: you're right that the 10-day/watermark framing was off — cold start loads everything and the watermark tracks time. Tracing the actual code, my best hypothesis is a durability gap, not the scan:

_scanLastUpdated() commits the watermark at its end (vs-vsac.js:620) — i.e. at the end of phase 1b, before phase 2 (:188–211) actually fetches/upserts the queued URLs. Each processContentAndHistory is try/catch → requeued once (:195) → a second failure is just logged and dropped. So if a per-URL /ValueSet?url=X fetch throws twice (transient VSAC error), or the process restarts mid-phase-2, that value set is never written — and the watermark has already moved past its lastUpdated, so it's never re-attempted. One missed attempt = permanently stale.

Per run that's a low-probability path, but with no retry it compounds — which fits a handful of stranded value sets over months rather than a wholesale failure. Not proven, though: vsac_runs (do runs reach finishRun, or does the watermark advance even on a restart?) and vsac_events (any per-URL errors) would confirm. Cheapest tell: re-sync just the 5 stuck URLs — if that fixes them, they were simply never re-attempted.

Fix direction: advance the watermark only after phase 2 + requeue succeed (or never past the lowest un-applied change).

[grahamegrieve]

I'll do that, but I'm sceptical that this is the cause. We can't know, of course :-(

[grahamegrieve]

ok, updated, and I added a way to resend individual value sets at /r4/info/vsac. consult me elsewhere for the password