Cache key moved on an rlang upgrade, silently orphaning every cached cube and fetch

[NewGraphEnvironment]

Both frozen cache-key goldens fail on main

test-dft_stac_cube.R:62   expected "638a2be11fdf"   actual "45685ccbda33"
test-dft_stac_fetch.R:88  (same class, golden "79f67b7b9dae")

Verified against a clean main worktree, not a feature branch: main is
FAIL 2 | PASS 407. stac_cube_cache_key() and stac_cache_key() are
byte-identical to the commits that pinned those hashes, so nothing in drift
changed — this is environment drift underneath rlang::hash()
(sf / PROJ / R serialization version).

Why it matters more than a red test

The hash is the cache filename. <source>/cube_<key>.tif and
<year>_<key>.nc. So when the key moved, every cached cube and every cached
fetch in every user's dft_cache_path() was silently orphaned: nothing hits
cache any more, and every call re-streams from the Planetary Computer.

Each cube costs 10–30 minutes to rebuild. There is no error, no warning, and no
log line — the symptom is "the pipeline got slower for no reason", which is
exactly the kind of thing nobody files.

The tests were doing their job. They are the only reason this is known.

What NOT to do

Do not re-pin the goldens to the current values. That is loosening an assertion
until it stops failing, and it would discard the one signal that says the key
moved. The tests' own comments say the hash must be stable precisely because
caches are named by it.

What to establish first

• When did it move, and how far back? Bisect rlang::hash() of the key
  parts list against installed sf / PROJ / rlang versions. git log on the
  two key functions will show they did not change, which localises it to the
  environment.
• Which member drifted? Hash each element of parts separately. The prime
  suspect is sf::st_as_binary() output or the serialization of the sfc
  attributes — code-check.md already records that an sfc carries a
  PROJ-generated CRS WKT that drifts across PROJ versions, which is why the key
  hashes WKB and passes the CRS separately. Worth confirming that reasoning
  still holds.
• Is the key stable going forward, or will it move again on the next PROJ
  bump? That is the question that decides the fix. A key that silently
  re-shuffles on a dependency upgrade is a recurring outage, not a one-off.

Candidate fixes, in rough order of preference

1. Make the key robust to the drifting member — hash a canonical
   representation rather than a serialized R object, so a PROJ or sf upgrade
   cannot move it.
2. Version the cache directory (cache/v2/...), so a deliberate key change
   is a migration rather than a silent orphaning, and old caches are findable.
3. Detect and report, at minimum: on a cache miss where a same-AOI file
   exists under a different key, say so, rather than silently re-streaming.

Related

• dft_stac_fetch cache key omits AOI -> second area silently gets first area's raster #25 — the cache-key-covers-every-input work that established these goldens
• filter_geom evaluated and rejected on measurement; ship gdalcubes parallel instead (was: adopt the fixed filter_geom) #47 — where this surfaced (unrelated branch; the two failures are identical on
  main, so they did not block it)

---

Reconciliation — what the work established

Recorded at merge time (PR #57). The analysis above is kept as written; these are the points where
measurement gave a different answer.

1. The suspected cause is wrong. It is not sf/PROJ — it is rlang::hash() itself.

The body nominates sf::st_as_binary() output or sfc attribute serialization drifting across
PROJ versions. Measured, that is not what happened. rlang::hash() was rewritten in rlang
1.3.0, and rlang's own NEWS says so:

hash() now uses its own walking strategy… This does mean that with this version all hash
values will now be different. …you should assume it's always possible for a new version to
invalidate existing hashes.

This matters because it changes the fix. Candidate fix 1 ("make the key robust to the drifting
member") presumes there is a drifting member; there is not, and a fix aimed at the geometry
member would have left the real cause in place to recur on the next rlang bump.

2. "Which member drifted?" has no answer — the whole hash function moved. Evidence:

key	pinned	recomputed today
fetch @ cf04bfe (#36)	79f67b7b9dae	68d66c3fbad9 (moved)
cube @ 90f9d93 (#38)	638a2be11fdf	45685ccbda33 (moved)
fetch @ fc2e861 (#51)	2264b5dbef6e	unchanged (holds)

The key function extracted from its own pinning commit reproduces today's value, so drift's code
never changed. rlang 1.3.0 was installed 2026-08-06, between the #38 pin (2026-07-11) and the #51
re-pin (2026-09-01), which is exactly why only the older goldens fail.

3. The passing golden is not a control. It holds only because #51 re-pinned it after the
upgrade. Its contemporaneous #36-era value fails too — so "the fetch key is fine, therefore the
shared members are fine" is the available wrong inference, and this body's FAIL 2 count would
have reinforced it.

4. "Is the key stable going forward?" — not with rlang::hash(), by rlang's own statement.
That answer is what selected the fix, and it also settles the pinning question: no rlang pin,
because after the fix rlang is not in the key path at all.

5. All three candidate fixes shipped, with fix 1 reframed. Not "robust to the drifting member"
but keyed by content: a canonical string hashed by its bytes with
digest::digest(algo = "xxhash64", serialize = FALSE). Fix 2 (versioned cache directory) shipped as
<cache>/v2/<source>/. Fix 3 (detect and report) shipped as dft_cache_info()'s
n_files_superseded / size_mb_superseded plus dft_cache_clear(scheme = "superseded").

6. The failure count in this body is stale. It reports FAIL 2 | PASS 407; main at the time
of the fix was FAIL 1 | PASS 522 — only the cube golden. The fetch golden passes, for the reason
in point 3.

7. The stated cost overstates what was being paid. "Each cube costs 10–30 min to rebuild" is
true of Sentinel-2 cubes but the cache held zero cube entries — it was entirely io-lulc fetch
entries. Measured re-fetch is ~10 s per entry and flat in entry size, so the real one-time cost
is minutes, not hours. The cube figure remains the forward-looking reason this mattered: the next
rlang bump would have destroyed those silently.

[NewGraphEnvironment]

Measurements from #51 / PR #52, and a way this got worse

Confirming this issue's central claim and correcting one thing I got wrong while working alongside it.

Both goldens were red at 66b2c36 (pre-#51 main), as this issue says:

test-dft_stac_cube.R:62   Expected `cube_key()`  to equal "638a2be11fdf"
test-dft_stac_fetch.R:88  Expected `cache_key()` to equal "79f67b7b9dae"
[ FAIL 2 | WARN 0 | SKIP 5 | PASS 177 ]

New: the cube golden has been red since the commit that introduced it. Bisected — 90f9d93 (#38), which pinned 638a2be11fdf, fails its own assertion, as does every commit after it. 90f9d93 is an ancestor of v0.9.0, and v0.9.0 was tagged over the same FAIL 1. So for the cube at least, "the key moved" may be the wrong frame: the literal may never have matched on this machine. Worth distinguishing before hunting a dependency, since the two have different fixes.

That is a cheap check: compute the key from the fixture as it stood at 90f9d93 and see whether any historical version produces 638a2be11fdf. If none does, the golden was recorded from a different environment and there is no drift event to bisect for the cube.

And a way #51 made this harder, which is on me. PR #52 added a deliberate salt to stac_cache_key() and moved the fetch golden 79f67b7b9dae → 2264b5dbef6e. My stated reason was cache invalidation: the paging fix changes no key parameter, so a raster already written from a truncated item set would keep being served under force = FALSE.

That reasoning still holds on its own terms. But I believed the fetch golden was passing on main when I moved it, and it was not — I had only run the suite filtered to dft_stac_cube, and inferred the rest. So what I actually did was re-pin a golden that was already failing, to the value this environment currently produces — which is precisely what this issue says not to do.

Two consequences worth naming:

1. Users' fetch caches were already orphaned by the drift, before dft_stac_fetch() reads one STAC page — a wide AOI can build a raster from a truncated item set #51. v0.10.0's NEWS presents the rebuild as a consequence of the deliberate salt. That is now only half true, and the NEWS is misleading about the cause.
2. 2264b5dbef6e is presumably as environment-specific as 79f67b7b9dae was. If the drift is real, the new golden will fail on any other machine — including on the R-CMD-check workflow Cube cache-key guardian has been red since it was written, and no CI runs the tests #54 asks for. The salt does not make the key stable; it only moved it.

I did not re-freeze the cube golden for exactly the reason this issue gives. I should have applied the same reasoning to the fetch one and did not, because I had not measured it.

This does not change the recommended fix — options 1 and 2 in the body (canonical representation, or a versioned cache directory) both still apply, and option 2 would have made #51's intentional invalidation a migration instead of another silent key move.

Closing #54 as a duplicate of this.