fix(anchor): default to a Rekor shard that exists, and name it when it fails - #9845
Conversation
…t fails DEFAULT_REKOR_SHARD_BASE_URL was https://log2026-1.rekor.sigstore.dev -- a shard Sigstore has not deployed. Found on a live self-host instance with 1,626 ledger rows: the anchor signed correctly, then failed with `fetch failed` because the hostname is NXDOMAIN. Every deployment that enabled anchoring without overriding the env var was in that state, publishing no anchor at all. Sigstore shards annually as log<year>-<rev>; 2025-1 is live and answers the v2 POST path, so the default moves there. Guessing the next shard ahead of its deployment is strictly worse than lagging behind it -- a stale-but-real default still anchors, a not-yet-existent one silently anchors nothing. The test that should have caught this asserted `stringContaining("rekor.sigstore.dev")`, which a nonexistent subdomain satisfies. It now pins the host exactly. Second fix: the recorded failure said only "fetch failed", Node's generic message, with no URL. An operator could not distinguish a shard that does not resolve from blocked egress from a log that is down -- three different fixes, and diagnosing the real one meant probing DNS inside the container by hand. The thrown-error path now wraps the cause with the attempted endpoint, preserving it as `cause` so nothing inspectable is lost. #9271 published these failures so anyone can see anchoring is broken; naming the endpoint is what makes that actionable. Closes #9844
|
Tip ✅ LoopOver review result - approve/merge recommendedReview updated: 2026-07-29 14:06:27 UTC
Review summary Nits — 5 non-blocking
Decision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionPartially addressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9845 +/- ##
==========================================
- Coverage 91.64% 90.77% -0.88%
==========================================
Files 916 916
Lines 112753 112754 +1
Branches 27085 27086 +1
==========================================
- Hits 103338 102348 -990
- Misses 8126 9315 +1189
+ Partials 1289 1091 -198
Flags with carried forward coverage won't be shown. Click here to find out more.
|
…t fails (#9845) DEFAULT_REKOR_SHARD_BASE_URL was https://log2026-1.rekor.sigstore.dev -- a shard Sigstore has not deployed. Found on a live self-host instance with 1,626 ledger rows: the anchor signed correctly, then failed with `fetch failed` because the hostname is NXDOMAIN. Every deployment that enabled anchoring without overriding the env var was in that state, publishing no anchor at all. Sigstore shards annually as log<year>-<rev>; 2025-1 is live and answers the v2 POST path, so the default moves there. Guessing the next shard ahead of its deployment is strictly worse than lagging behind it -- a stale-but-real default still anchors, a not-yet-existent one silently anchors nothing. The test that should have caught this asserted `stringContaining("rekor.sigstore.dev")`, which a nonexistent subdomain satisfies. It now pins the host exactly. Second fix: the recorded failure said only "fetch failed", Node's generic message, with no URL. An operator could not distinguish a shard that does not resolve from blocked egress from a log that is down -- three different fixes, and diagnosing the real one meant probing DNS inside the container by hand. The thrown-error path now wraps the cause with the attempted endpoint, preserving it as `cause` so nothing inspectable is lost. #9271 published these failures so anyone can see anchoring is broken; naming the endpoint is what makes that actionable. Closes #9844
submitToRekor was v2 on the request side -- hashedRekordRequestV002, POST /api/v2/log/entries -- while parseRekorResponse expected v1: an object keyed by entry uuid, a numeric logIndex, and a uuid field. Rekor v2 returns the TransparencyLogEntry directly, encodes logIndex as a proto3 int64 (JSON "42", a string), and has no uuid at all. All three checks failed on every real response, so this backend could never record a successful anchor even when the log accepted the submission -- it wrote status: failed with "did not match the expected TransparencyLogEntry shape". Surfaced on a live self-host instance: once #9845 fixed the shard URL, the error moved from fetch failed (DNS) to this parse failure, i.e. the submission now reaches a real log and is rejected only at our end. The fixture was itself v1-shaped and its assertion's title read "reading the entry under its dynamic uuid key", so the test agreed with the bug and could not fail. Replaced with a real-shaped entry; the scheduler's inline mock had the same problem. backendRef carries the inclusion proof's checkpoint where uuid used to be -- v2's locator, since an entry is identified by log index plus the checkpoint the proof was issued against. An absent checkpoint parses to null rather than failing: the entry IS in the log, only the offline re-check is unavailable, and discarding a real submission over an optional field is the worse trade. An unparseable logIndex still fails, so NaN can never reach a published backendRef. Closes #9854
Closes #9844
Found in production, not in review
Provisioning anchoring on a live self-host instance (1,626 decision-ledger rows) produced this published anchor row:
{"seq":1626,"keyId":"6b6490126ad44b51","backend":"rekor","status":"failed","error":"fetch failed"}The signing worked — the key was used and the tip read. The submission did not, because the default shard does not exist:
Sigstore shards Rekor v2 annually as
log<year>-<rev>, and the 2026 shard has not shipped. The constant's comment described it as "the current shard as of when this was written" — but it was written ahead of a shard that never existed. Every deployment that enabled anchoring without overriding the env var published no anchor at all.Why the tests did not catch it
A nonexistent subdomain satisfies that substring perfectly. The exact host is the thing under test here, so it is now asserted exactly — a future wrong guess fails.
The second, compounding fix
"fetch failed"is Node's generic message with no URL in it. An operator cannot tell apart:Three different fixes. Diagnosing the real one required probing DNS from inside the container by hand.
That matters more here than in ordinary code: #9271 made anchoring failures publicly visible so "anchoring has been failing for a week" is checkable by anyone. A published failure that does not say what failed only half-delivers that.
The thrown-error path now wraps the cause with the attempted endpoint and preserves the original as
cause, so nothing previously inspectable is lost. The!response.okpath already carried status and body and is untouched, as is therecordLedgerAnchorAttemptcontract — the raw value still reaches the persistence layer, which stays the single place that normalizes an unknown error.Note on the pre-existing test I changed
records status:'failed' … on a network exceptionassertederror: "network down"exactly. It now asserts the cause is still legible and that the endpoint appears. That is a deliberate contract change, called out here rather than slipped in.Validation
tsc --noEmitclean. 60 tests pass across the rekor, scheduler and persistence suites.ledger-anchor-rekor.tsat 100% lines / 100% branches.Verified end to end on the instance that surfaced it: with the shard corrected, the container reaches
POST /api/v2/log/entriesand gets 400 on a deliberately empty body — endpoint live, request rejected as expected.