feat(DEV-762): surface GCS integrity headers in feed downloads and status tools - #36
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Resolves DEV-762.
The
content-length/x-goog-hash(CRC32C/MD5) file-validation headers live on the final GCS response after thefeeds.spur.us→storage.googleapis.com302 redirect, not the pre-redirect response, so they're easy to miss when validating a downloaded feed. Our own tooling had the same blind spot: the CLI download followed the redirect but discarded every header exceptContent-Length(progress bar only), and thefeed_status/feed_downloadMCP tools never touched the CDN at all.This surfaces the CDN integrity values (size + CRC32C + MD5) in the CLI download and both MCP tools, and adds an opt-in local
--verify.What changed
internal/spur: newFeedIntegritytype +parseGoogHash/integrityFromResponsehelpers;doStream/DownloadFeednow returnFeedIntegrity(size prefersContent-Rangetotal, elseContent-Length, else -1);ExportFeed's external signature is unchanged. New best-effortProbeFeedIntegrity(aRange: bytes=0-0GET; the drain is capped at 1 MB so a CDN that ignoresRangecan't trigger a full download).internal/app/feed.go:feed downloadprints asize / crc32c / md5summary to stderr (unless--silent); new--verifyhashes the raw pre-gzip stream (CRC32C Castagnoli big-endian + MD5, base64 to matchgsutil/x-goog-hash) and fails on mismatch or when the CDN reported no checksum.internal/tools/feeds.go:feed_statusandfeed_downloadgain a best-effortIntegrityfield viaProbeFeedIntegrity(nil on failure, never fails the tool).Test evidence
go build ./...,go vet ./...,go test ./...all clean.x-goog-hashparsing (comma-joined + multi-line, partial, empty);Content-Range-total precedence and size fallback;DownloadFeedintegrity population;ProbeFeedIntegrity206/200/non-2xx + request headers;--verifymatch / crc32c mismatch / md5 mismatch / raw-gzip-bytes hashing / no-checksum error; MCPIntegritypopulated on success and nil (tool still succeeds) on probe failure, with no probe on error paths.Risk
Low.
doStream/DownloadFeedreturn type changed, but the only external caller (ExportFeed) keeps itsint64signature and all callers compile/pass unchanged.feed_status/feed_downloadnow make one extra cheap CDN round-trip per call (bounded, best-effort). Behavior degrades to the prior output if the probe fails.--verifyis opt-in; default download behavior (stdout decompress,-oraw, MMDB terminal refusal) is unchanged. Note: on a--verifymismatch with-o, the partially-written file is left on disk (documented in the flag help) since streaming verification can't un-write it.