Skip to content

feat(DEV-762): surface GCS integrity headers in feed downloads and status tools - #36

Merged
jjunqueira merged 10 commits into
mainfrom
feat/dev-762
Jul 23, 2026
Merged

feat(DEV-762): surface GCS integrity headers in feed downloads and status tools#36
jjunqueira merged 10 commits into
mainfrom
feat/dev-762

Conversation

@jjunqueira

@jjunqueira jjunqueira commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Context

Resolves DEV-762.

The content-length / x-goog-hash (CRC32C/MD5) file-validation headers live on the final GCS response after the feeds.spur.usstorage.googleapis.com 302 redirect, not the pre-redirect response, so they're easy to miss when validating a downloaded feed. Our own tooling had the same blind spot: the CLI download followed the redirect but discarded every header except Content-Length (progress bar only), and the feed_status / feed_download MCP tools never touched the CDN at all.

This surfaces the CDN integrity values (size + CRC32C + MD5) in the CLI download and both MCP tools, and adds an opt-in local --verify.

What changed

  • internal/spur: new FeedIntegrity type + parseGoogHash/integrityFromResponse helpers; doStream/DownloadFeed now return FeedIntegrity (size prefers Content-Range total, else Content-Length, else -1); ExportFeed's external signature is unchanged. New best-effort ProbeFeedIntegrity (a Range: bytes=0-0 GET; the drain is capped at 1 MB so a CDN that ignores Range can't trigger a full download).
  • internal/app/feed.go: feed download prints a size / crc32c / md5 summary to stderr (unless --silent); new --verify hashes the raw pre-gzip stream (CRC32C Castagnoli big-endian + MD5, base64 to match gsutil / x-goog-hash) and fails on mismatch or when the CDN reported no checksum.
  • internal/tools/feeds.go: feed_status and feed_download gain a best-effort Integrity field via ProbeFeedIntegrity (nil on failure, never fails the tool).

Test evidence

  • go build ./..., go vet ./..., go test ./... all clean.
  • New unit tests across all three layers: x-goog-hash parsing (comma-joined + multi-line, partial, empty); Content-Range-total precedence and size fallback; DownloadFeed integrity population; ProbeFeedIntegrity 206/200/non-2xx + request headers; --verify match / crc32c mismatch / md5 mismatch / raw-gzip-bytes hashing / no-checksum error; MCP Integrity populated on success and nil (tool still succeeds) on probe failure, with no probe on error paths.
  • Independent code review pass: no blockers, no leaked secrets.

Risk

Low.

  • doStream/DownloadFeed return type changed, but the only external caller (ExportFeed) keeps its int64 signature and all callers compile/pass unchanged.
  • feed_status / feed_download now make one extra cheap CDN round-trip per call (bounded, best-effort). Behavior degrades to the prior output if the probe fails.
  • --verify is opt-in; default download behavior (stdout decompress, -o raw, MMDB terminal refusal) is unchanged. Note: on a --verify mismatch with -o, the partially-written file is left on disk (documented in the flag help) since streaming verification can't un-write it.

@jjunqueira
jjunqueira merged commit 28855ad into main Jul 23, 2026
4 checks passed
@spurintelbot spurintelbot Bot mentioned this pull request Jul 23, 2026
@jjunqueira
jjunqueira deleted the feat/dev-762 branch July 23, 2026 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant