Skip to content

Add audit-logs download command#195

Merged
yummybomb merged 26 commits into
mainfrom
hypeship/audit-logs-download
Jul 16, 2026
Merged

Add audit-logs download command#195
yummybomb merged 26 commits into
mainfrom
hypeship/audit-logs-download

Conversation

@yummybomb

@yummybomb yummybomb commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds kernel audit-logs download for verified chunked audit-log exports.

  • Requires an explicit, exclusive [start, end) range of at most 30 days.
  • Downloads gzip-compressed JSONL with no alternate format modes.
  • Walks the API cursor sequentially and verifies every source chunk checksum before processing it.
  • Retries transient per-chunk failures (network errors, 429/5xx, checksum mismatches) up to 7 attempts (6 retries) with capped exponential backoff (~31s total) before failing the download.
  • Uses the SDK's multi-value method exclusions to send the default GET exclusion and --exclude-method together to the API.
  • Writes to an adjacent .partial file (replacing any stale one from a prior run) and publishes the final .jsonl.gz only after all chunks complete.
  • Removes incomplete partial output after transfer failures and preserves a completed partial if final publication fails.
  • Refuses to replace an existing output unless --force is passed; forced replacement happens only during the final rename.
  • Chunk requests disable the SDK's built-in retries so the download's own retry policy is authoritative.

Test plan

  • go test ./...
  • go vet ./...
  • Race-enabled audit-log tests
  • Windows amd64 test-binary cross-compile
  • Stacked method exclusions match search and download semantics
  • Finalization failures preserve completed partial and existing output files
  • Live download smoke test with stacked GET and POST exclusions
  • SDK retry suppression asserted against a real httptest server
  • Checksum-mismatch recovery and stale-partial overwrite tests

Note

Medium Risk
New CLI path writes sensitive audit data to disk with careful partial-file handling; behavior is well tested but touches export APIs and local file I/O.

Overview
Adds kernel audit-logs download for bulk export of gzip-compressed JSONL over a required exclusive [start, end) window (max 30 days), reusing the same filter flags as search (including default GET exclusion via auditLogExcludeMethods).

The implementation pages through ExportChunk, validates each response with X-Content-Sha256 and pagination headers, and applies its own retry policy (up to 7 attempts, capped backoff; SDK retries disabled) for transient failures. Output is written to a .partial file with restrictive permissions and only renamed to the final path after all chunks succeed; failed runs drop the partial, and a completed partial is kept if final rename fails. --force is required to overwrite an existing destination.

AuditLogsService gains ExportChunk; tests cover multi-chunk assembly, retries, checksum/header validation, overwrite semantics, and finalize edge cases.

Reviewed by Cursor Bugbot for commit 0cda746. Bugbot is set up for automated code reviews on this repo. Configure here.

yummybomb and others added 10 commits July 9, 2026 14:26
Wraps GET /audit-logs/export/chunk (kernel-go-sdk v0.75.0). Chunks are
verified against X-Content-Sha256 before being appended to the output
file, and a sidecar state file persists the cursor and committed byte
offset so an interrupted download resumes without duplicating rows.
Ranges over the API's 30-day cap fail fast with ready-to-run per-window
commands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
--include-get lifts the exclusion. Because chunks are appended verbatim,
a second exclusion can't stack client-side the way search filters, so
--exclude-method with the GET default active is rejected with guidance
rather than silently replacing it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Bind resume state to a credential identity (hashed API key or OAuth
  org ID) so switching credentials can't mix organizations in one archive
- Validate the output file against recorded progress on resume; a missing
  or shortened file now fails instead of being zero-extended and reported
  complete
- Fail closed on missing X-Has-More and X-Content-Sha256 headers, and
  validate pagination headers before any bytes are written
- Commit initial state before the first fetch so an early failure retries
  without --force; --force removes stale state immediately
- Write state via os.CreateTemp and use 0600/0700 modes for outputs
- Cap buffered chunk responses at 256 MiB
- Fix stale -o references in errors; document that resume needs explicit
  --start/--end

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@yummybomb
yummybomb marked this pull request as ready for review July 10, 2026 15:12
yummybomb and others added 2 commits July 10, 2026 15:14
The server controls chunk sizing and every chunk is checksum-verified,
so the client-side byte ceiling only added a failure mode.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The params fingerprint already invalidates stale state files, and the
state is too short-lived to need format versioning.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread cmd/audit_logs_download.go
Comment thread cmd/audit_logs_download.go
Comment thread cmd/audit_logs_download.go
@yummybomb yummybomb changed the title Add audit-logs download command with resumable chunked export Add audit-logs download command Jul 10, 2026
Comment thread cmd/audit_logs_download.go

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit ee8ccd4. Configure here.

Comment thread cmd/audit_logs_download.go Outdated
Comment thread cmd/audit_logs_download.go
Comment thread cmd/audit_logs_download.go
@pulumi

pulumi Bot commented Jul 13, 2026

Copy link
Copy Markdown

⚠️ Pulumi could not deploy preview(s) for this pull request because GitHub reports it is not mergeable (mergeable state: dirty). This usually means the branch has merge conflicts with its base branch. Resolve the conflicts and push a new commit to retry.

@yummybomb
yummybomb requested a review from sjmiller609 July 13, 2026 20:55

@sjmiller609 sjmiller609 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, either as follow up or before merge, maybe you want to test interrupting one of the chunk downloads, e.g. cut your wifi then turn it back on, see if it can retry the missed chunks properly

- Pass option.WithMaxRetries(0) to ExportChunk so the chunk loop
  exclusively owns retries; the SDK's default two inner retries
  tripled request volume against 429/5xx responses.
- Require --force to overwrite an existing .partial file so a rerun
  cannot silently truncate a completed download preserved after a
  finalization failure.
- Deduplicate the output writability check and drop a redundant Chmod.
- Document the 30-day range limit and add a usage example.
- Cover SDK retry suppression against a real httptest server, checksum
  mismatch recovery, existing-partial protection, rerun after a
  finalization failure, and the remaining chunk header validations.
Seven total attempts give six retry delays (1+2+4+8+8+8s = 31s);
the previous six attempts only backed off for 23 seconds.
@yummybomb
yummybomb force-pushed the hypeship/audit-logs-download branch from 1854044 to 58060a0 Compare July 16, 2026 16:48
Partial files are internal transfer state, not user output; requiring
--force to replace one made reruns after an interrupted or
partially-finalized download needlessly strict. --force again gates
only the final output path.
fang title-cases the first word of error messages, which mangles a
leading path ("/tmp/out.gz" renders as "/Tmp/Out.gz"). Start the
messages with "output" / "partial file" so the path is untouched.
OpenFile's 0600 mode only applies when the file is created; a stale
partial reused via O_TRUNC keeps its previous permissions, and the
final rename would publish the archive with them. Chmod the partial
after opening and assert the published file mode in the stale-partial
test.
@yummybomb
yummybomb merged commit 89c4d23 into main Jul 16, 2026
7 checks passed
@yummybomb
yummybomb deleted the hypeship/audit-logs-download branch July 16, 2026 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants