Skip to content

cli: reset command to clear the derived projection - #55

Merged
alsmola merged 4 commits into
mainfrom
feat/reset-command
Jul 26, 2026
Merged

cli: reset command to clear the derived projection#55
alsmola merged 4 commits into
mainfrom
feat/reset-command

Conversation

@alsmola

@alsmola alsmola commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Adds trailtool reset, the first of the two reset-and-replay primitives from #49 (design in #54, docs/design/cloudtrail-replay.md §4).

Reset empties every derived DynamoDB table (people, sessions, roles, resources, accounts, services, relations, identity-links) plus the ingested-files markers, so the projection can be rebuilt from the CloudTrail log by a later replay. The log is never touched, so a reset is always recoverable by replaying it.

What it does

  • Store.Reset truncates each table in place: a key-only Scan (projecting just the primary-key attributes, discovered from the table's schema so no per-table key layout is hardcoded) feeds batched BatchWriteItem deletes. The table and its GSIs stay intact, so live ingestion resumes the moment reset finishes.
  • The ingested-files markers are cleared with the rest. Leaving them would make a subsequent replay skip every object and rebuild nothing.
  • Missing tables are skipped, not fatal, so reset works on a partially-deployed stack.
  • Throttled BatchWriteItem leftovers (UnprocessedItems, a successful response the SDK does not auto-retry) re-submit with capped exponential backoff, honoring context cancellation.

Safety

  • Prints the tables and their approximate item counts, then requires a typed yes. --yes skips the prompt for scripting. Empty/EOF input cancels.
  • Help notes reset should run while ingestion is quiet: objects written mid-reset may be missed. The intended flow is reset, then replay the range.

Counts in the prompt come from DescribeTable's ItemCount, which DynamoDB refreshes roughly every six hours, so they are estimates (shown with ~). Tests cover the confirmation guard (decline / EOF / yes / --yes), JSON output, table coverage, and the key-projection and chunking helpers.

Replay (the trailtool replay half) is the next slice.

Part of #49.

Adds `trailtool reset`, which empties every derived DynamoDB table
(people, sessions, roles, resources, accounts, services, relations,
identity-links) plus the ingested-files markers, so the projection
can be rebuilt from the CloudTrail log by a later replay. The log is
never touched; a reset is recoverable by replaying it.

Store.Reset truncates each table in place via a key-only scan and
batched deletes, keeping the table and its GSIs intact so live
ingestion resumes immediately. Missing tables are skipped, not fatal.
Throttled BatchWriteItem leftovers retry with capped backoff. The
markers table is cleared with the rest, so replay does not skip
every object afterward.

The command prints the tables and their approximate item counts and
requires a typed "yes" (or --yes) before deleting.

Part of #49

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
--yes now skips CountItems entirely (it only fed the confirmation
prompt), so a scripted reset makes no pointless DescribeTable calls.

Help text states reset is an offline operation and why: truncation is
not atomic across tables, so an object written mid-reset can land split
across cleared and not-yet-cleared tables and be double-counted on the
next replay.

Part of #49

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@alsmola

alsmola commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the review items that touch reset:

  • [P1] Reset unsafe during live ingestion — reset is now scoped and documented as an offline operation. The help states why: truncation is not atomic across tables, so an object written mid-reset can land split across cleared and not-yet-cleared tables and be double-counted on the next replay. A coordinated rebuild that fences ingestion (pausing the EventBridge rule) is the principled fix, tracked as a follow-up in the design (Design: reset and replay from the CloudTrail log #54 §11). For v1 the offline constraint is the contract.
  • Simplification: skip CountItems under --yes — the count query only fed the confirmation prompt, so --yes now skips it entirely (no pointless DescribeTable calls in a scripted reset). New test asserts it isn't called.

The design doc (#54) is updated to match: the reset section now leads with the offline requirement and the cross-table non-atomicity, and lists the ingestion fence as the deferred fix.

This PR is now the base of the stack; #56 (replay) is retargeted onto it, since both previously inserted at the same main.go line. Merge this first.

Alex Smolen and others added 2 commits July 26, 2026 14:44
Drops the labored cross-table-atomicity wording for a one-line
"run while live delivery is quiet" note.

Part of #49

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Store.Reset takes a progress callback, invoked per scan page (so the
count ticks up as a large table drains) and once per table on
completion. The reset command passes a text-mode callback that writes
a single rewriting stderr line (clearing N/M  <table>  K deleted),
matching the replay command's progress style. JSON mode passes no
callback, so machine output stays clean.

Part of #49

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@alsmola
alsmola merged commit f11e1ec into main Jul 26, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant