cli: reset command to clear the derived projection - #55
Merged
Conversation
Adds `trailtool reset`, which empties every derived DynamoDB table (people, sessions, roles, resources, accounts, services, relations, identity-links) plus the ingested-files markers, so the projection can be rebuilt from the CloudTrail log by a later replay. The log is never touched; a reset is recoverable by replaying it. Store.Reset truncates each table in place via a key-only scan and batched deletes, keeping the table and its GSIs intact so live ingestion resumes immediately. Missing tables are skipped, not fatal. Throttled BatchWriteItem leftovers retry with capped backoff. The markers table is cleared with the rest, so replay does not skip every object afterward. The command prints the tables and their approximate item counts and requires a typed "yes" (or --yes) before deleting. Part of #49 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 26, 2026
--yes now skips CountItems entirely (it only fed the confirmation prompt), so a scripted reset makes no pointless DescribeTable calls. Help text states reset is an offline operation and why: truncation is not atomic across tables, so an object written mid-reset can land split across cleared and not-yet-cleared tables and be double-counted on the next replay. Part of #49 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
Addressed the review items that touch reset:
The design doc (#54) is updated to match: the reset section now leads with the offline requirement and the cross-table non-atomicity, and lists the ingestion fence as the deferred fix. This PR is now the base of the stack; #56 (replay) is retargeted onto it, since both previously inserted at the same |
Drops the labored cross-table-atomicity wording for a one-line "run while live delivery is quiet" note. Part of #49 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Store.Reset takes a progress callback, invoked per scan page (so the count ticks up as a large table drains) and once per table on completion. The reset command passes a text-mode callback that writes a single rewriting stderr line (clearing N/M <table> K deleted), matching the replay command's progress style. JSON mode passes no callback, so machine output stays clean. Part of #49 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
trailtool reset, the first of the two reset-and-replay primitives from #49 (design in #54,docs/design/cloudtrail-replay.md§4).Reset empties every derived DynamoDB table (people, sessions, roles, resources, accounts, services, relations, identity-links) plus the
ingested-filesmarkers, so the projection can be rebuilt from the CloudTrail log by a later replay. The log is never touched, so a reset is always recoverable by replaying it.What it does
Store.Resettruncates each table in place: a key-onlyScan(projecting just the primary-key attributes, discovered from the table's schema so no per-table key layout is hardcoded) feeds batchedBatchWriteItemdeletes. The table and its GSIs stay intact, so live ingestion resumes the moment reset finishes.ingested-filesmarkers are cleared with the rest. Leaving them would make a subsequent replay skip every object and rebuild nothing.BatchWriteItemleftovers (UnprocessedItems, a successful response the SDK does not auto-retry) re-submit with capped exponential backoff, honoring context cancellation.Safety
yes.--yesskips the prompt for scripting. Empty/EOF input cancels.Counts in the prompt come from
DescribeTable'sItemCount, which DynamoDB refreshes roughly every six hours, so they are estimates (shown with~). Tests cover the confirmation guard (decline / EOF / yes /--yes), JSON output, table coverage, and the key-projection and chunking helpers.Replay (the
trailtool replayhalf) is the next slice.Part of #49.