Add batchextract command for parallel package extraction#6
Merged
thesprockee merged 2 commits intomainfrom Mar 14, 2026
Merged
Conversation
Extracts all packages from a _data directory in one command, using a configurable worker pool (default: runtime.NumCPU()). batchextract -data ./rad/_data -output ./extracted batchextract -data ./rad/_data -output ./extracted -workers 8 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new batchextract CLI command to batch-extract every package referenced by manifests in an EVR _data directory, using a configurable parallel worker pool and a simple progress display.
Changes:
- Introduces
cmd/batchextractcommand with flags for-data,-output,-workers, and-verbose. - Enumerates manifest files and extracts packages concurrently via a worker pool.
- Aggregates results into a final summary with per-package failure reporting.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+23
to
+36
| var ( | ||
| dataDir string | ||
| outputDir string | ||
| workers int | ||
| filterType string | ||
| verbose bool | ||
| ) | ||
|
|
||
| func init() { | ||
| flag.StringVar(&dataDir, "data", "", "Path to _data directory containing manifests/ and packages/") | ||
| flag.StringVar(&outputDir, "output", "", "Output directory for extracted files") | ||
| flag.IntVar(&workers, "workers", runtime.NumCPU(), "Number of parallel extraction workers") | ||
| flag.StringVar(&filterType, "filter", "", "Only extract files of this type symbol (hex, e.g. beac1969cb7b8861)") | ||
| flag.BoolVar(&verbose, "verbose", false, "Print each package as it is extracted") |
Comment on lines
+71
to
+73
| if err := os.MkdirAll(outputDir, 0755); err != nil { | ||
| return fmt.Errorf("create output directory: %w", err) | ||
| } |
Comment on lines
+163
to
+169
| dest := filepath.Join(outputDir, name) | ||
| if err := os.MkdirAll(dest, 0755); err != nil { | ||
| return 0, fmt.Errorf("create output dir: %w", err) | ||
| } | ||
|
|
||
| if err := pkg.Extract(dest); err != nil { | ||
| return 0, fmt.Errorf("extract: %w", err) |
Comment on lines
+124
to
+131
| } else { | ||
| totalFiles.Add(int64(r.files)) | ||
| if verbose { | ||
| fmt.Printf("[%d/%d] %s (%d files)\n", done.Load(), total, r.name, r.files) | ||
| } else { | ||
| fmt.Printf("\r[%d/%d] extracting... ", done.Load(), total) | ||
| } | ||
| } |
Comment on lines
+83
to
+98
| fmt.Printf("Found %d manifests, extracting with %d workers...\n", len(names), workers) | ||
| start := time.Now() | ||
|
|
||
| // Feed work through a channel | ||
| work := make(chan string, len(names)) | ||
| for _, n := range names { | ||
| work <- n | ||
| } | ||
| close(work) | ||
|
|
||
| results := make(chan result, len(names)) | ||
| var wg sync.WaitGroup | ||
|
|
||
| for range workers { | ||
| wg.Add(1) | ||
| go func() { |
| names = append(names, e.Name()) | ||
| } | ||
| } | ||
|
|
…e empty manifests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
thesprockee
added a commit
that referenced
this pull request
Mar 14, 2026
* origin/main: Add batchextract command for parallel package extraction (#6) chore: prepare v1.0.0 release Add test coverage for analyze and inventory modes Add analyze and diff modes to evrtools (#10) Add symhash tool and pkg/hash for EVR symbol hash computation (#8) # Conflicts: # cmd/symhash/main.go # pkg/hash/hash_test.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cmd/batchextract— a new CLI tool that extracts all packages from a_datadirectory in one commandruntime.NumCPU())[N/M] extracting...) with a final summaryUsage
Addresses the Phase 2 gap from EXTRACTION_PLAN.md — previously required running
evrtools -mode extractonce per package.Test plan
_datadirectory with multiple manifests-verboseflag shows per-package file counts-workers 1(serial) and-workers 16(parallel)🤖 Generated with Claude Code