Finder And Orchestrator

Finder and Orchestrator

Two scripts for working across many repositories instead of one file.

scraper.py, the Finder

Searches GitHub for Python repositories matching your criteria and writes the matches to JSON.

python3 scraper.py --max-repos 10 --max-workers 1 \
  --min-lines 700 --max-lines 1500 --quality-threshold 20

Initial Rate Limit Status: Core Remaining: 4470, Search Remaining: 30
Searching repositories...   100%
Processing repositories...  100%
2 Items added to JSON
0 Items skipped (already present)
Results saved to 20250218-074839-10repos-Min700-Max1500-Quality20.0-20240219-20250218.json

Filters include file size in lines, a quality score, and the last update date via --start-date and --end-date. The output filename encodes the search, which is what makes a run reproducible later.

Watch the search rate limit. GitHub's code search budget is small, 30 requests in the example above against 4470 for the core API, and it is the one that runs out first. --max-workers 1 is a sensible default for that reason; raising it mostly buys you a longer backoff.

process.py, the Orchestrator

Takes the Finder's JSON and runs the Optimizer over each entry.

python process.py --input output.json --token $GITHUB_TOKEN \
  --config config.toml --branch main --output results.json --fork

Results go to results.json, one record per repository and file.

Doing this responsibly

This combination opens pull requests on other people's repositories, at whatever rate you set. Before running it beyond your own account:

• Start with --dry-run in the config or on the command line, and read the diffs the Optimizer produces. An LLM refactor that passes static analysis can still be wrong
• Keep forking on. Work on your own copy; the pull request comes from there
• Keep the volume low. Unsolicited machine-generated pull requests are unwelcome on most projects, and arriving in bulk is how a well-meant contribution becomes spam
• Read what you send. Your name is on the pull request, not the model's

The Finder's quality thresholds select repositories that look neglected. That is exactly the set where an automated pull request is most likely to be ignored, and where a wrong one is least likely to be caught by a reviewer.