fix(#705): paginate all open issues in pre-scribe backlog fetch - #708
fix(#705): paginate all open issues in pre-scribe backlog fetch#708fullsend-ai-coder[bot] wants to merge 1 commit into
Conversation
Replace `gh issue list --limit 1000` with `gh api --paginate` to fetch all open issues regardless of repo size. The previous limit caused the scribe agent to miss issues not updated recently on repos with >1000 open issues, leading to duplicate issue creation. The REST API `/issues` endpoint includes pull requests, so the pipeline filters them with `select(.pull_request == null)` and maps `html_url` to `url` to match the expected backlog format. Add `open_issue_total` and `backlog_truncated` metadata fields to `scribe-meta.json` for observability of backlog coverage. Add `pre-scribe-test.sh` with tests covering pagination, PR filtering, body truncation, empty repos, and metadata fields. Note: pre-commit hooks could not run (network restrictions in sandbox). shellcheck was not available. The post-script runs an authoritative pre-commit on the runner. Closes #705
|
🤖 Finished Review · ✅ Success · Started 3:12 PM UTC · Completed 3:27 PM UTC |
ReviewFindingsMedium
Low
Labels: PR modifies scribe agent pre-script (scripts/pre-scribe.sh) and adds scribe-specific tests |
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
| @@ -176,7 +179,7 @@ if [[ "${DOC_COUNT}" -eq 0 ]]; then | |||
| --argjson closed_count "${CLOSED_COUNT}" \ | |||
There was a problem hiding this comment.
[low] incomplete-metadata-update
backlog_truncated is hardcoded to false and open_issue_total is set to the same value as backlog_issues. Since --paginate fetches all issues, truncation cannot occur, making these values correct. However, neither field can ever carry a different value — the new fields add no dynamic runtime information beyond what backlog_issues already provides.
|
|
||
| local escaped_fixture="${fixture_file//\//\\/}" | ||
| perl -pi -e "s/FIXTURE_PLACEHOLDER/${escaped_fixture}/g" "${MOCK_BIN}/gh" | ||
| chmod +x "${MOCK_BIN}/gh" |
There was a problem hiding this comment.
[low] test-coverage-gap
The run_backlog_fetch function duplicates the gh api/jq pipeline from pre-scribe.sh rather than sourcing the actual script. This is a reasonable trade-off given Drive credential requirements, but future pipeline changes in pre-scribe.sh require a parallel update in the test.
|
|
||
| if [[ -n "${JQ_EXPR}" ]]; then | ||
| jq -r "${JQ_EXPR}" "${FIXTURE}" | ||
| else |
There was a problem hiding this comment.
[low] test-robustness
build_mock_gh escapes only forward slashes in the fixture path before Perl regex substitution. Characters special to Perl regex (., +, $) in mktemp paths could theoretically cause failures, though in practice Linux mktemp paths are safe.
fullsend-ai-review
Bot
added
requires-manual-review
|
/fs-fix |
1 participant
Summary
gh issue list --limit 1000withgh api --paginateinscripts/pre-scribe.shto fetch all open issues, eliminating the truncation that caused duplicate issue creation on repos with >1000 open issuesopen_issue_totalandbacklog_truncatedmetadata fields toscribe-meta.jsonfor backlog coverage observabilityscripts/pre-scribe-test.shwith 7 tests covering pagination, PR filtering, body truncation, empty repos, null bodies, metadata fields, and label/milestone preservationContext
The scribe agent's pre-script used
gh issue list --limit 1000which returns issues ordered by most recently updated. On repos with >1000 open issues (e.g., ~1,879 at time of failure), issues not updated recently fell out of the agent's matching context. The agent then filed duplicate new issues for topics that already had trackers — even when meeting notes acknowledged those existing issues.The fix uses
gh api --paginatewith the REST API, which followsLinkheader pagination to fetch all pages. The REST/issuesendpoint includes pull requests in the response, so the pipeline filters them withselect(.pull_request == null)and mapshtml_url→urlfor format compatibility.Test plan
pre-scribe-test.shpasses all 7 testspost-scribe-test.shpasses (one pre-existing failure due to missingbcin sandbox — unrelated)backlog.jsonscribe-meta.jsonincludesopen_issue_totalandbacklog_truncatedfieldsCloses #705
Post-script verification
agent/705-paginate-backlog-issues)91f61f3441baedf3f912c9afd4bd574c98793b96..HEAD)