Skip to content

feat(selfhost): use conditional REST validators for backfill segments#1901

Merged
JSONbored merged 1 commit into
JSONbored:mainfrom
oktofeesh1:codex/conditional-backfill-rest-validators
Jun 30, 2026
Merged

feat(selfhost): use conditional REST validators for backfill segments#1901
JSONbored merged 1 commit into
JSONbored:mainfrom
oktofeesh1:codex/conditional-backfill-rest-validators

Conversation

@oktofeesh1

Copy link
Copy Markdown
Contributor

Summary

  • Adds conditional REST validator support to queued open-data backfill segment reads.
  • Handles 304 Not Modified as a fresh terminal segment while preserving counts, cursors, validators, and current-open scan markers.
  • No issue because this is a narrow self-hosted backfill optimization with no schema, config, UI, or deployment-surface change.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked an issue, or this is small enough that the summary explains why an issue is not needed.

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage locally; codecov/patch requires ≥97% coverage of the lines AND branches you changed (aim for 98%+ on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.
  • npm run test:workers
  • npm run build:mcp
  • npm run test:mcp-pack
  • npm run ui:openapi:check
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:build
  • npm audit --audit-level=moderate
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

  • None. npm run test:ci passed after rebasing on upstream main; the scoped test/unit/backfill.test.ts suite also passed.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
  • API/OpenAPI/MCP behavior is updated and tested where needed.
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
  • Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

UI Evidence

Not applicable; backend-only queue/backfill behavior change.

Notes

  • Reuses stored ETag / Last-Modified values only for safe single-page segment validation.
  • Keeps current-open scan resume markers intact by using Last-Modified instead of replaying the marker as an ETag.
  • Treats not-modified segments as fresh in repository sync state and still schedules PR detail hydration for unchanged open PR lists.

@loopover-orb

loopover-orb Bot commented Jun 30, 2026

Copy link
Copy Markdown

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-06-30 23:44:08 UTC

2 files · 1 AI reviewer · no blockers · readiness 73/100 · CI green · unknown

⏸️ Suggested Action - Manual Review

  • Touches a guarded path — held for manual review

Review summary
This change adds conditional REST validators to single-page backfill segment reads and treats 304 responses as fresh terminal segment rows while preserving prior counts, cursors, and validators. The visible implementation is coherent: it gates conditional requests to fresh single-page segments, avoids sending the current-open marker as an ETag, and refreshes repo sync timestamps for not_modified rows. The added tests exercise the important paths: initial 200, 304 reuse, unauthenticated fallback, changed totals, and paginated-segment bypass.

Nits — 6 non-blocking
  • nit: src/github/backfill.ts:1260 relies on lastCursor/nextCursor to identify a single-page segment but does not also require previous.pageCount === 1, so an inconsistent stored row could still send validators when the stored page count says otherwise.
  • nit: src/github/backfill.ts:1320 intentionally uses Last-Modified-only validation for current-open PR scans because the ETag column stores the current-open marker; that is worth a short comment because it is weaker than the ETag path and easy to misread later.
  • In src/github/backfill.ts:1260, add previous.pageCount === 1 to conditionalRequestForSegment so the eligibility check matches the stated single-page invariant directly.
  • In src/github/backfill.ts:1320, document why current-open scans cannot send If-None-Match here, or split the marker from the stored REST validator in a follow-up so current-open scans can use ETags too.
  • Readiness score is below the configured threshold — Use the readiness panel as advisory maintainer context; the score does not block this PR.
  • Touches a guarded path — held for manual review — A maintainer must review and merge this change.
Signal Result Evidence
Code review ✅ No blockers 1 reviewer
Linked issue ✅ No-issue rationale PR body explains why no issue is linked.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (size label size:L; no linked issue context).
Validation posture ❌ 5/25 Preflight is holding this PR; address the blocker before review.
Contributor workload ✅ 10/10 Author activity: 73 registered-repo PR(s), 61 merged, 6 issue(s).
Contributor context ✅ Confirmed Gittensor contributor oktofeesh1; Gittensor profile; 73 PR(s), 6 issue(s).
Gate result ⚠️ Not blocking Advisory; not blocking this PR.
Review context
  • Author: oktofeesh1
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository registration is not available in the local Gittensory cache.
  • Public profile languages: not available
  • Official Gittensor activity: 73 PR(s), 6 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Fix the blocker.
  • Triage stale or unlinked PRs.
  • Refresh registry data or choose a registered active repo.
  • Link the issue being solved, or explicitly explain why this is a no-issue PR.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot added gittensor gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. labels Jun 30, 2026
@dosubot dosubot Bot added the lgtm label Jun 30, 2026
@JSONbored
JSONbored merged commit ebae742 into JSONbored:main Jun 30, 2026
7 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in gittensory - v1 roadmap Jun 30, 2026
JSONbored added a commit that referenced this pull request Jul 1, 2026
…dence (#1942) (#1976)

#1901 added ETag/If-Modified-Since validators for backfill segments, but
fetchPagedSegment loaded the prior segment (and so built the conditional
request) only when mode===resume. The scheduled cadence runs light/full, so
the 304 fast-path never fired on the path that actually runs — every 30-min
light crawl re-listed labels/open-issues/open-PRs at full cost.

Load the prior segment for every mode so a light/full crawl sends If-None-Match
and an unchanged single-page list returns a 0-body 304. Resume pagination stays
gated on canResumePreviousScan; open-scan reconciliation segments still force
allowEtag:false, so close-detection is unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants