feat: reconcile popular repositories weekly

.github/workflows/seed-popular.yml

@@ -2,7 +2,8 @@ name: Seed popular repos
 
 on:
   schedule:
-    - cron: '0 3 * * *' # 03:00 UTC daily
+    # Weekly, away from the top-of-hour GitHub Actions load spike.
+    - cron: '17 3 * * 0' # 03:17 UTC every Sunday
   workflow_dispatch:
     inputs:
       daily_limit:
@@ -13,6 +14,10 @@ on:
         description: 'Repos to inspect for tool intelligence'
         required: false
         default: '250'
+      max_additions:
+        description: 'Abort-before-write repository addition bound (0-100)'
+        required: false
+        default: '100'
 
 concurrency:
   group: starboard-embedding
@@ -47,16 +52,18 @@ jobs:
       - name: Run seed script
         run: pnpm db:seed-popular
         env:
-          # The runner token has enough GitHub Search quota for a bounded walk.
+          # The runner token has enough GitHub Search quota for one complete,
+          # rate-paced catalog reconciliation.
           # Keep this independent of long-lived PAT rotation so a stale secret
           # cannot break a scheduled seed run with 401 Bad credentials.
           GITHUB_TOKEN: ${{ github.token }}
           CLOUDFLARE_ACCOUNT_ID: ${{ vars.CLOUDFLARE_ACCOUNT_ID }}
           D1_DATABASE_ID: ${{ vars.D1_DATABASE_ID }}
           CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
           SEED_EMBED_MODE: 'worker'
-          # Operational row-read circuit breaker. The script also hard-caps this at 25.
-          SEED_METADATA_PAGE_LIMIT: '10'
+          # Fail before repository writes when the source delta is anomalous.
+          SEED_MAX_ADDITIONS: ${{ github.event.inputs.max_additions || '100' }}
+          SEED_MIN_SOURCE_REPOS: '5000'
           MIN_STARS_FLOOR: '5000'
 
       - name: Embed refreshed repos through Worker bindings

AGENTS.md

@@ -36,7 +36,7 @@ pnpm test:e2e          # playwright
 pnpm lint              # biome check .
 pnpm db:migrate        # apply migrations/* to isolated local D1
 pnpm db:migrate:remote # validate config + apply migrations/* to remote D1 (approval required)
-pnpm db:seed-popular   # cold-seed popular repos (≥5k stars) — used by daily GH Action
+pnpm db:seed-popular   # reconcile missing popular repos (≥5k stars) — weekly GH Action
 pnpm db:seed-embeddings# backfill repo_embeddings
 pnpm docs:check        # validate docs/ links + structure
 pnpm docs:dev          # blume dev (local docs site; requires pnpm add -D blume)

PROJECT_STATUS.md

@@ -1,6 +1,6 @@
 # starboard — PROJECT STATUS
 
-Last updated: 2026-08-09
+Last updated: 2026-08-13
 
 ## Why/What
 
@@ -28,7 +28,7 @@ installation, alerts, reports, digest email, and stack generation.
 | Client state | SWR (data), nuqs (URL-backed filters/sort) |
 | AI / search | Cloudflare Workers AI `@cf/baai/bge-base-en-v1.5` (768d); optional `knowledgebase` Worker via service binding |
 | Deploy | Cloudflare Workers via OpenNext (`@opennextjs/cloudflare`) |
-| CI | GitHub Actions — push CI + manual SHA-tagged deploy + daily bounded seed/enrich/embed |
+| CI | GitHub Actions — push CI + manual SHA-tagged deploy + weekly complete additions-only seed/enrich/embed |
 
 **Local dev:** `pnpm install && cp .env.example .env.local && pnpm dev` → http://localhost:3000
 
@@ -66,12 +66,21 @@ provenance. The workflow is free and has no billing or entitlement gate.
 | Secrets | `AUTH_SECRET`, `AUTH_GITHUB_ID`, `AUTH_GITHUB_SECRET`; `AI_GATEWAY_API_KEY` for authenticated operator jobs; `RAG_SERVICE_KEY` for relevance RAG. Any legacy unused `TURSO_*` bindings are separate credential-cleanup work. |
 | Embedding model | `@cf/baai/bge-base-en-v1.5` — change model, dimension, and replacement Vectorize index together |
 | Project connections | Additive `0003_user_projects.sql`; remote migration requires explicit approval before application rollout |
-| Data refresh jobs | Daily bounded `seed-popular` at 03:00 UTC plus manual seed/enrich/embed dispatches |
+| Data refresh jobs | Weekly complete additions-only `seed-popular` at Sunday 03:17 UTC plus manual seed/enrich/embed dispatches |
 | Deploy | `pnpm deploy:cf` or manual `deploy.yml` dispatch; both attach the full Git SHA |
 | Smoke | `pnpm test` + `pnpm build`; for search/DB changes also `pnpm db:migrate` and `pnpm build:cf` |
 
 ## Timeline
 
+- **2026-08-13 (weekly catalog reconciliation implemented locally)** — Replaced
+  the multi-day popular-repository cursor walk with one complete weekly GitHub
+  identity reconciliation. Immutable creation-date partitions fit in single
+  Search responses, all stored D1 IDs are read once, and only source-only
+  additions are fetched and inserted. Incomplete or unstable source evidence
+  and deltas above 100 fail before writes; deletion and existing-row refresh
+  remain out of scope. Focused reconciliation, D1 regression, and refresh
+  evidence tests pass; production activation awaits normal review and push.
+
 - **2026-08-09 (discovery entry and cache isolation complete locally)** — Made
   Discover the default generic sign-in destination, removed the 100-repository
   GitHub picker cap through safe pagination, tightened Discover card sizing and
@@ -199,7 +208,7 @@ provenance. The workflow is free and has no billing or entitlement gate.
 | Repo intelligence | Shared-shell repository detail (`/explore`), similar projects, tool evidence, public shared lists, legal/marketing shell |
 | Semantic search | knowledgebase Worker integration for relevance search; README-backed sync ingest; local embeddings retained for non-RAG Starboard features |
 | Connected projects | Shipped public GitHub project connections, public preview, GitHub picker, and evidence-based repository and tool recommendations |
-| Discovery & tools | Public Discover, daily bounded seed/enrich/embed with manual dispatch, stored growth sorting, and Tool Intelligence |
+| Discovery & tools | Public Discover, weekly complete additions-only seed/enrich/embed with manual dispatch, stored growth sorting where snapshot history exists, and Tool Intelligence |
 | Removed 2026-08-08 | Fleet project catalog, Alerts, Reports, Stack Builder, standalone Radar, weekly digest |
 | Ops hardening (2026-06-20) | `.env.example`, Vitest + Playwright path, pre-push lint, self-contained TypeScript/Astro landing for green CF builds |
 
@@ -264,8 +273,9 @@ provenance. The workflow is free and has no billing or entitlement gate.
   collection controls but is not required to browse, search, sort, filter,
   paginate, or open repo details.
 - Discover supports paginated 30-day growth ordering and detected-tool facets from indexed local snapshot/tool tables.
-- Daily bounded GitHub Actions seed/enrich popular repos in D1 and embed through
-  native Worker bindings; manual dispatch remains available for operator checks.
+- Weekly complete, additions-only GitHub Actions reconciliation inserts missing
+  popular repos in D1 and embeds through native Worker bindings; manual dispatch
+  remains available for operator checks.
 - Star history and fastest-grower APIs/surfaces: `/api/repos/[repoId]/star-history`, `/api/growth`, Discover growth sorting, and repo-detail mini history from stored `repo_star_snapshots`.
 - Tool Intelligence: additive `repo_tools` index, `/api/tools`, `/api/repos/[repoId]/tools`, `/tools`, and `pnpm db:enrich-tools` for bounded SBOM/tree/manifest-based detection with source/confidence labels. Repository evidence is server-filtered and paginated in 48-item pages. Accuracy disclaimer is shown in-product because manifest/SBOM evidence is stronger than README/topic/metadata inference and C/C++ monorepos vary.
 - SaaS Maker feedback widget integrated; product analytics run directly through PostHog.

docs/architecture/data-flow.md

@@ -15,16 +15,19 @@ filtered by `session.user.githubId`.
 ## Public discovery
 
 ```text
-daily or manually dispatched seed-popular job
-  → GitHub Search metadata
-  → repos + star snapshots in D1
+weekly or manually dispatched seed-popular job
+  → complete, creation-date-partitioned GitHub Search identity set
+  → diff against all stored D1 repository IDs
+  → fetch details and insert source-only additions + initial snapshots
   → bounded embedding and tool-enrichment jobs
   → Vectorize + repo_tools
   → public Discover and Tool Intelligence
 ```
 
-Discover requests do not synchronously call GitHub. Growth uses stored snapshot
-history and remains empty when insufficient samples exist.
+Discover requests do not synchronously call GitHub. The weekly catalog job is
+additions-only: it does not delete or refresh existing repository rows. Growth
+uses snapshots captured at ingest or user sync and remains empty when
+insufficient samples exist.
 
 ## Connected projects
 

docs/development/commands.md

@@ -41,7 +41,7 @@ The `--webpack` flag remains the verified OpenNext production build path.
 | `pnpm db:convert-turso-vectors` | Validate/convert extracted 768-d vectors to Vectorize NDJSON |
 | `pnpm db:snapshot-turso-logical starboard` | Produce a row-free deterministic Turso digest for final cutover reconciliation |
 | `pnpm db:enrich-repos` | `tsx scripts/enrich-repos.ts` — AI metadata enrichment |
-| `pnpm db:seed-popular` | `tsx scripts/seed-popular.ts` — cold-seed popular repos (≥5k stars) |
+| `pnpm db:seed-popular` | `tsx scripts/seed-popular.ts` — fully reconcile and add missing popular repos (≥5k stars) |
 | `pnpm db:enrich-tools` | `tsx scripts/enrich-tools.ts` — SBOM/tree/manifest tool detection |
 
 ## Testing

docs/operations/data-map.md

@@ -28,16 +28,16 @@ retirement requires separate approval.
 | Store | Class | Owner | Reconstruction | Expected cost | Last verified |
 | --- | --- | --- | --- | --- | --- |
 | D1 `users` | irreplaceable-user | NextAuth GitHub OAuth | Not reconstructable — GitHub is the source of identity, but user records (email, created_at) must be exported | n/a — export required | 2026-08-02 |
-| D1 `repos` (popular ≥5k seeded) | authoritative-source | `scripts/seed-popular.ts` | Re-walk GitHub Search ≥`MIN_STARS_FLOOR` | ~hours (rate-limited, resumable cursor) | 2026-08-02 |
+| D1 `repos` (popular ≥5k seeded) | authoritative-source | `scripts/seed-popular.ts` | Fully reconcile creation-date-partitioned GitHub Search ≥`MIN_STARS_FLOOR`; insert source-only IDs | ~15–30 minutes | 2026-08-13 |
 | D1 `user_repos` (starred/saved state) | irreplaceable-user | GitHub sync via `/api/stars/sync` | Re-sync from GitHub starred list (ETag + HTML scrape) | ~seconds per user | 2026-08-02 |
 | D1 `user_projects` | irreplaceable-user | Project connection UI/API | Reconnect public GitHub repositories manually | ~seconds per user | 2026-08-08 |
 | D1 `user_lists`, `user_repo_lists` | irreplaceable-user | User UI actions | Not reconstructable — user-curated collections | n/a — export required | 2026-08-02 |
 | D1 `comments`, `likes`, `comment_votes` | historical inactive user storage | Removed discussion UI; legacy user actions | Not reconstructable — retained for export until a separately approved retention migration | n/a — export required | 2026-08-09 |
 | D1 `repo_embeddings` hashes + Vectorize `starboard-repos` values | derived | Worker binding embedding jobs | Re-embed from `repos` + `repo_ai_metadata` text via Workers AI | ~minutes (Workers AI quota) | 2026-08-02 |
 | D1 `repo_ai_metadata` | derived | `scripts/enrich-repos.ts` (free-ai gateway) | Re-enrich from `repos` metadata via AI | ~minutes per batch | 2026-08-02 |
 | D1 `repo_tools` | derived | `scripts/enrich-tools.ts` | Re-detect from GH tree/manifest/SBOM | ~minutes per batch | 2026-08-02 |
-| D1 `repo_star_snapshots`, `repo_threshold_events` | derived | `seed-popular.ts` snapshot inserts | Re-derive from `repos` star counts over time | rebuilt on each seed run | 2026-08-02 |
-| D1 `seed_cursor` | derived (walk state) | `seed-popular.ts` | Reset to defaults; walk restarts from top | seconds | 2026-08-02 |
+| D1 `repo_star_snapshots`, `repo_threshold_events` | derived | Seed and user-sync snapshot inserts | Re-derive initial samples from `repos`; historical series requires repeated user sync | additions only during weekly reconciliation | 2026-08-13 |
+| D1 `seed_cursor` | legacy derived state (inactive) | No active writer | Safe to ignore; retained to avoid a destructive migration | n/a | 2026-08-13 |
 | D1 `insight_reports`, `user_alert_preferences` | historical inactive storage | Removed product features | Retained to avoid a destructive migration; no active writers | n/a | 2026-08-08 |
 | Cloudflare Worker `starboard` (deployed bundle) | cache | `pnpm deploy:cf` or manual deploy workflow | Rebuild + redeploy | ~minutes | 2026-07-18 |
 | knowledgebase Worker RAG index (`STARBOARD_RAG_INDEX_ID`) | derived (RAG index of user repos) | `src/lib/knowledgebase.ts` ingest | Re-ingest from `repos` + README text per user | ~seconds per user | 2026-07-18 |
@@ -67,10 +67,11 @@ window.
 ### Full popular-pool rebuild (bounded)
 
 `pnpm db:migrate:remote` → scheduled or manual `seed-popular` workflow
-(resumable cursor walk of GitHub Search ≥5k stars plus bound Worker embedding backfill) →
-`pnpm db:enrich-tools` (tool detection). Total runtime
-is bounded by `SEED_METADATA_PAGE_LIMIT` (default 10, hard cap 25 pages/run)
-and the workflow's `daily_limit` input (default 1000 embeddings/run). See
+(complete one-response date partitions of GitHub Search ≥5k stars, D1 ID diff,
+source-only inserts, and bounded Worker embedding backfill) →
+`pnpm db:enrich-tools` (tool detection). The pre-write
+`SEED_MAX_ADDITIONS` bound defaults to 100 in the workflow, and `daily_limit`
+defaults to 1000 embeddings per run. See
 [`jobs.md`](jobs.md) §seed-popular.
 
 ### Embedding dimension drift
@@ -92,15 +93,15 @@ Each scheduled or dispatched `seed-popular` GitHub Action records a structured
 manifest at `data/refresh-manifest.json` and copies it to the existing GitHub
 Actions run summary before the ephemeral runner is discarded. The manifest includes:
 
-- `source_watermark` — GitHub Search cursor (`next_max_stars`/`next_page`)
-  and run timestamp
-- `bounds` — `METADATA_PAGE_LIMIT`, `DAILY_LIMIT`, `MIN_STARS_FLOOR`
+- `source_watermark` — verified GitHub unique-ID count
+- `bounds` — source/stored/addition/stored-only counts,
+  `SEED_MAX_ADDITIONS`, `SEED_MIN_SOURCE_REPOS`, and `MIN_STARS_FLOOR`
 - `timeout` — workflow `timeout-minutes: 60`
-- `idempotency` — `INSERT … ON CONFLICT(id) DO UPDATE` for `repos`;
-  `INSERT OR IGNORE` for `repo_star_snapshots` and `repo_threshold_events`
+- `idempotency` — in-memory source/stored ID diff plus `INSERT OR IGNORE` for
+  additions; no existing-row update or stored-only deletion
 - `retries` — `withDbRetry` (4 attempts, exponential backoff) for D1;
   `ghSearch` (4 attempts + rate-limit sleep) for GitHub
-- `output_counts` — `upsertedThisRun`, `embedded`, pool totals
+- `output_counts` — inserted additions, embedded additions, and pool totals
 - `quality_signal` — non-zero output check + pool coverage ratio
 - `freshness` — run wall-clock + delta from prior success
 - `failure_state` — unresolved failure state within that run's manifest

docs/operations/env.md

@@ -47,6 +47,12 @@ The `seed-popular` workflow deliberately uses `${{ github.token }}` for GitHub
 Search (not a long-lived PAT) so a stale PAT cannot break a scheduled or manual
 seed run with 401 Bad credentials.
 
+The scheduled reconciliation sets `MIN_STARS_FLOOR=5000`,
+`SEED_MIN_SOURCE_REPOS=5000`, and `SEED_MAX_ADDITIONS=100`. The last value is a
+pre-write Cloudflare budget circuit breaker and is exposed as an explicit
+manual-dispatch input that can lower the bound. Values above the code-level hard
+limit of 100 are rejected before GitHub or D1 access.
+
 ## Public keys
 
 `NEXT_PUBLIC_SAASMAKER_API_KEY` is a public key (expected to be visible in

docs/operations/jobs.md

@@ -6,18 +6,21 @@ annotates intent, inputs, and dependencies.
 
 ## seed-popular (`.github/workflows/seed-popular.yml`)
 
-- **Schedule:** daily at 03:00 UTC and manual `workflow_dispatch`. Automation
-  was restored after the D1 cutover removed the prior Turso row-read allowance;
-  the bounded walk and unchanged-row protections remain in force.
-- **Inputs:** `daily_limit` (default 1000), `tool_enrich_limit` (default 250).
+- **Schedule:** Sundays at 03:17 UTC and manual `workflow_dispatch`. The
+  non-top-of-hour minute reduces GitHub Actions scheduling contention.
+- **Inputs:** `daily_limit` (embedding limit, default 1000),
+  `tool_enrich_limit` (default 250), and `max_additions` (default 100).
 - **Concurrency:** shared group `starboard-embedding`, `cancel-in-progress:
   false`, so seed and standalone backfill cannot duplicate embedding work.
 - **Timeout:** 60 minutes.
 - **Steps:**
   1. `pnpm db:migrate:remote` (approval-gated D1 migrations).
-  2. `pnpm db:seed-popular` (`scripts/seed-popular.ts`) — GitHub Search for
-     repos ≥ `MIN_STARS_FLOOR=5000`, with a resumable cursor in `seed_cursor`.
-     Uses `${{ github.token }}` deliberately so a stale PAT cannot break seeding.
+  2. `pnpm db:seed-popular` (`scripts/seed-popular.ts`) — completely enumerate
+     GitHub Search repos ≥ `MIN_STARS_FLOOR=5000` through non-overlapping
+     creation-date partitions that each fit one response; compare the resulting
+     IDs with one `SELECT id FROM repos`; fetch details and insert only IDs absent
+     from D1. Uses `${{ github.token }}` deliberately so a stale PAT cannot break
+     reconciliation.
   3. Authenticated Worker operator request — Workers AI embeddings → Vectorize
      binding, with drift hashes written through the D1 binding.
   4. `pnpm db:enrich-tools` (`scripts/enrich-tools.ts`) — SBOM/tree/manifest
@@ -26,9 +29,24 @@ annotates intent, inputs, and dependencies.
 - **Credentials:** scoped D1 `CLOUDFLARE_API_TOKEN`, non-secret
   account/database variables, and the existing AI gateway key as the Worker
   operator bearer. GitHub does not receive Vectorize API access.
-- **Safety controls:** metadata walks default to 10 GitHub Search pages and
-  hard-cap at 25; unchanged repos do not update or fire FTS maintenance;
-  snapshots are written only when star counts change.
+- **Completeness controls:** any `incomplete_results`, duplicate identity,
+  truncated date partition, source-count drift, or unique-ID mismatch fails the
+  run before D1 writes. Root source counts are checked before and after the walk.
+- **D1 budget controls:** the job reads all stored IDs once (currently roughly
+  15,000 rows, about 0.3% of Cloudflare's 5 million free daily row-read
+  allowance), then applies `SEED_MAX_ADDITIONS` before detail fetches or writes.
+  The scheduled default is 100 additions against the 100,000 free daily
+  row-write allowance, and code rejects manual values above 100 before GitHub or
+  D1 access. Existing rows are not updated, stored-only rows are not deleted,
+  and new rows are inserted in batches of 50. The existing embedding and tool
+  enrichment steps remain bounded at 1,000 and 250 repositories respectively;
+  because unchanged hashes are skipped, a normal weekly run processes only new
+  or independently changed rows. See Cloudflare's
+  [current D1 pricing](https://developers.cloudflare.com/d1/platform/pricing/).
+- **GitHub budget controls:** Search requests are sequential and paced at 2.1
+  seconds, below the authenticated 30 requests/minute Search bucket. Complete
+  enumeration is expected to use roughly 250 requests, below the workflow
+  token's 1,000 requests/hour per-repository allowance.
 
 ## embed-pending (`.github/workflows/embed-pending.yml`)