Skip to content

Make label_events writes idempotent on backfill (#25)#26

Open
hunnyboy1217 wants to merge 1 commit intoentrius:testfrom
hunnyboy1217:fix/25-idempotent-label-events
Open

Make label_events writes idempotent on backfill (#25)#26
hunnyboy1217 wants to merge 1 commit intoentrius:testfrom
hunnyboy1217:fix/25-idempotent-label-events

Conversation

@hunnyboy1217
Copy link
Copy Markdown

Summary

Make label_events writes idempotent so backfill re-runs (and BullMQ retries) no longer duplicate rows.

The webhook + backfill paths called labelEventRepo.save() without a natural-key conflict path. Because LabelEvent.id is @PrimaryGeneratedColumn() and label_events had no UNIQUE constraint, every backfill INSERTed a fresh row for every label event already in the table. The pr_labels_by_actor / issue_labels_by_actor views collapse duplicates via DISTINCT ON, so API output stayed correct while the table grew unbounded — eventually slowing the miners API as view scans degrade.

Changes:

  1. packages/db/11_label_events_dedup.sql — one-shot DELETE collapsing exact-duplicate rows by (repo_full_name, target_number, target_type, label_name, action, timestamp), keeping the lowest id. Idempotent (no-op on already-clean tables) and safe on fresh installs. Includes a batched form in the comments for very large production tables.
  2. packages/db/12_label_events_constraints.sqlCREATE UNIQUE INDEX ... NULLS NOT DISTINCT on the natural key. The CONCURRENTLY variant for online application to a running database is documented in the file's comments.
  3. packages/das/src/webhook/github-fetcher.service.tssaveLabelTimelineEvents: per-node repo.save() loop replaced with a single batched insert().values(rows).orIgnore().execute(). One round-trip per timeline instead of N, and ON CONFLICT DO NOTHING makes re-runs no-ops.
  4. packages/das/src/webhook/handlers/label.handler.ts — same save → insert().orIgnore() swap on the webhook write path. Defense-in-depth alongside the existing webhook_deliveries delivery-id dedup.

Deploy order for existing production databases:

  1. Deploy the code first (new orIgnore() is a no-op behavior change without the constraint).
  2. Apply 11_label_events_dedup.sql.
  3. Apply 12_label_events_constraints.sql (use the CONCURRENTLY variant in the file comments).

Reversing steps 2/3 before deploying the code would cause the old save() calls to throw on unique-violations and fail backfill jobs.

Related Issues

Fixes #25

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Documentation
  • Other (describe below)

Testing

No test framework exists in this repo; verified manually.

Reproduction (matches issue #25 steps):

# 1. Register a repo with at least one labeled PR/issue
curl -X POST .../api/v1/admin/repos/register \
     -H "x-api-key: ..." \
     -d '{"repoFullName":"owner/repo"}'

# 2. Wait for backfill, then snapshot
docker exec das-postgres psql -U das -c \
  "SELECT COUNT(*) FROM label_events;"

# 3. Trigger a second backfill
curl -X POST .../api/v1/admin/backfill \
     -H "x-api-key: ..." \
     -d '{"repoFullName":"owner/repo"}'

# 4. Recount
docker exec das-postgres psql -U das -c \
  "SELECT COUNT(*) FROM label_events;"
  • Before fix: count doubles on step 4.
  • After fix: count is unchanged on step 4.

Build / lint:

  • npm run build — passes (NestJS compiler, full TS typecheck)
  • npm run lint — clean
  • npm run format:check — clean

Out of scope (separate follow-ups):

  • Webhook handler uses new Date().toISOString() while backfill uses GraphQL LabeledEvent.createdAt. The same logical labeling action can be represented by two rows with slightly different timestamps, which won't collide on the new UNIQUE. Bounded leak (≈1 extra row per labeling action that occurs while the app is running, per backfill that follows). Proper fix is to drop the webhook's label_events write entirely and enqueue a refresh job — architectural change deserving its own PR.
  • DISTINCT ON (repo, target, label_name) ORDER BY timestamp DESC in the labels views still seq-scans after dedupe. A purpose-built index would help, but is a separate perf PR.

Checklist

  • I have read the Contributing Guide
  • Code builds without errors
  • New and existing tests pass (if applicable) — n/a, no test suite in the repo
  • Documentation updated (if applicable) — operational deploy order documented inline in the SQL files
  • No unnecessary dependencies added

@xiao-xiao-mao xiao-xiao-mao Bot added the bug Something isn't working label May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Non-idempotent label event insertion duplicates rows on every backfill

1 participant