Skip to content

feat: add provider-neutral SCM observer#76

Merged
whoisasx merged 20 commits into
mainfrom
feat/75
Jun 4, 2026
Merged

feat: add provider-neutral SCM observer#76
whoisasx merged 20 commits into
mainfrom
feat/75

Conversation

@whoisasx
Copy link
Copy Markdown
Collaborator

@whoisasx whoisasx commented Jun 2, 2026

Closes #75

Summary

add provider-neutral ports.SCMObservation DTOs and lifecycle ApplySCMObservation projection

  • implement SCM observer polling with repo/commit ETag guards, PR detection, GraphQL batching, CI log-tail enrichment, review refresh, semantic hashing, DB writes, and lifecycle notification
  • extend SQLite PR/check/review schema and store transactional SCM persistence
  • wire daemon to start the GitHub-backed SCM observer when GitHub credentials are available
  • Tests

  • go test ./..
  • go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.1.6 run ./...

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Jun 2, 2026

Greptile Summary

This PR introduces a provider-neutral SCM polling observer that owns the full lifecycle of PR/CI/review data: ETag-guarded polling, semantic hashing, transactional SQLite persistence, and lifecycle notification. It also extends the SQLite schema with 30+ new columns, a new pr_review_threads table, updated CDC triggers, and wires the GitHub-backed observer into daemon startup with lazy credential validation.

  • Observer core (observe/scm/observer.go, 1185 lines): implements the poll loop, ETag guards, GraphQL batching of up to 25 PRs, review-cadence scheduling, CI log enrichment, semantic hash diffing, and the two-phase write (pending hash first, real hash after lifecycle succeeds).
  • GitHub provider (adapters/scm/github/observer_provider.go, 682 lines): normalizes GitHub REST/GraphQL payloads into the provider-neutral ports.SCMObservation DTO; includes bounded review-thread pagination (2 pages max) and check-context pagination.
  • Persistence (store/pr_store.go): splits the legacy WritePR path (which now uses UpsertLegacyPR + DeleteLegacyPRComments to avoid clobbering SCM-observer-written rows) from the new WriteSCMObservation path; adds reviewMode flags to control thread/comment replacement vs. merge.

Confidence Score: 4/5

Safe to merge with awareness that most of the complex two-phase write and hash-preservation paths have been addressed in this revision; the remaining observations are minor gaps in edge-case review cadence and log-tail attribution.

The PR is a large, well-structured feature with thorough tests. Issues raised in prior review rounds — review-thread table clobbering, corrupted hashes on review-only refresh, credential gate bypass, transient-error retry, and legacy comment stranding — all appear addressed correctly in this version. The two open findings are: a log-tail fallback in scmToPRObservation that could attribute a combined multi-check log to a single failing check name (confusing agent nudge), and a narrow cadence gap where a review change on an already-approved PR could be missed until the next ETag invalidation. Neither affects data integrity or security, but both touch the agent-nudge path.

backend/internal/lifecycle/reactions.go (log-tail fallback in scmToPRObservation) and backend/internal/observe/scm/observer.go (needsReviewRefresh cadence for non-ChangesRequest reviews) are the two areas worth a second look.

Important Files Changed

Filename Overview
backend/internal/observe/scm/observer.go Core observer: 1185-line orchestration of ETag guards, GraphQL batching, review refresh, semantic hashing, and two-phase DB writes. Hash preservation for local-only review refreshes is in place; credential-check retry logic is correct (credentialsChecked only set on definitive outcome).
backend/internal/adapters/scm/github/observer_provider.go GitHub provider normalization: REST ETag guards, GraphQL batching, check-context pagination, and review-thread pagination bounded at 2 pages. scmChecksFromGraphQL correctly assigns ch.Status inside each switch case with no post-switch overwrite.
backend/internal/storage/sqlite/store/pr_store.go Legacy WritePR now uses ReviewWritePreserve (no review thread delete) and DeleteLegacyPRComments (thread_id='' only) to avoid clobbering SCM-observer rows. WriteSCMObservation uses UpsertPR (full fields).
backend/internal/storage/sqlite/migrations/0004_scm_observer_schema.sql Adds 30+ columns to pr, extends pr_checks/pr_comment, creates pr_review_threads table, safely rebuilds change_log with new CDC event types, and recreates all triggers.
backend/internal/lifecycle/reactions.go Adds ApplySCMObservation → scmToPRObservation projection for the provider-neutral lifecycle path. Correctly filters resolved and bot comments. Log-tail fallback uses combined multi-check tail (P2).
backend/internal/daemon/scm_wiring.go Wires the observer with SkipTokenPreflight=true for lazy startup; credential validation moves to the first background poll via SCMCredentialsAvailable.
backend/internal/ports/scm_observations.go New provider-neutral SCM DTO definitions. Well-documented with field-level comments. ErrSCMNotFound sentinel properly defined.
backend/internal/observe/scm/observer_test.go 833-line test file covering ETag guard decisions, batching, log enrichment, review cadence, credential retry, and dedup semantics using fake collaborators.

Sequence Diagram

sequenceDiagram
    participant D as Daemon
    participant O as Observer (30s tick)
    participant P as GitHub Provider
    participant S as SQLite Store
    participant L as Lifecycle Manager

    D->>O: Start(ctx)
    loop Every 30s
        O->>S: ListAllSessions / GetProject / ListPRsBySession
        O->>P: SCMCredentialsAvailable (first poll only)
        O->>P: RepoPRListGuard (ETag, per repo)
        alt ETag changed or no prior ETag
            O->>P: DetectPRByBranch (for sessions with no known PR)
            O->>P: CommitChecksGuard (ETag, per PR head SHA)
            O->>P: FetchPullRequests (GraphQL batch ≤25 PRs)
            O->>P: FetchFailedCheckLogTail (per failed check)
        end
        alt Review cadence triggered
            O->>P: FetchReviewThreads (2-page max, 50 threads/page)
        end
        O->>O: prepareForPersistence (semantic hash diff)
        alt Changed.Metadata or Changed.CI or Changed.Review
            O->>S: WriteSCMObservation (pending hashes)
            O->>L: ApplySCMObservation → agent nudge
            O->>S: WriteSCMObservation (commit real hashes)
        end
        O->>O: Update ETag/review caches (on success)
    end
Loading

Reviews (5): Last reviewed commit: "fix: harden scm observer review follow-u..." | Re-trigger Greptile

Comment thread backend/internal/adapters/scm/github/observer_provider.go Outdated
Comment thread backend/internal/adapters/scm/github/observer_provider.go
@whoisasx whoisasx requested a review from illegalcall June 2, 2026 07:07
Comment thread backend/internal/daemon/scm_wiring.go
Comment thread backend/internal/storage/sqlite/store/pr_store.go
Comment thread backend/internal/observe/scm/observer.go Outdated
Copy link
Copy Markdown
Collaborator

@illegalcall illegalcall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Detailed SCM observer migration review. I focused on behavior that can change persisted PR/CI/review state or user-visible session status.

Comment thread backend/internal/adapters/scm/github/observer_provider.go Outdated
Comment thread backend/internal/adapters/scm/github/observer_provider.go Outdated
Comment thread backend/internal/observe/scm/observer.go Outdated
Comment thread backend/internal/observe/scm/observer.go
Comment thread backend/internal/observe/scm/observer.go Outdated
Copy link
Copy Markdown
Collaborator

@illegalcall illegalcall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up after a second pass: two more observer-level failure modes that look worth fixing before merge.

Comment thread backend/internal/observe/scm/observer.go Outdated
Comment thread backend/internal/observe/scm/observer.go
Copy link
Copy Markdown
Collaborator

@illegalcall illegalcall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up on latest head b3c10b0: prior fixes address most of the earlier findings, but these two current failure modes still look worth fixing before merge.

Comment thread backend/internal/observe/scm/observer.go
Comment thread backend/internal/adapters/scm/github/observer_provider.go
Copy link
Copy Markdown
Collaborator

@illegalcall illegalcall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up on latest head 481f44a: the CI pagination fix looks reasonable; one lifecycle retry durability issue remains.

Comment thread backend/internal/observe/scm/observer.go Outdated
@whoisasx
Copy link
Copy Markdown
Collaborator Author

whoisasx commented Jun 3, 2026

Implementation plan for the CI/review GraphQL cost tuning discussed in this thread:

CI contexts

  • Reduce the batch GraphQL CI context page from contexts(first: 100) to contexts(first: 20).
  • Keep CI.Summary driven by GitHub's aggregate statusCheckRollup.state, not by the visible first page of contexts.
  • Add pageInfo { hasNextPage endCursor } to the batch query.
  • If hasNextPage=false, persist the fetched contexts directly.
  • If hasNextPage=true, run an explicit fallback query for that PR to fetch the remaining CI contexts, append them, and then recompute checks, failed checks, failed fingerprint, and failure details.
  • If the CI fallback fails, do not persist partial CI state and do not advance hashes/ETags.

Review threads

  • Use fixed provider constants, not new runtime config flags:
    • githubReviewThreadPageSize = 50
    • githubReviewCommentLimitPerThread = 5
    • githubReviewThreadMaxPages = 2
  • Change review fetching to request the latest review window:
    • reviewThreads(last: 50)
    • comments(first: 5) per thread
    • pageInfo { hasPreviousPage startCursor }
  • If hasPreviousPage=false, treat the review snapshot as complete.
  • If hasPreviousPage=true, inspect the oldest thread from the latest 50:
    • if the oldest fetched thread is resolved, skip older pages and treat the fetched set as a partial window to merge;
    • if the oldest fetched thread is unresolved, fetch exactly one older page with reviewThreads(last: 50, before: startCursor).
  • Never fetch more than two review-thread pages total.

Persistence behavior

  • GitHub provider should not read/write the DB directly. It only returns fetched review data and whether the result is partial.
  • Observer chooses the persistence mode:
    • complete review fetch: replace stored review threads/comments;
    • partial review fetch: merge fetched threads/comments while preserving older unseen rows;
    • review fetch failure: preserve existing review rows/hash/decision and retry later.
  • For partial merge:
    • upsert fetched review threads;
    • for each fetched thread, replace comments only for that thread;
    • preserve older unseen threads/comments.

Correctness constraints

  • Do not advance CI hash on partial/failed CI fallback.
  • Do not treat partial review data as a complete replacement.
  • Preserve the existing lifecycle durability behavior: semantic hashes should only advance after lifecycle side effects are accepted.

Tests to add/update

  • CI with <=20 checks does not call fallback.
  • CI with >20 checks calls fallback and persists all checks.
  • CI summary still follows statusCheckRollup.state.
  • CI fallback failure does not persist partial CI state.
  • Review query shape uses last:50 and comments(first:5).
  • Complete review fetch uses replace mode.
  • Partial review fetch uses merge mode.
  • Oldest unresolved review thread triggers exactly one older-page fallback.
  • Review fetch never exceeds two pages.
  • Partial merge preserves older stored rows while updating fetched threads/comments.

@whoisasx
Copy link
Copy Markdown
Collaborator Author

whoisasx commented Jun 3, 2026

@greptileai

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Introduces a provider-neutral SCM observer pipeline (DTOs + polling observer + GitHub provider implementation), persists normalized SCM state (PR metadata, CI checks, review threads/comments, semantic hashes) in SQLite, and wires the observer into the daemon when GitHub credentials are available. This implements the Issue #75 direction of moving the observer→lifecycle contract from provider-specific data / legacy PRObservation to a new ports.SCMObservation plus ApplySCMObservation.

Changes:

  • Add provider-neutral SCM DTOs (ports.SCMObservation et al.) and a new lifecycle entrypoint (ApplySCMObservation) that projects into the existing PR reaction path.
  • Implement the SCM polling observer (ETag guards, batching, review refresh cadence, semantic hashing, persistence + lifecycle notification ordering).
  • Extend SQLite schema/queries/mappings to persist SCM metadata, CI check detail, review threads, bot filtering, and observation timestamps; wire GitHub-backed observer into daemon startup.

Reviewed changes

Copilot reviewed 24 out of 30 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
backend/internal/storage/sqlite/store/store_test.go Adds store tests validating WriteSCMObservation persistence semantics (replace/merge/preserve) and legacy WritePR compatibility.
backend/internal/storage/sqlite/store/pr_store.go Implements SCMWriter via WriteSCMObservation, adds review thread persistence + merge/replace modes, and expands PR/check/comment mappings.
backend/internal/storage/sqlite/queries/pr.sql Expands PR upsert/select to include SCM metadata fields + semantic hashes/timestamps; excludes bot comments from display review count.
backend/internal/storage/sqlite/queries/pr_review_threads.sql New sqlc queries for upserting/listing/deleting normalized PR review threads.
backend/internal/storage/sqlite/queries/pr_comment.sql Extends PR comments schema interactions (thread_id/url/is_bot) and adds delete-by-thread for merge refresh.
backend/internal/storage/sqlite/queries/pr_checks.sql Extends PR checks persistence with conclusion/details fields.
backend/internal/storage/sqlite/migrations/0003_scm_observer_schema.sql Adds SCM observer columns/tables (PR metadata, hashes, timestamps, check details, comment bot/thread info, pr_review_threads).
backend/internal/storage/sqlite/gen/projects.sql.go Tightens ArchiveProject update to be idempotent (only archive once).
backend/internal/storage/sqlite/gen/pr.sql.go Regenerates sqlc PR query bindings for expanded PR schema + bot filtering in display facts.
backend/internal/storage/sqlite/gen/pr_review_threads.sql.go New generated sqlc bindings for review thread queries.
backend/internal/storage/sqlite/gen/pr_comment.sql.go Regenerates sqlc bindings for extended PR comment schema + delete-by-thread.
backend/internal/storage/sqlite/gen/pr_checks.sql.go Regenerates sqlc bindings for extended PR check schema.
backend/internal/storage/sqlite/gen/models.go Extends generated models for new PR/check/comment/review-thread fields.
backend/internal/session_manager/manager.go Minor allocation improvement when collecting cleaned session IDs.
backend/internal/ports/scm_observations.go Adds provider-neutral SCM DTOs used across provider/observer/store/lifecycle boundaries.
backend/internal/ports/outbound.go Adds ReviewWriteMode and SCMWriter port for transactional SCM persistence semantics.
backend/internal/observe/scm/observer.go New SCM observer orchestrator (polling loop, guards, batching, review refresh, semantic hashing, persistence + lifecycle notification).
backend/internal/observe/scm/observer_test.go Adds orchestration tests for observer behavior (ETags, batching, log tails, review cadence, lifecycle/hash acknowledgement).
backend/internal/lifecycle/reactions.go Adds ApplySCMObservation that projects SCM observations into legacy PRObservation reactions; minor slice preallocation.
backend/internal/lifecycle/manager_test.go Adds test asserting SCM observation projection triggers existing CI nudge behavior.
backend/internal/domain/pr.go Extends domain PR/check/comment models and introduces normalized review-thread model + semantic hashes/timestamps.
backend/internal/daemon/scm_wiring.go Wires SCM observer into daemon with GitHub provider + lazy token preflight avoidance.
backend/internal/daemon/lifecycle_wiring.go Ensures daemon stop waits for SCM observer goroutine as well as reaper.
backend/internal/daemon/daemon.go Starts SCM observer during daemon boot via lifecycle stack.
backend/internal/adapters/scm/github/provider.go Adds lazy credential preflight option + SCMCredentialsAvailable; adjusts mergeability blocking for draft/review-required.
backend/internal/adapters/scm/github/provider_test.go Adds tests covering SCM check normalization, mergeability blockers, GraphQL batching fallback, and review thread window behavior.
backend/internal/adapters/scm/github/observer_provider.go New GitHub implementation of the provider-neutral observer Provider contract (guards, detection, batching, log tail, review pagination).
backend/internal/adapters/scm/github/doc.go Updates package docs to reflect SCM observer usage and cache ownership split.
backend/internal/adapters/scm/github/client.go Adds doRESTWithETag for observer-owned ETag handling and helper for ETag header fallback.
backend/internal/adapters/scm/github/auth.go Adds FallbackTokenSource for env/gh token lookup and invalidation forwarding.
Files not reviewed (6)
  • backend/internal/storage/sqlite/gen/models.go: Language not supported
  • backend/internal/storage/sqlite/gen/pr.sql.go: Language not supported
  • backend/internal/storage/sqlite/gen/pr_checks.sql.go: Language not supported
  • backend/internal/storage/sqlite/gen/pr_comment.sql.go: Language not supported
  • backend/internal/storage/sqlite/gen/pr_review_threads.sql.go: Language not supported
  • backend/internal/storage/sqlite/gen/projects.sql.go: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread backend/internal/adapters/scm/github/observer_provider.go
Comment thread backend/internal/observe/scm/observer.go
Comment thread backend/internal/storage/sqlite/migrations/0004_scm_observer_schema.sql Outdated
Comment thread backend/internal/storage/sqlite/store/pr_store.go
Comment thread backend/internal/observe/scm/observer.go
Comment thread backend/internal/observe/scm/observer.go Outdated
Copy link
Copy Markdown
Collaborator

@illegalcall illegalcall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up on latest head e84d4ee: the previous durability and pagination comments are addressed; I found one new CDC side effect in the acknowledgement write.

Comment thread backend/internal/observe/scm/observer.go Outdated
@whoisasx whoisasx requested a review from illegalcall June 4, 2026 07:10
Copy link
Copy Markdown
Collaborator

@harshitsinghbhandari harshitsinghbhandari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High-recall review pass against feat/75 at 6302844. 10 findings, ranked by severity — correctness first, then durability/efficiency, then altitude/cleanup. Most prior @illegalcall + @greptile threads look correctly addressed; these are what's left after deduping against resolved threads.


Correctness

1. backend/internal/lifecycle/reactions.go:116 — failed-check dedup signature degenerates when CI.HeadSHA is empty

scmToPRObservation copies obs.CI.HeadSHA into every PRCheckObservation.CommitHash (reactions.go:116). On a review-only refresh that fetches threads but no fresh CI batch, the observation flows back through ApplyPRObservation with CommitHash="". ApplyPRObservation then builds sig = ch.CommitHash + ":" + ch.LogTail (reactions.go:52). If both are empty/stable the signature collapses to a constant for that (key, PR, check-name); a later genuinely-new failing CI on a different commit silently no-ops in sendOnce because react.seen[key] == sig already. The agent never gets the new failure nudge.

2. backend/internal/storage/sqlite/queries/pr_comment.sql:5InsertLegacyPRComment is INSERT OR IGNORE, silently dropping updates

Pre-PR the legacy WritePR path replaced the full comment set per PR (DeletePRComments → InsertPRComment). The new InsertLegacyPRComment uses INSERT OR IGNORE and writePR no longer deletes legacy comments first (pr_store.go:74-77 gates delete on ReviewWriteReplace). If a reviewer edits the body of an existing comment and the legacy path runs (e.g. through service/pr.Manager.ObservePR, today only wired by integration/lifecycle_sqlite_test.go:97), the updated body is dropped and the stale body persists. The fix was likely targeted at not clobbering observer-written rows, but it changed the semantics for the legacy path too.


Durability / efficiency

3. backend/internal/observe/scm/observer.go:340-374 — two-phase "lifecycle is the cursor" is only durable within one daemon process

Poll writes the row with preserved (old) hashes (line 358), calls lifecycle.ApplySCMObservation (line 364), then writes again with final hashes (line 369). reactions.sendOnce dedups in an in-memory react.seen/attempts map (reactions.go:172-194). If the daemon crashes after ApplySCMObservation returns nil but before the second write commits, the in-memory dedup is lost; on the next start, the same observation re-fetches (hashes unchanged in DB), lifecycle re-fires, and the agent receives a duplicate nudge. The design comment at observer.go:340-345 claims "lifecycle is the cursor" — but the cursor is durable only across polls within the same process. A durable outbox row (or persisting react.seen/attempts in the DB) would close this.

4. backend/internal/adapters/scm/github/observer_provider.go:168FetchReviewThreads always fetches a second 50-thread page when the newest thread is unresolved

After fetching the latest 50 threads, the code issues a second GraphQL page whenever len(latest) == 0 || !latest[0].Resolved (observer_provider.go:168). For any actively-reviewed PR the newest thread is typically unresolved, so the 2-minute review cadence fetches 100 threads + their leading 5 comments every cycle, not 50. The intent (recover older unresolved threads) only fires when the newest happens to be unresolved — which is the wrong signal, since older threads need a revisit regardless of the newest one's state. Cost: ~2× GraphQL points per review poll across all open PRs.

5. backend/internal/observe/scm/observer.go:339, 357domainFromObservation is called twice per changed PR; each call recomputes three SHA256 hashes

When lifecycle is wired, domainFromObservation runs once with the pending-hash options (line 338) and again with the final-hash options (line 357). Each call invokes metadataSemanticHash, ciSemanticHash, and reviewSemanticHash (observer.go:765-783). prepareForPersistence (line 740) also computes the same three hashes. For 50 changed PRs/poll that's ~450 hash invocations where ~150 suffice. Compute the three hash pairs once in prepareForPersistence and thread them through.

6. backend/internal/observe/scm/observer.go:626enrichFailureLogs does O(M×N) match between FailedChecks and all Checks

For each failed check (~M) the inner loop walks all checks (~N) to copy LogTail back into obs.CI.Checks. A PR with 80 checks and 10 failing is 800 string comparisons per CI refresh. Pre-build map[checkKey]int once, then update obs.CI.Checks[idx].LogTail directly. applyStoredFailedLogTails (line 635-665) already uses the cleaner name → tail map shape — same pattern is applicable here.


Altitude / cleanup

7. backend/internal/storage/sqlite/store/pr_store.go:50writePR mixes legacy and SCM-observer modes via a bool + enum; legacy WritePR has no production caller

writePR carries replaceLegacyComments bool plus ReviewWriteMode and branches on the combination (lines 69-101). Searching the tree, service/pr.Manager (the only legacy WritePR consumer) is wired only by integration/lifecycle_sqlite_test.go:97 — no daemon path uses it. Deleting service/pr.Manager, InsertLegacyPRComment, genLegacyCommentParams, and the bool collapses writePR to just WriteSCMObservation's body and incidentally resolves finding #2.

8. backend/internal/observe/scm/observer.go:1128-1166 — three near-identical FIFO eviction helpers

cacheSetString, cacheSetTime, cacheSetBool, and evictStrings duplicate the same "append-to-order-slice; evict-when-over-cap" pattern, parameterized only by value type. A generic cacheSet[V any](m map[string]V, order *[]string, key string, value V, max int) collapses to one helper. Any future fix (proper LRU touch, deterministic eviction, TTL) currently has to land in four places.

9. backend/internal/observe/scm/observer.go:716-718ReviewRefreshFailed reset only flips value to false; reviewFailedOrder grows monotonically

On a successful review refresh, Cache.ReviewRefreshFailed[pkey] = false is set but the key is never removed from reviewFailedOrder (line 1146-1155 evicts by slice length). Over a long-lived daemon, eviction begins discarding genuinely-failed entries earlier than the configured cap implies, because cleared false entries still hold slots. Fix: delete(map, pkey) and remove from the order slice on success.

10. backend/internal/domain/pr.go:51-53 — provider-specific enum strings leak into the provider-neutral domain

domain.PullRequest carries ProviderState, ProviderMergeable, ProviderMergeStateStatus — raw GitHub strings — alongside the normalized AO enums. mergeabilityFromProviderFacts (observer.go:950) inspects GitHub-flavored values like DIRTY/BEHIND/UNSTABLE. A second provider (Linear/GitLab) repeats this contamination. A provider-opaque ProviderPayload []byte (or a JSON column) plus moving the provider→AO normalization fully into the provider package keeps the domain layer purely normalized.


Verified refuted (not findings, for transparency)

  • observer_provider.go:406 scmChecksFromGraphQL doesn't nil-panic — Go nil-map reads return zero values, and the loop over nodes(nil["nodes"]) runs zero iterations.
  • reactions.go:122-138 doesn't drop human review comments via the IsBot skip — scmThreadFromGraphQL (observer_provider.go:555-570) only sets thread.IsBot=true when every comment is bot-authored.
  • 0004_scm_observer_schema.sql change_log drop+recreate is safe — goose migrations run in a transaction; no concurrent write window exists.
  • cdc.EventType widening (pr_review_thread_added/_resolved) is safe — no exhaustive switch over EventType exists in the tree.

@whoisasx
Copy link
Copy Markdown
Collaborator Author

whoisasx commented Jun 4, 2026

@harshitsinghbhandari thanks for the high-recall pass. I pushed a5f96a9 with the correctness fixes that were actionable in this PR, plus regression coverage. Per finding:

  1. CI.HeadSHA empty in SCM → legacy reaction projection — fixed. scmToPRObservation now falls back to PR.HeadSHA when CI.HeadSHA is absent, so CI nudge dedup signatures remain commit-specific on review-only/local-reconstructed observations. Added TestSCMObservationUsesPRHeadWhenCIHeadMissing.

  2. InsertLegacyPRComment / legacy replacement semantics — the existing path did delete unthreaded legacy comments before inserting, so edited legacy comments were not actually stranded by INSERT OR IGNORE. I added explicit regression coverage (TestWritePRReplacesLegacyCommentBodies) to lock that behavior down. I also fixed the related real collision risk: legacy WritePR now uses a narrow UpsertLegacyPR that updates only legacy scalar columns and preserves SCM-owned metadata/hashes/timestamps from WriteSCMObservation.

  3. Crash after lifecycle success but before hash acknowledgement — acknowledged as a remaining at-least-once-delivery edge. The current two-phase write intentionally prevents the worse failure mode (lost lifecycle effects after a failed delivery/restart). Exactly-once behavior across a crash after successful lifecycle delivery would require a durable reaction/outbox acknowledgement model, which is larger than this PR and should be tracked separately.

  4. Review-thread second page trigger — this was a naming/readability issue, not the “newest unresolved” behavior. GitHub returns last:N connection nodes in connection order, so latest[0] is the oldest thread in the latest window. I added an explicit comment and oldestLatestUnresolved variable to make the boundary condition clear. Existing provider tests cover both “oldest resolved → no fallback” and “oldest unresolved → one older page”.

  5. Repeated semantic hash computation — valid efficiency note, but not correctness-blocking. I left this unchanged to avoid widening the patch; the current cost is bounded and the next clean refactor is to carry prepared hash values through domainFromObservation.

  6. enrichFailureLogs O(M×N) matching — fixed. It now pre-indexes checks by (name, providerID) and updates matching check rows directly while preserving duplicate-key behavior.

  7. Legacy/SCM mixed write path cleanup — partially addressed. I kept the legacy path because tests and the legacy service still compile against it, but separated the legacy PR upsert so it no longer touches SCM-owned observer fields. Full legacy deletion can be a separate cleanup PR.

  8. Duplicate FIFO cache helpers — valid cleanup suggestion, but no correctness issue. I did not fold all helpers in this patch; I added only the needed generic cacheDelete helper for the retry-cache fix below.

  9. ReviewRefreshFailed false entries holding FIFO slots — fixed. A successful review refresh now deletes the retry-cache entry and removes it from the order slice. Added TestPoll_SuccessfulReviewRefreshClearsRetryCacheSlot.

  10. Provider-specific raw fields in domain — valid architecture note, but this PR intentionally persists raw provider diagnostics alongside normalized fields so the observer can preserve/debug GitHub mergeability facts. Moving that to an opaque provider payload is a broader schema/API cleanup and should be separate from this migration PR.

Local validation run after the patch:

  • go test ./...
  • go vet ./...
  • go test -race ./...
  • go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2 cache clean && go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2 run --path-mode=abs
  • go test -tags e2e -v ./internal/cli/...

@harshitsinghbhandari harshitsinghbhandari added this to the rewrite milestone Jun 4, 2026
@harshitsinghbhandari
Copy link
Copy Markdown
Collaborator

Let's merge this PR @whoisasx

@whoisasx whoisasx merged commit 19b6ca5 into main Jun 4, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Plan: provider-neutral SCM observer

5 participants