feat(indexer): bridge on-chain plays into the plays table#881
Merged
Conversation
The vendored ETL play processor only writes etl_plays, which nothing in api/ reads. Restore the legacy Python index_core_plays behavior with a PlaysHook (go-openaudio #322) that writes each on-chain play into the `plays` table — the row every downstream consumer depends on (the on_play trigger's aggregates/milestones/notifications, the challenge processors, trending, hourly play counts). Field mapping mirrors index_core_plays exactly: play_item_id = int(track_id) (skip non-integer), user_id = int(user_id) or NULL for anonymous listens, source = "relay", created_at = play timestamp, slot = Core block height. Challenge-event dispatch is intentionally omitted — the new challenge processors reconcile from `plays` by polling. Runs in the same DB tx as etl_plays, so the rows commit atomically. Bumps the go-openaudio pin to the commit that adds RegisterPlaysHook. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
raymondjacobson
force-pushed
the
api/index-core-plays-hook
branch
from
705ec6a to
5818401
Compare
raymondjacobson
added a commit
that referenced
this pull request
May 30, 2026
…)" (#885) ## Summary Reverts #883, pinning go-openaudio back to `v1.3.1-0.20260529221831-4d1c9dfdfb52`. The halt-on-error behavior from upstream go-openaudio#323 is correct in isolation, but is **incompatible with the current dual-run state**: Python and api-side ETL both write to overlapping tables, and the on-chain plays bridge from #881 doesn't ON CONFLICT-protect against rows Python has already written. So: - Pre-#883: the failure was silently swallowed by `continue` — ETL was effectively a no-op on essentially every block since #881 deployed, but block_diff stayed green because Python's writes kept `MAX(blocks.height)` moving. Block-level data loss masked by Python carrying the load. - Post-#883: the same failure crashes the indexing loop. We saw it tonight at `processBlock failed` on block 25415514, reproducibly across pod restarts because Python writes the same plays in the same block before the ETL gets to it. Once #884 (the api-wrapper fix that makes that halt actually exit the process) ships, every pod would crashloop the moment it tries to index any recent block. So shipping #883 + #884 without first handling the cross-writer collision points would convert today's silent wedge into a continuous outage that takes the parity jobs (`IndexChallengesJob`, `UserListeningHistory`, `HourlyPlayCounts`, etc.) down with the ETL. Strictly worse. ## Plan 1. **This PR**: pin upstream back to the pre-halt version. Today's silent wedge stays in place — bad, but bounded — and the parity jobs keep ticking. 2. Close #884 (already done). The diagnosis there is correct, but it amplifies #883's bad sequencing, so we re-land it after #883 is safe to re-ship. 3. Revert OpenAudio/go-openaudio#323 upstream too, so no future bump trips this accidentally. 4. **Audit + fix the cross-writer collision points in pkg/etl** — start with the plays bridge (#881), apply the same ON CONFLICT pattern #319 used for the `blocks` table. Then sweep anywhere else ETL and Python touch the same row. 5. Re-land go-openaudio#323, then api#883, then api#884 (in that order). At that point the halt-on-error guarantee is honest. ## Bump details (revert direction) | | from | to | |---|---|---| | `github.com/OpenAudio/go-openaudio` | `v1.3.1-0.20260529230137-819100b28c94` | `v1.3.1-0.20260529221831-4d1c9dfdfb52` | | `github.com/OpenAudio/go-openaudio/pkg/etl` | `v1.3.1-0.20260529230137-819100b28c94` | `v1.3.1-0.20260529221831-4d1c9dfdfb52` | ## Test plan - [x] `go build ./...` clean. - [ ] After deploy: confirm new pod boots, no `processBlock failed` halt log on block 25415514 (it'll go back to silent `continue`). - [ ] Verify parity jobs still tick and block_diff stays at 0 (no functional change vs. pre-#883 prod). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The vendored ETL play processor only writes its own
etl_playstable — nothing in api/ readsetl_plays. Theplaystable is the one every downstream consumer depends on, so under the new ETL indexer plays were effectively a no-op. This restores the legacy Pythonindex_core_playsbehavior by bridging each on-chain play intoplays.Why
playsmatters:on_playtrigger fans each row out toaggregate_plays,aggregate_monthly_plays,milestones,notification, anduser_distinct_play_*.listen_streak,play_count_milestones) pollplaysdirectly.plays.How
Uses the
RegisterPlaysHookextension point added upstream in go-openaudio#322. The hook runs in the same DB transaction the ETL play processor used foretl_plays, soplaysrows commit atomically withetl_playsand the rest of the block.Field mapping mirrors
index_core_playsexactly:play_item_id=int(track_id); the play is skipped iftrack_idisn't an integer.user_id=int(user_id), or NULL for an anonymous listen (non-integer user_id).source="relay",created_at= play timestamp,updated_at= now().slot= Core block height (monotonic; shared by all plays in a block, the same shape as Python's per-txnext_slot). No api/ Go consumer readsplays.slot; theon_playtrigger only forwards it onto milestone/notification rows.Intentionally not ported: Python's challenge-event dispatch (
track_listen/track_played). The new challenge processors reconcile fromplaysby polling, so the bridge only needs to land the rows.Hook errors are logged but non-fatal (the
etl.PlaysHookcontract) — a malformed play must not roll backetl_playsor halt the indexer.Dependency
4d1c9df). This PR is rebased on latestmainand pins that merged commit.Test plan
go build ./...andgo vet ./indexer/...cleanindexer/plays_hook_test.go(real test DB): inserts a play with correct fields + slot=block height; anonymous listen → NULL user_id; non-integer track_id skipped (valid sibling still lands); empty tx is a no-op; theon_playtrigger incrementsaggregate_playsfor written rows./indexer/package suite passes against the merged go-openaudio commit🤖 Generated with Claude Code