feat(cluster): wire IngestHandler on readers for query freshness#385
Merged
feat(cluster): wire IngestHandler on readers for query freshness#385
Conversation
Reader nodes now apply replicated WAL entries to their local ArrowBuffer, enabling near-real-time query freshness across the cluster. Changes: - Added ingestBuffer field + SetIngestBuffer() to Coordinator - buildReplicationIngestHandler() parses WAL envelope, decodes both columnar and row msgpack formats, writes to ArrowBuffer via WriteColumnarDirectNoWAL (avoids WAL double-write) - Added IngestHandlerFunc adapter (follows http.HandlerFunc pattern) - Extracted ParseEnvelope() to wal package as shared utility, replacing inline parsing in both reader.go and coordinator.go - main.go calls SetIngestBuffer(arrowBuffer) alongside SetWAL()
Contributor
There was a problem hiding this comment.
Code Review
This pull request implements reader query freshness by enabling reader nodes to apply replicated WAL entries directly to their local in-memory ArrowBuffer. It introduces an IngestHandler within the cluster coordinator to decode both columnar and row-formatted WAL payloads and write them to the buffer. The review feedback highlights several improvement opportunities: ensuring the row-format decoder correctly identifies measurement metadata keys, protecting access to the ingest buffer with read locks to prevent data races, and optimizing the rowsToColumns conversion to filter out metadata and improve efficiency.
- Read-lock c.mu when accessing ingestBuffer in handler closure to prevent data race with SetIngestBuffer - Added _measurement key check for row format records (writer's columnarToWALRecords uses _measurement prefix) - Optimized rowsToColumns to single-pass and filter all metadata keys (_measurement, measurement, m, _database, database)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
buildReplicationIngestHandler()parses WAL envelope, decodes both columnar and row msgpack formats, writes viaWriteColumnarDirectNoWAL(avoids WAL double-write since receiver's LocalWAL already handles persistence)IngestHandlerFuncadapter type (followshttp.HandlerFuncpattern)wal.ParseEnvelope()as shared utility — replaces inline envelope parsing in bothwal/reader.goandcoordinator.goData flow (after this change)
Test plan
go build ./cmd/... ./internal/...passesgo test ./internal/cluster/...passesgo test ./internal/wal/...passes