feat(timeseries): propagate ILP-inferred schema to catalog for SQL column resolution#3
Merged
farhan-syah merged 1 commit intoNodeDB-Lab:mainfrom Mar 30, 2026
Merged
Conversation
…lumn resolution On first ILP ingest the Data Plane infers a columnar schema from the batch (tags → Symbol, fields → Float64/Int64, timestamp). This schema was only stored in the in-memory memtable, invisible to the SQL planner which still saw the default (timestamp, value) fields from CREATE TIMESERIES. Propagate the inferred schema back to the Control Plane via the ingest response payload. The ILP listener updates StoredCollection.fields in the catalog so DataFusion resolves the actual column names and types. Additionally, adjust the Arrow schema builder for timeseries collections: omit the synthetic 'id' column (timeseries rows have no document ID) and map TIMESTAMP to Int64 (internal millis representation).
7 tasks
This was referenced Apr 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Propagate ILP-inferred schema to the catalog: On first ILP ingest, the Data Plane infers a columnar schema (tags → Symbol/VARCHAR, fields → Float64/Int64, timestamp). Previously this schema only existed in the in-memory memtable, invisible to the SQL planner which still saw the default
(timestamp, value)fields fromCREATE TIMESERIES. Now the inferred schema is returned in the ingest response and the ILP listener updatesStoredCollection.fieldsin the catalog.Fix Arrow schema for timeseries collections: Omit the synthetic
idcolumn (timeseries rows don't have document IDs) and mapTIMESTAMPtoDataType::Int64(internal millis representation) instead ofUtf8.Unblocks queries like:
Test plan
DESCRIBE collectionshows all ILP-inferred columns after first ingestGROUP BYon tag columns (qtype, client, status) returns correct countsAVG/MIN/MAXon numeric fields (elapsed_ms) returns correct valuesWHERE + GROUP BYworks for aggregate queriesidprefix)cargo clippyclean,cargo fmt --allapplied