algebraic: project dynamic-template fields into typed docfacts at ingest - #144
Closed
ajroetker wants to merge 7 commits into
Closed
algebraic: project dynamic-template fields into typed docfacts at ingest#144ajroetker wants to merge 7 commits into
ajroetker wants to merge 7 commits into
Conversation
Wire table-level dynamic templates into the algebraic sidecar so that fields matching a template are promoted into typed group/measure/time docfacts at write time, without requiring a schema version bump or a reindex (Elasticsearch-style runtime-adaptive mapping). - schema_capability compiles bounded dynamic templates (keyword/numeric/ boolean/datetime) into capability `dynamic_field_rules`; unbounded text templates and selector-less/negative-only templates stay on the schemaless path-fact path (cardinality guard). The capability fingerprint now reflects template changes so drift is detected. - index.Config carries dynamic_field_rules; writeDocFacts and the coalesced-update path both project static + dynamic facts through a single helper so the stored docfact row stays consistent and deletes clean up symmetrically. Explicit schema fields take precedence over templates; matching reuses schema.zig glob/match_mapping_type semantics. Tests cover template compilation, fingerprint drift on a template-only type change, and runtime docfact projection + overwrite cleanup.
Make dynamic-template updates reach the algebraic sidecar without a table recreate or schema version bump, on both the durable and live paths. - api/tables.zig: applySchemaUpdateRecord now regenerates every algebraic index's schema-derived config (dynamic_field_rules + capability fingerprint) from the updated schema on every schema update, including template-only changes that preserve the version. Public algebraic indexes are always schema-derived, so each is regenerated; only user-tunable runtime knobs are carried forward. - algebraic/index.zig: add Index.reloadConfigJson to swap the parsed config in place (lazy backfill — only new/rewritten docs reflect new rules). - catalog/index_manager.zig + db.zig: reloadAlgebraicSchemaConfigs regenerates and applies configs to live indexes, skipping the swap when unchanged so periodic reconciles don't churn. - api/table_writes.zig: applyLocalTableSchemaJson refreshes live algebraic configs after setSchema, so template changes apply to a running DB immediately (semantically equivalent to a reopen). Tests: durable refresh (metadata.schema update refreshes algebraic dynamic templates without recreate) and live in-place reload (algebraic reloadConfigJson applies new dynamic template rules to live writes). Docs updated in SCHEMA.md and ALGEBRAIC.md.
…test
Query-side wiring so aggregations over template-promoted fields route to
the algebraic sidecar:
- index.zig: Index.fieldConfig now falls back to dynamic_field_rules when a
queried field isn't statically declared, synthesizing a FieldConfig keyed
by the full query path (matching the ingest-time fact identity). Only rules
with a name/path selector resolve a concrete query field; match_mapping_type
-only rules can't be evaluated without a value. resolveField/resolveUniqueField
inherit this, so the planner's group/measure/time gates accept dynamic fields
and route group-by/sum/term queries to the docfact fold scan.
Also fix a pre-existing, consistently-failing assertion in api/indexes.zig
("index encoders expose compact algebraic public status"): index_type is
emitted once in the aggregate `status` and once per group in `shard_status`
(consistent with every other index kind), so the single-group fixture yields
two occurrences, not one. The normal CI filter lists never exercised this
test, so the stale assertion went unnoticed until the full algebraic filter
ran it.
Tests: query-time resolution + end-to-end terms aggregation grouped by a
dynamic field; the full algebraic suite is now green.
Add a lib-api-logic-test step (and wire it into the unit-test bucket) that runs the api/tables.zig and api/indexes.zig encoder/parser/schema-update logic tests, including the dynamic-template durable-refresh test and the algebraic status-encoder test. Previously these only ran under broad ad-hoc filters, so the schema-update and status assertions weren't gated in CI. - root.zig: reference public_api.tables and public_api.indexes in the test block so their named tests are reliably collected (mirrors the existing public_api.http_server reference that gates the auth tests). - build.zig: new step with declared-name filters covering both files, using the shared simple test runner. Five pre-existing embeddings/shard status-encoder assertions in api/indexes.zig have bit-rotted against the current encoder (they were never collected by any gated step) and are unrelated to dynamic templates. They are skipped via --skip-test-filter with a comment pending separate triage, rather than rewritten blind, so the new step is green without masking them.
Repair the five pre-existing api/indexes.zig status-encoder tests that had bit-rotted against the current encoder (they were never collected by a gated step). Their final behavior was set in c8c6afd; the assertions were stale leftovers. Each is updated to assert the actual, deliberate encoder behavior (or, for one, a fixture inconsistency is corrected), and the lib-api-logic-test skip list is removed so all 56 api logic tests run: - "expose metadata-backed configs": a no-runtime index (null local status, no ranges) reports empty status/shard_status objects, not a synthesized status. - "expose local shard runtime status": backfill_progress is recomputed from the replay sequences (3/5 = 0.600) while catching up, not the raw fixture value. - "aggregate preserved synthetic shard counters": a stale-only shard aggregates as rebuilding (index_type first, rebuilding:true) while the synthetic counters and the ready per-shard status are still preserved. - "keeps published visibility separate from replay debt": once published visibility is complete the encoder reports replay caught up (applied bumped to target, catch-up cleared) rather than surfacing raw applied=0 debt. - "prefers replay completion once docs are indexed": fixture corrected to index both table docs (doc_count 1 -> 2, consistent with replay applied==target==2); the prior 1-of-2 coverage gap is a real backfilling state, contradicting the test's "once docs are indexed" ready assertions.
…mmetry Address three review findings on the dynamic-template -> algebraic feature: 1. Partial-aggregate trap (HIGH): when a dynamic template is added/retyped on a table that already holds data, existing docs are not re-projected, so routing group/sum/terms over the newly-promoted field to the sidecar returned aggregates over only post-change documents. Add a per-index `dynamic_rules_backfill_pending` flag: while set, query-time resolution of dynamic-template fields is withheld (those queries fall back to the complete scan and return correct results); static fields keep accelerating. The flag is set on a real capability change (differing fingerprint) both in the live reload and durably in the regenerated config, so it survives reopen, and is cleared on rebuild. Table create takes the fingerprint-equality fast path and is never flagged. 2. Ingest/query asymmetry (MED): a `match_mapping_type`-only template compiled into a rule that projected facts at ingest but could never resolve at query time (no value to classify). Require a name/path selector (`match` / `path_match`) to promote a template, in both schema_capability compilation and index.zig validateConfig. 3. Live reload divergence (MED): reloadAlgebraicSchemaConfigs compared bare schema-derived JSON against the stored full config (so the skip never fired) and dropped user-tunable knobs that the durable path preserves. Now it skips when the capability fingerprint is unchanged and carries forward the same tunable knobs (adaptive policy, pathfact policy, planner/result limits) the durable regeneration preserves, keeping live and durable configs in agreement. Tests: backfill-pending withholds dynamic resolution; validateConfig rejects selector-less / mapping-type-only rules; durable regeneration sets the flag on a real change and preserves knobs without flagging on a no-op. Docs updated in SCHEMA.md.
Address the remaining review findings on dynamic-template handling: - Overlapping-rules asymmetry (#4): ingest now stops at the first selector-matching rule (Elasticsearch order) instead of falling through to a later overlapping rule when a value fails to coerce — a non-coercible value simply yields no fact, like a static typed field. Query-time dynamicFieldConfig resolves a field only when all name/path-matching rules agree on the scalar type; on disagreement it declines (the aggregation falls back to a complete scan) rather than reading a type ingest may have stored differently. Together these guarantee query never reads a type ingest didn't store. New test covers the overlapping-disagreement case end to end. - Duplicated matcher: dynamicRuleMatches and dynamicRuleResolvesField are unified behind a single dynamicRuleSelectorMatches(rule, path, name, ?value) evaluator, so ingest and query selector semantics cannot drift. match_mapping_type is unsatisfiable when value is null (query time), and query additionally requires a name/path selector. - query_field aliasing: documented that dynamicFieldConfig's returned name/path alias the caller's query_field and that this is safe because every consumer is query-scoped (planner copies the name into owned canonical metadata tuples; resolveField already borrows query_field into .public). No code change needed. - reloadConfigJson concurrency: documented the invariant that the in-place config swap/free is only called under the provisioned write source's exclusive structural-mutation lock on a freshly-opened unshared DB handle (mirroring the adjacent core.setSchema swap), so it is not a use-after-free; added a guard note for any future shared-reader use. Investigation also surfaced a separate, pre-existing UAF unrelated to this work: the serverless and C-API aggregation paths free aggregation requests without the cloneSearchAggregationResultLabelsDeep that table_reads.zig performs, leaving result name/field/type borrows dangling. Not touched here; flagged for a dedicated fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wire table-level dynamic templates into the algebraic sidecar so that
fields matching a template are promoted into typed group/measure/time
docfacts at write time, without requiring a schema version bump or a
reindex (Elasticsearch-style runtime-adaptive mapping).
boolean/datetime) into capability
dynamic_field_rules; unbounded texttemplates and selector-less/negative-only templates stay on the
schemaless path-fact path (cardinality guard). The capability
fingerprint now reflects template changes so drift is detected.
coalesced-update path both project static + dynamic facts through a
single helper so the stored docfact row stays consistent and deletes
clean up symmetrically. Explicit schema fields take precedence over
templates; matching reuses schema.zig glob/match_mapping_type semantics.
Tests cover template compilation, fingerprint drift on a template-only
type change, and runtime docfact projection + overwrite cleanup.