feat(search): check the index schema on startup and refuse breaking changes - #3197
Draft
dschmidt wants to merge 17 commits into
Draft
feat(search): check the index schema on startup and refuse breaking changes#3197dschmidt wants to merge 17 commits into
dschmidt wants to merge 17 commits into
Conversation
…hanges Both engines now diff the stored/live index schema against the schema generated from code when the service starts. A shared recursive classifier in the mapping package is the single oracle: - equal: start normally. - additive (new fields without any indexed data): applied in place. OpenSearch gets a PUT _mapping with the full code properties, bleve persists the code mapping into the index (SetInternal + reopen) so the new fields are properly typed immediately and later startups classify equal. A startup warning lists the new fields because documents indexed before the upgrade lack them until re-indexed. - breaking (changed definitions or analyzers, removed or renamed fields, or new fields that already contain data of unknown form): refuse to start with an error describing the rebuild procedure (delete the index, start, run "opencloud search index --all-spaces") and the OC_EXCLUDE_RUN_SERVICES=search escape hatch. PUT _mapping is deliberately only the apply mechanism, never the judge: its merge semantics cannot see removals or renames and it accepts in-place updatable param changes with an ack. bleve additionally checks idx.Fields() so previously dynamically indexed data (which leaves no schema trace in bleve) is caught, matching by exact name and by path prefix. While at it: the OpenSearch startup check runs with a real, minute-bounded context instead of context.TODO(), bleve indexes are opened with a 5s bolt_timeout so a second process fails fast instead of hanging on the file lock, and the reversed errors.Is arguments in bleve.NewIndex were fixed. #3092
… delete step Addresses the two Copilot review comments on the PR: the additive opensearch log now matches the bleve warning (level and re-index hint), and the refuse message spells out how to delete the index per engine (DELETE /<name> vs removing the bleve directory).
- the additive warnings advertise --all-spaces --force-rescan; a plain walk skips unchanged documents and never backfills the new fields - Apply checks index existence first again, so a pre-provisioned index needs no create privilege and odd create-error shapes (string error bodies, cluster blocks) cannot fail a healthy startup; Create on 404 keeps the typed already-exists swallow as the creation-race backstop - number_of_replicas drift is not breaking, it is runtime-tunable and needs no rebuild - bleve returns the classification alongside post-persist errors and the server warns before the error check, so the one-time additive warning is not lost when close or reopen fails - a golden fixture pins the marshaled bleve mapping so a dependency bump that changes marshaling fails in CI instead of refusing every installation in the field
|
Label error. Requires at least 1 of: Type:Bug, Type:Enhancement, Type:Feature, Type:Breaking-Change, Type:Test, Type:Documentation, Type:Maintenance, Type:Security, Type:Dependencies, Type:DevOps, dependencies. Found: |
Not up to standards ⛔
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2659. The base
tmp/refactor-search-mappingis a temporary upstream snapshot of the #2659 branch so this PR shows only its own commits; it gets retargeted tomain(and the temp branch deleted) once #2659 is merged. Both PRs need to ship in the same release. Refs #3092.Two complementary mechanisms keep the index schema and the code in sync.
Versioned indices (from the base #2659). The index name/path carries the schema version from
search.SchemaVersion(OpenSearchopencloud-resource-v3, blevebleve-v3). An intended breaking schema change is made by bumping the constant: the service targets a fresh, empty index and leaves the old one in place. No migration, no index-to-index reindex; content lives only in the index, the files are the source of truth, so a rescan (opencloud search index --all-spaces) rebuilds everything into the new index.Startup schema check (this PR). The version bump is manual, so this is the safety net for the schema changing without one. On startup both engines diff the existing (current-version) index schema against the schema generated from code, using one shared recursive classifier:
PUT _mappingwith the full code properties. bleve: the code mapping is persisted into the index (SetInternal+ reopen), so new fields are typed correctly right away and the next start classifies equal. A startup warning lists the new fields; documents indexed before the upgrade lack them until re-indexed.SchemaVersionand gets a fresh index. It fires in development when the mapping is changed in a breaking way without bumping, so the error is developer-facing (bumpsearch.SchemaVersion, or revert the change) and lists the diff that triggered it.Why this shape:
PUT _mappingis just the apply mechanism: its merge semantics hide removals/renames and it acks in-place updatable param changes, so it must not judge.idx.Fields()(exact name + path prefix). A field that is explicitly mapped now but already holds dynamically indexed data of unknown shape is breaking rather than a source of silently wrong results, no search beats wrong search.Known and accepted, no code changes:
bolt_timeout) instead of hanging forever.Deliberate consequences of the classifier rules (not bugs):
PUT _mapping/ persisted bleve mapping), the previous release sees a stored-only field and refuses to start. Roll forward or rebuild; refuse-in-both-directions is what keeps mismatched schemas from ever serving queries.