refactor: reflection-based search mapping + location geopoint - #3345
refactor: reflection-based search mapping + location geopoint#3345dschmidt wants to merge 31 commits into
Conversation
Discussion carried over from #2659Reposting the non-bot discussion from the original PR so it isn't lost with the move to this upstream branch. Quoted verbatim with attribution; the resolved bot review nits (codacy / copilot: embedded-pointer unwrap, Why reflection? (design rationale)
@dschmidt answered (click to expand)
@dschmidt on the diff size:
Related: #2715 Review feedback
Follow-up documentation issue opened by @fschade: #3092 @aduffeck (workflow):
Open item: breaking index change / re-indexing (unresolved)
@aduffeck (on
Resolved: ginkgo test conversion@aduffeck (on
@dschmidt: Converted geo_verify + mtime and the new mapping package tests to ginkgo/gomega. (This carried into the rebase: the opensearch backend suite is now ginkgo too.) Open item: Mtime typed as a date (unresolved)@dschmidt (on
An earlier symptom of the same typing (@aduffeck spotted the fixture choking the unit tests with a |
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 363 |
| Duplication | 127 |
🟢 Coverage 83.85% diff coverage · +0.34% coverage variation
Metric Results Coverage variation ✅ +0.34% coverage variation (-1.00%) Diff coverage ✅ 83.85% diff coverage Coverage variation details
Coverable lines Covered lines Coverage Common ancestor commit (8a9889c) 84276 19595 23.25% Head commit (e5c12a9) 84675 (+399) 19977 (+382) 23.59% (+0.34%) Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch:
<coverage of head commit> - <coverage of common ancestor commit>Diff coverage details
Coverable lines Covered lines Diff coverage Pull request (#3345) 991 831 83.85% Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified:
<covered lines added or modified>/<coverable lines added or modified> * 100%
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
Build the bleve and OpenSearch index mappings from the Go struct via reflection (json tags + per-field overrides) instead of hand-rolled mappings and hit deserializers. New mapping package: BleveBuildMapping, OpenSearchBuildMapping, Deserialize[T], PrepareForIndex; field decoding is fail-soft. Mtime is typed as a date so mtime ranges are chronological on both backends. Route CS3 facet parsing through mapping.DeserializeStringMap. The any-valued (bleve hit) and string-valued (CS3 metadata) deserializers share one generic fillStruct walker with a per-value setLeaf callback.
Add a TypeGeopoint field type. The libregraph Location facet is kept as an
object (retrieval / numeric queries) and a sibling <name>_geopoint field
carries the {lat,lon} form for geo-distance / bbox / polygon queries,
uniform across bleve and OpenSearch via the shared mapping. PrepareForIndex
splices the sibling in at write time.
Mtime is now a date field; the fixture's Go-format string fails OpenSearch date parsing.
The package's engine suite is ginkgo; these new tests were plain.
New package, so use the repo's standard test framework.
The Mtime field is mapped as an OpenSearch `date`, which rejects an empty value with `mapper_parsing_exception: cannot parse empty date`. The folder and root fixtures had no Mtime, so serializing them to `"Mtime": ""` made TestEngine_Purge/purge_resource_trees fail when the document was indexed. Give both a valid RFC3339 Mtime, matching the file fixture.
Both backends carry a shared search.SchemaVersion in the index name (OpenSearch <base>-vN) and data path (bleve-vN). A breaking schema change bumps the version so the service builds a fresh index instead of colliding with the incompatible previous one; the old index is left in place.
…penSearch OpenSearch lowercased every KQL query value, so exact-match queries on case-preserved keyword fields (facet values, ids) never matched their stored token. Fold the value only for fields with a lowercasing analyzer, mirroring the bleve backend. The field set is derived once in search.LowercaseValueFields and shared by both backends (bleve's local buildLowercaseFields is dropped).
The KQL parser produced its own validation errors but imported them from the search service's query package. Move them into pkg/kql and let the search backend consume kql.IsValidationError, so the parser stops depending on a service package.
…ource struct mapping.FieldNameIndex walks the struct and maps a lowercased field path to the real field name, including nested facet sub-fields. Backend-neutral.
query.Normalize resolves field names (query.ResolveField, from the derived index + a small alias overlay) and expands media-type restrictions (mimetype.Expand) once, between parse and backend compilation.
The bleve Creator runs query.Normalize before compiling; the compiler consumes a canonical AST with no field resolution or media-type special-casing.
KQLToOpenSearchBoolQuery runs query.Normalize, then only value lowercasing stays backend-specific; remapKey and unfoldValue are gone.
Keyword and path fields always index their case-preserved base and, when CaseInsensitive is set, an additional <field>_lowercase sibling used only for matching. The KQL lowering marks a restriction case-insensitive; each backend searches the sibling and lowercases the query value the same way the sibling is precomputed at index time (Go strings.ToLower on both sides, so non-ASCII stays consistent). Search always returns the case-preserved base, so the sibling never has to be read back. In bleve it is indexed but not stored, kept out of _all, and without doc values. In OpenSearch it deliberately stays in _source: excluding it would make every update-by-query script rebuild all siblings from the document via painless toLowerCase, which lowercases differently than Go and would drift from the query side. Keeping it in _source avoids that, and a lowercased copy of a name or path is negligible disk in a cluster. The OpenSearch move script keeps the base and its sibling in sync by swapping the moved prefix in Path_lowercase and setting Name_lowercase from Go-lowercased params, so case-insensitive search still finds a file after it moves (previously the sibling went stale). bleve re-indexes the whole document on move/delete/restore, so its siblings stay fresh for free. This also repairs OpenSearch path search (the query value was no longer folded to lowercase, so path:<Foo> returned nothing) and makes bleve path queries match a folder and its descendants like OpenSearch's path_hierarchy. The Path base stays case-preserved so the move/delete descendant update (an exact TermQuery on Path) matches mixed-case folders.
…bleve Single-term `content:` built an unanalyzed term query, so once this branch dropped the blanket query-value lowercasing, `content:Foo` missed on OpenSearch (bleve was unaffected, its query analyzes). Fielded full-text queries now use a match query. OpenSearch `Content` also gets a porter stemming analyzer (it used the default standard analyzer and never stemmed), so full-text search matches bleve on both case and stemming.
bleve compiled a path restriction to a DisjunctionQuery, which mapBinary redistributes as an OR-chain, so `path:/Foo AND name:bar` matched the folder itself unconditionally. It is now a BooleanQuery (should: folder OR descendants), which mapBinary keeps atomic under an enclosing AND. The OpenSearch full-text branch ran before the wildcard check, so `content:foo*` degraded to a phrase match and diverged from bleve; the wildcard check now comes first. Adds the missing coverage the review flagged: path AND term, content wildcard, case-insensitive tags (the array sibling branch), and a spaced path with descendants on OpenSearch.
…rays The []any branch skipped the sibling for an empty array while the []string branch wrote an empty one; both now write it, matching the base field.
CaseInsensitive routes queries to a <field>_lowercase sibling that is only generated for keyword/path fields, so marking any other type CaseInsensitive would silently match nothing. Validate now rejects it up front.
…ends Adds bleve and OpenSearch coverage for category (image), literal MIME (image/svg+xml, with + and /), and raw MimeType: queries. Documents why MimeType skips the bleve escaper: it is not a bug, bleve treats / and + as literals mid-term, so a literal MIME still matches exactly while the category wildcard image/* keeps its *.
mediatype:Folder / mediatype:IMAGE resolved to a literal MimeType search and matched nothing because Expand switched on the raw value. The value is now lowercased in the lowering pass, so categories and literal MIME types match regardless of case, consistently on both backends.
…them resolveField marked every anonymous field embedded, so walkFields (mapping, field index, validate) and fillStruct (deserializer) flattened a json-tagged embedded struct, while conversions.To/encoding/json on the write path nests it under the tag, mapping and deserializing it at the wrong path. An anonymous field is now embedded only without a json tag name, matching encoding/json; fillStruct also recurses into a value nested struct. No current type has a tagged embedded struct, so runtime behavior is unchanged; this hardens the reflection walker.
mediatype:file expands to a NOT restriction. Spliced inline as `NOT MimeType:httpd/unix-directory`, the bleve compiler's NOT branch left a stale operand, so `mediatype:file AND name:x` dropped `name:x` and matched nothing (the web Files filter). It is now wrapped in a group so the negation stays atomic; verified fixing both bleve and OpenSearch.
The guard only rejected CaseInsensitive when a non-keyword/path Type was set explicitly. With no Type, isCasedType treated the field as cased, so CaseInsensitive on an inferred numeric/bool/datetime field passed validation but produced no _lowercase sibling, and the query would silently match nothing. Validate now falls back to the inferred Go type.
…th start The move script rewrote Path/Path_lowercase with painless String.replace, which replaces every occurrence of the old path, not just the leading prefix. OpenCloud paths are ./-prefixed so the full old path only occurs at the start and the result is byte-identical, but startsWith + substring makes the prefix-only intent explicit and robust to any path format. Not a live bug fix, a hardening.
A path value with spaces went through a match_phrase query, which analyzes the query with the path_hierarchy analyzer; the resulting "." prefix token matches every document in the space, breaking descendant matching and the stale-path check after a move.
Cold boots take well over the 5s startup timeout, and a full host disk tripped the flood-stage create-index block mid-run; test indexes are tiny.
…ackends A leading NOT next to an operator was miscompiled: the bleve compiler left the consumed term in `next`, so `NOT x AND y` dropped `y` and produced a self-contradicting clause; the OpenSearch transpiler checked nextOp==AND before prevOp==NOT, so the negated term landed in `must` instead of `must_not`. NOT is unary and binds to the node directly after it regardless of what follows. This also fixes `mediatype:file AND <term>` (the web Files filter) at the root, so the earlier mediatype:file group workaround is dropped.
…ry level Paths act as references (location scoping, deep links): /Foo and /foo are distinct siblings, so path: matching must be exact. Case-insensitive folder discovery is served by name: and its lowercase sibling. This also matches bleve on main, where path queries have always been case-sensitive. Dropping the sibling removes its biggest maintenance cost: Path is the one mutable sibling field, a move rewrites the paths of a whole subtree and the OpenSearch move script had to rebuild Path_lowercase alongside the base field. With paths case-sensitive, the ref path scope moves into the query itself: bleve as a term/prefix disjunction on the keyword Path, OpenSearch as a term filter on its path_hierarchy tokens. The post-filter that used to drop out-of-scope hits after the query ran is gone; totals and paging now respect the scope instead of being computed over the whole space, and a wrong-cased scope simply matches nothing.
a7ed42f to
e5c12a9
Compare
1 participant
Summary
Merges the bleve and OpenSearch index mappings into one reflection-based package (
services/search/pkg/mapping) driven by thesearch.Resourcestruct + a small overrides map. Pullsservices/graph's duplicated reflection walker onto the same helpers and, as a showcase of what the refactor enables, turns onlocation_geopointindexing for spatial queries on both backends.Adding a new facet (motionPhoto, ...) is now roughly: one struct field on
content.Document, one line inservice.go, one line in graph, one line per backend hit converter, plus whatever tika extraction logic the facet actually needs. Everything else falls out of reflection.Behavior changes (deliberate)
Existing indexes keep their stored mapping; the new shape only applies to newly-created indexes.
OpenSearch
Tags/Favorites: dynamickeyword→ explicitkeyword(unified with bleve), searched case-insensitively via the_lowercasesiblings described below. No analyzer.OpenSearch facet sub-strings (
audio.*,photo.*,image.*): dynamictext + keywordmulti-field →keyword-only. The tokenized path was never reachable from KQL anyway (no dot-syntax + pre-fix(search): preserve value case for non-lowercased bleve fields #2633 lowercasing), so no working query regresses; aggregations now produce correct case-preserving buckets on both backends.location: the libregraph{longitude, latitude, altitude}object is preserved at thelocationkey on both backends (numeric sub-field queries likelocation.latitude:>49keep working). A siblinglocation_geopointis added for geo-distance / bounding-box / polygon queries.path:queries are case-sensitive on both backends; Path gets no_lowercasesibling. Paths act as references (location scoping, deep links) where/Fooand/fooare distinct siblings, and case-insensitive folder discovery is served byname:. This matches bleve on main, which always matched paths case-sensitively; only OpenSearch loses its case-insensitive path matching. Dropping the sibling also removes its costliest maintenance: Path is the one mutable sibling field, so the OpenSearch move script no longer has to rebuild a lowercased copy for a whole subtree. With paths case-sensitive, the ref path scope is applied at query level on both backends (term/prefix on bleve's keywordPath, term filter on OpenSearch'spath_hierarchytokens), replacing the old post-filter: totals and paging now respect the scope instead of being computed over the whole space, and a wrong-cased scope matches nothing.graph facet parsing: fail-soft per field. A malformed value drops only that field; the rest of the facet still populates.
Audio facet is now shown whenever
libre.graph.audio.*metadata is present: the read-sideaudio/guard is dropped from all three readers (bleve, OpenSearch, graph), so the facet follows the metadata rather than re-checking the MIME type. Extraction still only produces audio metadata foraudio/*files.OpenSearch Name/Tags tokenization now matches bleve: a single case-preserved keyword token (was word-tokenized), with case-insensitive matching handled by the
_lowercasesibling. No impact on the product: web always searches wildcarded (name:"*term*", see webuseSearch.ts), which matches regardless of tokenization; bleve has always been single-token. Only affects non-web clients sending a barename:reportand expecting a substring match.Mtime is typed as a date on both backends, so
mtime:>...ranges are chronological (was a keyword field / lexicographic compare on OpenSearch). Note: bleve has no sub-second date field, so a returned/re-indexedMtimeis second-precision (range queries stay exact). Previously the RFC3339Nano keyword round-tripped exactly.Resource.Hiddennow survives Move/Delete/Restore on bleve. The old hand-rolled deserializer never readHidden, so those ops silently reset it tofalse; the reflection deserializer reads every field, preserving it. Latent-bug fix.Case-insensitive search via per-field
_lowercasesiblings. Every keyword/path field indexes its case-preserved base (returned to clients, used for exact ops like the move/delete cascade) plus, when enabled, a lowercased sibling used only for matching. Queries route to the sibling and lowercase the value with the same Gostrings.ToLowerused at index time, so index and query stay consistent without an analyzer. The sibling is never read back, so it need not be stored: in bleve it is not stored, out of_all, no doc values; in OpenSearch it deliberately stays in_source, because excluding it would force every update-by-query script (move/delete/restore) to rebuild all siblings from the document via painlesstoLowerCase, which lowercases differently than Go and would drift from the query side. A lowercased copy of a name/path is negligible disk in a cluster. The OpenSearch Move script keeps base and sibling in sync via Go-lowercased params, so case-insensitive search still finds a file after it moves (the sibling used to go stale); bleve re-indexes whole documents on move/delete/restore and stays fresh for free. Also fixes a latentmainbug: the Move/Delete cascade matchedPathexactly against the lowercased index, so re-homing a mixed-case folder (e.g./Photos) skipped its descendants;Pathis now case-preserved. bleve path queries additionally match a folder and its descendants like OpenSearch'spath_hierarchy.OpenSearch full-text (
content:) search now analyzes the query. Single-term queries used an unanalyzed term query, so once this refactor dropped the old blanket query-value lowercasing,content:Foo(any uppercase) missed on OpenSearch (a regression introduced here; bleve was unaffected because its query analyzes). Fielded full-text queries now use an analyzed match query.Contenton OpenSearch also gets a porter stemming analyzer matching bleve's (it previously used the defaultstandardanalyzer and never stemmed, a pre-existing inconsistency), socontent:running/content:run/content:RUNNINGbehave the same on both backends.To decide - content wildcards (
content:foo*) are unanalyzed on both backends, so they match the stemmed, lowercased term dictionary literally:content:run*finds a document containingRunning(indexed termrun), butcontent:running*(past the stem) andcontent:Run*(uppercase) do not. This is inherent to a wildcard over an analyzed field and is now consistent across bleve and OpenSearch (previously OpenSearch degradedcontent:foo*to an exact match). Whether content wildcards should additionally be case-folded is left open.mediatype search is now case-insensitive on both backends: categories (
mediatype:Folder,mediatype:IMAGE, ...) and literal MIME types are lowercased in the KQL lowering. Related behavior change: the rawMimeTypefield no longer expands category words. On mainMimeType:folder/MimeType:fileexpanded to the folder / non-folder MIME set (a quirk of field-based expansion); nowmediatype:is the way to query categories andMimeType:only matches a literal MIME type.Upgrade note: the OpenSearch mapping (index
resource_v2) now lists all properties explicitly, so it differs from any existingresource_v2index. On startupApplyreturnsErrManualActionRequiredwith a clear message; operators upgrading in place must drop and reindexresource_v2. Fresh installs and the bleve backend are unaffected.Follow-ups (not in this PR)
NOT a OR bon OpenSearch: per [MS-KQL] (2.1.13, NOT has highest precedence) this is(NOT a) OR b, but a flat OpenSearch bool query cannot OR-combine amust_notclause, so it collapses toNOT a AND b. bleve represents it correctly (amust_notsub-query can be a disjunct). Rare (a NOT directly followed by OR) and a flat-bool limitation, not the NOT binding, which is now spec-correct on both backends.Media-type category + operator precedence:
X OR mediatype:<multi-type-category> AND Y(document/spreadsheet/presentation/archive) constrains only the last MIME type of the category, because the bleve compiler'smapBinaryredistributes a left disjunction as an OR-chain. Pre-existing (main had the same), a pure query-compiler issue.[]structround-trip: a json-tagged slice-of-struct field is mapped as a nested object but not read back (fillStruct/casing/geodo not descend into slices), and its_lowercase/_geopointsiblings would never be written. Latent: no current facet is a[]struct(facets are*struct, tags are[]string); harden before adding one.bleve
_all: unused (every query is fielded: the resolver always resolves a field, the compiler always emitsfield:value, and a bare term resolves toName→Name_lowercase). Disabling it shrinks the bleve index and turns the per-fieldIncludeInAllhandling into dead code to remove.KQL typed-value parity (booleans/numerics):
Hidden:T/Size:>1000are bleve query-string leakage: they parse as plain string restrictions and only work because the bleve compiler concatenates values into bleve's own query-string syntax; on OpenSearch the same literals hit typed fields and 400. The KQL-canonicalhidden:truecompiles to a query-stringHidden:trueon bleve and silently never matches (bleve indexes booleans as the term "T"); it should compile to a typed bool-field query. Canonical numeric ranges (Size>1000) don't parse at all, the grammar has range operators only for datetime values. Consolidate in the query layer: typed BooleanNode compile on bleve, grammar-level numeric ranges, then decide whether the query-string extras stay.?single-char wildcard parity: bleve treats?as a wildcard, OpenSearch does not (the wildcard check only looks for*), soname:Fo?diverges. Pre-existing, not introduced here.Field-name notation: KQL vs graph
sortProperties: thequeryStringexposes internal index field names (Name:foo*,Size:>1000,Tags:bar,Mtime:...), while the graph search endpoint'ssortPropertiesdeliberately accepts the property names a client sees on the hit resource (name,size,lastModifiedDateTime,mimeType,photo.takenDateTime, ...) and translates them to index fields via an alias table inpkg/search. Align the two surfaces: accept graph-notation field names in KQL (on top of that alias table) and decide whether the internal index names remain part of the public query surface.