0.0.2-alpha

WebExpress.WebIndex 2.0.0-alpha

Release date: 2026-10-31
Target framework: .NET 10 (net10.0)
Blog: WebExpress 2.0.0-alpha – Halloween update

This release reworks the WebExpress Query Language (WQL) from the ground up. It adds a precedence-aware grammar, new operators, more than twenty built-in functions and typed literals. A WQL statement can now be translated for database query providers. The on-disk reverse indexes now use self-balancing posting trees and are rebuilt automatically when their format changes.

Highlights

• WQL rework: not, operator precedence, between/not between, starts with/ends with/contains, !~, typed literals, comparisons between attributes, and functions on both sides of a condition.
• Database-friendly queries: ToQuery(WqlQueryMode.Translatable) builds expression trees that providers such as Entity Framework can translate into SQL.
• [WqlIgnore]: keeps a property out of WQL statements that may come from untrusted callers.
• Balanced posting trees: posting nodes of the term and numeric indexes are stored as AVL trees, so lookups no longer degrade to lists for sequential document ids.
• Automatic index rebuild: reverse index files of an older format, missing files and interrupted rebuilds are restored from the document store on open, with no manual reindex.
• Change notifications: the new IndexManager.Changed event reports inserts, updates, deletes and clears. A host can use it to keep several copies of an index in sync.

New Features

WQL grammar and operators

• New grammar with operator precedence: not binds most strongly, then and (&), then or (||). Chains are grouped from the left, and parentheses override precedence.
• New conditions:
  • not <filter> to negate a condition or a group (WqlExpressionNodeFilterNot).
  • between / not between with inclusive bounds (WqlExpressionNodeFilterConditionRangeBetween, ...RangeNotBetween).
  • starts with, ends with, contains: case-insensitive substring tests, also inside a word.
  • !~ as the complement of the word search ~ (WqlExpressionNodeFilterConditionBinaryNotLike).
• A function or a nested attribute path (Address.City) may stand on the left-hand side of a condition. An attribute may stand on the right-hand side (Number > Optional).
• Typed literals: numbers, single- or double-quoted strings (quotes are escaped by doubling them, e.g. 'O''Brien'), unquoted words, null, true and false. A literal is converted into the type of the attribute it is compared with. This covers strings, all numeric types including nullable forms and decimal, bool, DateTime, DateTimeOffset, DateOnly, TimeOnly, TimeSpan, Guid, char and enumerations (by member name or number). A conversion that would lose information is reported as an error (WqlValueConverter).
• Null semantics now follow C# nullable rules. The documentation describes how null behaves in comparisons, ranges, text searches, negations and sorting.
• skip and take form one window, applied in the order they are written, and both execution paths use the same window.

WQL functions

New built-in functions in addition to now(), day(), month(), year(), len(), lower(), upper() and trim():

Function	Purpose
date(text)	Reads an ISO 8601 date.
dateadd(unit, n, date), datediff(unit, start, end)	Date arithmetic with the units year, month, day, hour, minute and second (WqlDateUnit).
yearof(date), monthof(date), dayof(date)	Extract parts of a date.
concat(...), replace(text, search, replacement), substring(text, start, [length])	Text functions.
coalesce(...), nullif(a, b)	Null handling.
abs(n), round(n, [digits]), floor(n), ceil(n)	Numeric functions.

• day(), month() and year() now take an optional offset.
• Functions can be nested, and their arguments may be literals, attributes or other functions.
• The new base class WqlExpressionNodeFilterFunctionScalar<TIndexItem> makes custom functions easier to write. A single Evaluate method serves both execution paths. The constructor declares the minimum and maximum argument count, which the parser checks. ReturnType declares the result type. Functions written against WqlExpressionNodeFilterFunction<TIndexItem> keep working.
• WqlExpressionNodeFilterFunction<TIndexItem> gains MinArguments, MaxArguments and IsConstant.

Query execution paths

• IWqlStatement.ToQuery(WqlQueryMode mode) with the new enum WqlQueryMode:
  • Exact (the default for ToQuery()) evaluates text conditions in memory with the new WqlTextMatcher. The matcher uses the index's tokenizer, case and diacritic folding, phrases, distances, wildcards and similarity.
  • Translatable rewrites the tree into null tests, ToLower, Contains, StartsWith, EndsWith and string.Compare, which database providers can translate. A text search then selects a superset of the exact result, and a negated text search selects a subset.
• WqlTextMatcher.Terms exposes the folded words of a text search for providers that build their own conditions.
• In the index path (Apply(), IndexManager.Retrieve()), each condition decides whether its reverse index answers it exactly. Otherwise the condition scans the document store with the same compiled predicate that ToQuery() creates. Conditions retrieve their matches without the reverse index result limit, so combinations and complements stay exact beyond 10,000 hits.

Query API (IQuery<TIndexItem>)

• New members: Not(), WhereNot, WhereNotEquals (string and Guid), WhereIn, WhereGreaterThan, WhereGreaterThanOrEqual, WhereLessThan, WhereLessThanOrEqual, WhereBetween, WhereElementEquals, WhereElementEqualsIgnoreCase, WhereIsNullOrEmpty, WhereIsNotNullOrEmpty, ThenByAsc, ThenByDesc, WithSkip and WithTake.
• WithSkip allows an offset without a limit. WithTake accepts a limit of 0, which yields an empty result.

Security

• New WqlIgnoreAttribute ([WqlIgnore], namespace WebExpress.WebIndex.WebAttribute). A property with this attribute, and any path through it, cannot be filtered, sorted or used as a function argument in WQL. It is reported as an unknown attribute, and on the right-hand side its name reads as a plain word. IndexIgnore only skips the reverse index and leaves the property queryable by a scan; [WqlIgnore] is independent of it.

Search

• IndexRetrieveOptions.Similarity: minimum similarity in percent (1–99) for fuzzy searches. 0 and 100 mean exact matching, and values above 100 are rejected as a syntax error.
• Phrase and proximity searches consider every occurrence of a word. A document matches as soon as one arrangement of its words fits the distance (IndexTermMatcher, shared by the memory and the file-based term index).
• A distance now loosens a phrase search (=): each word may stand up to that many positions further from the previous one than in the query.
• Similarity and distance can be combined. The similarity applies to every word of the query.

Index management

• IndexManager.Changed event with IndexChangedEventArgs (ItemType, Kind, Item) and the enum IndexChangeKind (Insert, Update, Delete, Clear). A bulk reindex raises no events. OnChanged is protected virtual.
• IndexManager.DocumentTypes returns a snapshot of the item types that have a registered document.

Storage

• Reverse index files carry a format version, which is now 2 for wrt (term) and wrn (numeric) files. If a file with a different version is opened, it is discarded and rebuilt from the document store. A missing file is also rebuilt if the document store holds items.
• A *.rebuild marker file exists next to an index for as long as its rebuild runs. If an interrupted rebuild leaves the marker behind, the partial file is discarded and rebuilt on the next start.
• All reverse indexes of a type that need a rebuild are filled from a single pass over the document store.

Language assets

• The German and English stop word, synonym, irregular word and misspelling dictionaries were expanded and normalized.

Performance

• AVL-balanced posting trees in the term and numeric indexes bound each document lookup to about 1.44 log2(n) nodes. Previously, sequential ids could degrade the tree into a list.
• Cache maintenance in the storage buffer no longer allocates on each tick. Evicting entries no longer rebuilds the read cache dictionary.
• Storage segments are created through a cached factory (IndexSegmentFactory) instead of Activator.CreateInstance.
• Reverse indexes that need a rebuild are filled together in a single pass over the document store.

Bug Fixes

• Fuzzy search: the similarity is now 1 − Levenshtein distance / length of the longer word. Previously the raw distance ratio was returned, which inverted the similarity.
• Tokenizer: control characters such as line breaks now separate tokens exactly like whitespace and advance the token position. Phrase searches across line breaks no longer break.
• WQL paging: ToQuery() passed skip and take to the query in swapped order. Both execution paths now apply the same window.
• Query composition: And(IQuery) and Or(IQuery) now treat each side as the conjunction of its filters, so Where(a).Where(b).Or(c) means (a && b) || c. Combining with a side that has no filters returns the other side unchanged instead of failing or widening the result.
• Resource leaks: if opening an index document or an index file fails, for example because of a foreign header or a truncated segment, the files are released immediately and do not stay locked.
• Storage resets: the initialization and reset of index files are serialized with buffer maintenance, so direct stream writes cannot lose their position.
• Culture: value comparisons on update use the index culture instead of the current thread culture.
• Error reporting: new error messages in German and English for invalid similarity or distance, unconvertible literals, options on non-text operators, text operators on non-text attributes, invalid values after is/is not, and wrong function argument counts.

Breaking Changes

• IWqlStatement has a new member ToQuery(WqlQueryMode mode).
• IQuery<TIndexItem> has new members (Not, WhereNot, WhereNotEquals, WhereIn, WhereGreaterThan, WhereGreaterThanOrEqual, WhereLessThan, WhereLessThanOrEqual, WhereBetween, WhereElementEquals, WhereElementEqualsIgnoreCase, WhereIsNullOrEmpty, WhereIsNotNullOrEmpty, ThenByAsc, ThenByDesc, WithSkip, WithTake). Custom implementations of either interface must implement them.
• Removed type: WebExpress.WebIndex.Storage.IndexStorageBinaryTreeDirection.
• Built-in functions (now, day, month, year, len, lower, upper, trim) now derive from WqlExpressionNodeFilterFunctionScalar<TIndexItem>. The binary and set conditions were restructured on new base classes (WqlExpressionNodeFilterConditionBinaryComparison, WqlExpressionNodeFilterConditionRange).
• Erroneous statements: Apply() and ToQuery() throw an InvalidOperationException if the statement contains errors.
• Stricter parsing: the parser now reports unknown attributes in the sort order, literals that cannot be converted, similarity or distance on non-text operators, text operators on non-text attributes, values other than null/true/false after is/is not, wrong argument counts and similarities above 100. Statements that used to pass may now be reported as errors.
• Unquoted words that name an attribute of the item now refer to that attribute instead of being read as a string. Quote the value to compare against the literal text.
• Query composition: the changed semantics of And(IQuery)/Or(IQuery) (see Bug Fixes) can change the results of existing code.
• Fuzzy search: because the similarity calculation was corrected, fuzzy queries now return different results.
• Index files: existing wrt and wrn files (format version 1) are discarded and rebuilt from the document store when they are first opened. Plan for the rebuild time with large indexes. The document store format (wds, version 1) is unchanged.
• Solution file: src/WebExpress.WebIndex.sln was replaced by src/WebExpress.WebIndex.slnx.

Dependencies

Package	Change	Version	Licence
Microsoft.Sbom.Targets	added (build-time only, PrivateAssets="all")	4.1.13	MIT
xunit.v3 (test projects)	updated	3.2.2 → 4.0.2	Apache-2.0
Microsoft.NET.Test.Sdk (test projects)	removed	18.5.1	MIT

The library WebExpress.WebIndex has no runtime package dependencies. The new THIRD-PARTY-NOTICES.md lists the direct third-party components with links to their licence files and the full licence texts.

Build and Tests

• Version raised to 2.0.0.0 (package version 2.0.0-alpha).
• Release builds generate an SPDX SBOM. Microsoft.Sbom.Targets embeds the manifest in the NuGet package, and a copy is placed next to the package as <packageid>.spdx.json with a .sha256 checksum. Visual Studio builds hand the SBOM generation to dotnet msbuild.
• dotnet test runs on Microsoft.Testing.Platform, configured in the new global.json. Microsoft.NET.Test.Sdk and the TestingPlatformDotnetTestSupport property were removed.
• New tests cover the WQL infrastructure, semantics, conditions, expressions, consistency between execution paths and [WqlIgnore], posting trees, positional search, storage format versions and language assets. The number of test cases in WebExpress.WebIndex.Test rose from 590 to 689.
• The index tests now run in parallel, because each fixture works in its own index directory.
• GitHub Actions for the documentation workflow updated (actions/checkout@v7, actions/setup-dotnet@v6, actions/upload-pages-artifact@v5, actions/deploy-pages@v5).

Documentation

• docs/concept.md:
  • New WQL grammar, operator precedence, literals, the operator table and the null rules.
  • Error reporting, the two execution paths and WqlQueryMode, and the new function reference including custom functions on top of WqlExpressionNodeFilterFunctionScalar<TIndexItem>.
  • Corrected file names (.wrt/.wrn), format versions and automatic rebuild, and the AVL posting tree layout.
  • How the WebExpress.WebApp REST endpoints use WqlQueryMode.
• New THIRD-PARTY-NOTICES.md.
• README.md: revised project description, corrected links to the WebExpress guides and a new AI transparency notice.