Zulia 5.3.0

Zulia 5.3.0

New Features

• Transient indexes with lazy loading and eviction. A new per-index transientIndex flag (default false) lets an index load lazily and unload from memory when idle or when the node is over its transient cache bound, then reload on demand. This allows a node to serve many more indexes than fit in memory at once. New zulia.yaml settings control the behavior: transientIndexCacheSize, transientIndexIdleTimeoutSeconds, and transientIndexEvictReplicated (default false, so replicated indexes stay resident unless explicitly enabled). Every operation holds its index through a lease, so eviction always defers to in-flight work. The flag is settable at creation through ClientIndexConfig, and on existing indexes through UpdateIndex or zuliaadmin updateIndex --transientIndex. Existing and unflagged indexes keep the current behavior: load at startup and stay resident. GetNodes stats now report a per-index resident flag plus per-node resident index, load, and eviction counters.
• Vector quantization with oversampled full-precision rescore. Vector fields can now be stored quantized through a per-field Lucene codec, cutting graph-search memory roughly 4x at INT8 and more at the binary encodings. Supported encodings are FLOAT32, INT8, INT7, INT4, BBQ (1-bit Better Binary Quantization), and BBQ_2BIT, each with an HNSW graph or a flat exact index. A new VectorDescription on the field config records similarity, dimensions, and embedding model provenance, while VectorIndexingConfig on each IndexAs controls the encoding per representation, so one stored field can carry a BBQ representation next to a full-precision twin. Queries against quantized fields oversample candidates and rescore them with full-precision vectors before the cross-shard merge, tunable per query via vectorOversample (encoding-specific defaults apply, capped at 32x). The client adds FieldConfigBuilder.quantization/hnsw/flat/similarity/dimensions/model, VectorIndexingConfigBuilder, and VectorTopNQuery.oversample(float).

Breaking Changes

• New indexes quantize vector fields to INT8 by default. Indexes created on 5.3.0 or later store vector fields with 8-bit scalar quantization unless a field explicitly configures another encoding (including FLOAT32 to opt out). Existing indexes are unchanged and keep raw float32 vectors.
• Rolling upgrade note for vector users. Deploy this build to every node before creating or writing vector indexes. Replication and restarts open segments by their persisted codec name, so an older node cannot read segments written with the new per-field codec.

Improvements

• Null-safe document helpers. DocumentHelper gains getAsInt, getAsLong, getAsFloat, and getAsDouble accessors, including variants with default values, for reading numeric values out of stored documents without caring whether the source encoded them as int, long, or double.

Bug Fixes

• Index and taxonomy commits are now crash consistent under concurrent stores. The taxonomy was committed before the index, so a store landing between the two commits could leave a committed document referencing a facet ordinal that the committed taxonomy did not yet contain. Commit now uses Lucene's two-phase pattern: the index commit's document set is frozen first, then the taxonomy is made durable, then the frozen set is published.
• Int and long facet labels now match the declared field type. A whole number stored as a JSON or MongoDB double (for example 2020.0) was indexed and sorted as 2020 but faceted as "2020.0". Facet labels for NUMERIC_INT and NUMERIC_LONG fields are now coerced to the declared type. Existing taxonomies keep their old labels until a reindex, so a field can briefly show both forms.
• Query parser error handling and range fixes. Malformed numeric arguments such as (a b)@2.5 now report a clean parse error instead of an internal NumberFormatException. Exclusive int and long range bounds no longer overflow at the type extremes, so intField:{2147483647 TO *] returns no documents instead of matching everything. A malformed field boost like title^abc now throws the intended descriptive error. Multi-field configuration errors report a descriptive message instead of an opaque NPE.
• Resource leaks on the index load and unload paths. Single-node filesystem mode leaked a file handle on every settings read. A failure after the Lucene IndexWriter opened leaked the writer and its write.lock until JVM restart. A rollback failure during shard close skipped cleanup of the remaining resources.
• Federated requests join every subtask before reporting a failure. Previously the first failed subtask was rethrown while the rest were abandoned, letting a still-running subtask outlive its request. The first failure is now thrown with the others attached as suppressed exceptions.
• Client connection pool close races. Closing a connection during node removal no longer risks an NPE and instead fails with a clean retriable status. A closed pool now rejects new work up front, drains fully, and no longer masks a command's real failure with a pool-closed error on retry.
• zulia-ui-rest security fixes. Anonymous user creation allowed admin self-provisioning, a hardcoded default JWT signing secret enabled token forgery, and every authenticated user was granted ADMIN unconditionally. All three are fixed (see Breaking Changes for the new provisioning flow). This module is a work in progress, so production usage is not affected.
• zulia-data robustness fixes. A malformed line no longer sends JsonLineDataSource into an infinite loop when a non-throwing error handler is configured. Blank Excel rows read as empty instead of throwing NPEs, boolean cells are no longer dropped, and very large whole numbers no longer clamp to Long.MAX_VALUE. TSVSource.withDefaults actually splits on tabs instead of commas. JsonArraySource fails loudly on a non-object array element instead of silently truncating. JsonSourceRecord numeric getters no longer throw ClassCastException on JSON whole numbers. Empty single-use sources iterate to an empty result instead of throwing. NumberCSVWriter preserves BigDecimal, BigInteger, Decimal128, and other Number subtypes instead of writing null cells.

Dependencies

• Lucene 10.4.0 to 10.5.0 (required for the new vector quantization formats)
• gRPC 1.80.0 to 1.82.1 and Protobuf 4.34.1 to 4.35.1
• MongoDB driver 5.7.0 to 5.8.0
• jackson-databind 3.1.3 to 3.2.0 (clears a flagged vulnerability)
• Gradle 9.4.1 to 9.6.0 with plugin updates (Micronaut application 5.0.1, protobuf plugin 0.10.0, vanniktech maven publish 0.37.0)
• JUnit 6.0.3 to 6.1.0, plus minor updates to commons-codec, commons-io, FastCSV, Gson, jansi, JNA, Log4j API, Logback, and SLF4J