Skip to content

v1.30.0 - Runtime config management, Dynamic user management, Dynamic RAG model selection, Multi-value vector support, BlockMax WAND-based BM25

Choose a tag to compare

View all tags
@antas-marcin antas-marcin released this 03 Apr 19:39
· 11637 commits to main since this release
v1.30.0
b7b7715

Breaking Changes

BlockMax WAND Migration

This release contains the migration process to enable migration of existing property indexes to the new BlockMax WAND-based file format. This step is required to take advantage of the performance improvements. Note that the migration process produces segment files which are not backwards compatible with previous versions.

New Features

Runtime configuration management (GA)

Dynamically update select configuration variables at runtime—no restarts, no downtime. This streamlines operational workflows, enabling rapid experimentation and fine-tuning without disrupting active processes. Built with extensibility in mind, this lays the groundwork for broader runtime configurability in future releases, making the system more adaptable and resilient.

  • feat(runtime-config): Add support for runtime config manager by @kavirajk in #7344
  • feat: Integrate runtime config for maximum_allowed_collection_limit by @kavirajk in #7501
  • feat: Support AUTOSCHEMA_ENABLED as runtime overrides by @kavirajk in #7572
  • feat: Support ASYNC_REPLICATION_DISABLED as runtime overrides by @kavirajk in #7573
  • chore: Some cleanup related to runtime overrides. by @kavirajk in #7605

Dynamic user management (GA)

User and API key management just got a major upgrade with dynamic configuration. With a simple REST API, you can now create, update, and revoke users and API keys on the fly. This makes it easier to integrate with automated workflows, improve security by rotating credentials seamlessly, and adapt access controls in real time.

Dynamic RAG model selection (GA)

Adapt your retrieval-augmented generation (RAG) pipeline on the fly by selecting the generative model at runtime. This flexibility allows you to optimize responses based on query complexity, cost constraints, or latency requirements—switching between models dynamically to get the best results for each use case. No more one-size-fits-all; now every query gets the model it deserves.

  • Support images in Dynamic RAG group generation by @tsmith023 in #7313
  • Double timeouts over GQL and gRPC for generative module tests by @tsmith023 in #7342
  • Dynamic Users: Create endpoints and wire them up by @dirkkul in #7317
  • Dynamic user management: Block creating dynamic users with same name as static users by @dirkkul in #7385
  • Split images and imageProperties in dynamic provider by @tsmith023 in #7386
  • Enable Dynamic RAG syntax by @antas-marcin in #7416
  • Remove redundant old code adding nvidia case to extractFromQuery by @tsmith023 in #7435
  • chore: add an option of falling back to ENABLE_EXPERIMENTAL_DYNAMIC_RAG_SYNTAX env variable by @antas-marcin in #7442
  • Make model optional in GenerativeOpenAI by @tsmith023 in #7479
  • chore: change the order of generative images passed with prompt by @antas-marcin in #7727

BlockMax WAND-based BM25 (GA)

Keyword search just got a major speed boost with BlockMax WAND, enabling significantly faster BM25 ranking. This enhancement accelerates query execution while maintaining relevance, making large-scale search workloads more efficient. A provided production-ready online migration process ensures seamless adoption without downtime, allowing for a smooth transition for existing datasets.

  • Fix/Blockmax tombstone handling and global idf computation by @amourao in #7535
  • Feat: Blockmax migrator by @aliszka in #7578
  • Improvements: blockmax migrator concurrent objects read/process by @aliszka in #7607
  • Fix(bm25_block): Memtable filtered count by @amourao in #7640
  • feat(bm25_block): ✨ Use inverted format and blockmax by default by @amourao in #7639
  • fix: unswapped buckets with map strategy by @aliszka in #7650
  • Feat: shard reindexer v3 and improved blockmax migration by @aliszka in #7670
  • Improvement (blockmax migration): Revert lazy shutdown by @aliszka in #7708
  • fix(bm25_block): 🐛 Fix concurrent access to segment and memtable tombstones by @amourao in #7735

Multi-value vector support (GA)

Multi-value vector search is now production-ready, bringing robust support for ColBERT-style embeddings. This allows for more expressive and granular retrieval, improving performance on tasks that benefit from late interaction scoring. Previously experimental, this release is battle-tested for real-world workloads, delivering scalable and efficient multi-vector search at production scale.

Additionally, all multi-vector indexes now support all quantization options.

Modules

  • Add xAI Generative Module by @erika-cardenas in #7487

Performance & Observability Improvements

Testing Improvements

Chores and Docs

  • Remove old locking mechanism by @dirkkul in #7135
  • chore(dependabot): change interval to monthly by @moogacs in #7191
  • doc: remove duplicated blog link in readme by @kumarlokesh in #7124
  • chore: download gRPC health probe instead of building it by @antas-marcin in #7293
  • Makefile: add --load flag to buildx to add it to local registry by @kavirajk in #7296
  • chore: makes number of goroutines stared by cyclemanger configurable by @aliszka in #7311
  • [HNSW] Expose more quantization stats by @trengrj in #7297
  • chore: iterative replace cursor by @jeroiraz in #7323
  • improvement: DX - show only vector names in error msg with wrong target vector by @melsonic in #7329
  • Upgrade golanci-lint by @dirkkul in #7350
  • async: add ASYNC_INDEXING_BATCH_SIZE env var by @asdine in #7348
  • [CLU-30] ensure catching up log is not printed past catching up by @reyreaud-l in #7349
  • chore: Add concurrency limit to shard loading by @faustuzas in #7343
  • chore: Refactor code to avoid checking whether target vectors are enabled by @faustuzas in #7357
  • chore: log shutdown error instead of panicking by @faustuzas in #7359
  • Remove grpc_health_probe binary from Weaviate's docker image by @antas-marcin in #7358
  • refactor: replace golang.org/x/exp with stdlib by @Juneezee in #7171
  • chore: Refactor code to avoid checking whether target vectors are enabled II by @faustuzas in #7363
  • chore(ci): enable multiarch docker image builds for branch builds by @antas-marcin in #7376
  • chore: refactor shard metrics to capture more use cases by @faustuzas in #7373
  • test: check nodes are running after restart by @jeroiraz in #7375
  • chore(acceptance-test): add class with limitation by @moogacs in #7392
  • Text2vec weaviate auth by @augustas1 in #7404
  • [CLU-35] store node id next to their addr to improve logging when bootstrapping raft by @reyreaud-l in #7395
  • chore: refactor dimension tracking to support mixed vectors by @faustuzas in #7414
  • chore: batch objects to propagate in non-consecutive ranges by @jeroiraz in #7405
  • chore: refactor to use less checking for enabled named vectors III by @faustuzas in #7425
  • housekeeping: update go client version to v5 by @antas-marcin in #7419
  • Binary quantization using bit shifts. by @tobias-weaviate in #7430
  • chore: refactor to use less checking for enabled named vectors IV by @faustuzas in #7433
  • chore: rename env variable enabling dynamic RAG syntax by @antas-marcin in #7440
  • chore: add go sched latency metric by @faustuzas in #7437
  • chore(test) fix test usage of reflect because of changes of in go reflect pkg 1.24 by @moogacs in #7450
  • Build nodes api multinode backport by @jfrancoa in #7456
  • chore: remove high cardinality target vector tracking metrics by @faustuzas in #7438
  • chore: refactor module config to support mixed vectors by @faustuzas in #7445
  • chore: initialize mixed vector indexes in shard by @faustuzas in #7471
  • chore: fix flaky RAFT bootstrap test by @moogacs in #7475
  • chore: add support for mixed vectors in class creation by @faustuzas in #7465
  • Validate all property fields besides description when updating collections by @tsmith023 in #7480
  • feat: add support for mixed vectors in ingestion by @faustuzas in #7482
  • chore(raft): add concurrency tests for RAFT schema ops by @moogacs in #7508
  • chore: remove unnecessary deep copy of Sharding state by @antas-marcin in #7521
  • remove unused experimental FQDN resolver by @reyreaud-l in #7472
  • chore: add support for ollama in dev environment by @faustuzas in #7527
  • chore: replica finder remove unused named error by @nathanwilk7 in #7530
  • Take out authz check when performing auto tenant activation by @tsmith023 in #7537
  • chore(tests): remove AI21 models from generative-aws module tests that reached end of life by @antas-marcin in #7538
  • chore(sec): security updates #1 by @antas-marcin in #7547
  • chore: experimental flag to reject collection creation with mixed vectors by @faustuzas in #7550
  • chore: add acceptance test for mixed vectors in backups by @faustuzas in #7548
  • chore(recovery-logging): log empty recovery wal filenames once per dir by @moogacs in #7576
  • Use errorgroup to limit concurrency in batch inserts by @dirkkul in #7601
  • Set GH action test runners on dedicated group by @parkerduckworth in #7582
  • chore: yield processor during hashtree initialization by @jeroiraz in #7566
  • backport: chore: refactor vector index and queue access to be thread-safe by @faustuzas in #7651

Security Updates

  • dependencies(update): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.5.0 to 1.6.0 by @dependabot in #7185
  • dependencies(update): bump github.com/aws/aws-sdk-go-v2 from 1.36.0 to 1.36.1 by @dependabot in #7184
  • dependencies(update): bump golang.org/x/sync from 0.10.0 to 0.11.0 by @dependabot in #7188
  • dependencies(update): bump golang.org/x/text from 0.21.0 to 0.22.0 by @dependabot in #7187
  • sec: project security updates #1 by @antas-marcin in #7199
  • Upgrade golanci-lint by @dirkkul in #7353
  • Ugrade go-jose to v4.0.5 by @dirkkul in #7352
  • chore(deps): upgrade jwt dependency to latest v4.5.2 version by @antas-marcin in #7687

Fixes

  • fix: set timeout for boltdb initialization to prevent infinite loop by @llinvokerl in #7210
  • common: add PagedCache implementation by @asdine in #7303
  • fix: pass initialPages argument to constructor by @bevzzz in #7308
  • fix(backup): memory control consumption by setting minio specific part size by @moogacs in #7057
  • Fix query search degration after restart by @robbespo00 in #7269
  • fix: prevent race during shard initialization by @jeroiraz in #7319
  • fix: validate requested secondary index is present in the segment by @jeroiraz in #7315
  • Remove ai21.j2-mid-v1 from generative-aws module tests by @tsmith023 in #7345
  • fix: default value for segment checksum enablement by @faustuzas in #7322
  • shutdown before replacing buckets to avoid race condition by @donomii in #6997
  • fix: allows concurrent writes and search in rangeable memtable by @aliszka in #7377
  • fix: include last update time during object propagation by @jeroiraz in #7365
  • fix: include last update time during object propagation by @jfrancoa in #7372
  • fix: remove empty wal upon start by @jeroiraz in #7025
  • fix: do not back up subdirectories by @faustuzas in #7418
  • fix(raft): stop retrying in case of error in applies by @moogacs in #7421
  • fix: getting default maxTokens value in generative anthropic module by @antas-marcin in #7439
  • Fix nodes api with collection name on multi node [backport] by @dirkkul in #7454
  • fix: add locks around vector index configuration access by @faustuzas in #7468
  • Fix parsing and replying of Debug and ReturnMetadata by @tsmith023 in #7467
  • fix: vector nil and empty considered equal by @aliszka in #7469
  • fix: linter errors in TestUpdatePropertyDescription test by @antas-marcin in #7502
  • fix(raft): data races on schema.Classes by @moogacs in #7506
  • fix(db-internals): handle cases where class is nil due to EC by @moogacs in #7509
  • fix(raft): randomize shard owner to tolerate down nodes by @moogacs in #7515
  • fix: improve vector selection in queries for mixed vectors by @faustuzas in #7518
  • fix: add mixed vector support in aggregate queries by @faustuzas in #7531
  • Remove deprecated codepath for pre-1.23 grpc code by @tsmith023 in #7536
  • Handle missing or not parseable numbers from openai by @dirkkul in #7562
  • Fix class capitalisation in batch with autoschema by @tsmith023 in #7545
  • fix: propagate distance parameter in remote search by @faustuzas in #7569
  • fix: context-aware aggreation when no filters are specified by @jeroiraz in #7559
  • fix: add BYOV support for mixed vectors by @faustuzas in #7575
  • fix: add mixed vector support in near queries by @faustuzas in #7543
  • fix: provide a better error message when configuration for named vector does not exist by @faustuzas in #7603
  • async indexing: fix deadlock during upgrade of dynamic index by @asdine in #7602
  • Fix integer conversion and assignment by @donomii in #7614
  • Port fix/deadlock dynamic index by @asdine in #7616
  • fix: upgrade jwt library by @faustuzas in #7618
  • fix: add support for mixed vectors in grpc by @faustuzas in #7589
  • Fix potential vectorization batch deadlock by @dirkkul in #7658
  • Revert "Merge pull request #7406 from weaviate/hnsw-nodes-refactor" by @asdine in #7643
  • fix: fix race condition in config manager test by @faustuzas in #7665
  • If nested object is nil, return NilValue in grpc by @tsmith023 in #7680
  • Fix typos in username in auth test by @dirkkul in #7690
  • fix: prevent creating dynamic index with async indexing disabled by @faustuzas in #7701
  • Fix race between delete and deactive/activate by @dirkkul in #7723

Others

New Contributors

Full Changelog: v1.29.0...v1.30.0

Contributors

aliszka, kumarlokesh, and 26 other contributors
Assets 17
Loading