v1.26.0 - Tenant Offloading, Multi-Target Vector Search, Scalar Quantization, Async Replication, Improved Range Queries

Breaking Changes

Tenant activity status update requests now limited to 100 tenants. This is mitigated by the official client libraries which batch requests containing greater than 100 tenants in the background.

New Features

Tenant Offloading

Designed to optimize storage costs and improve resource utilization by separating compute and storage for inactive tenants. This new feature allows inactive tenant data to be offloaded to object storage, thus reducing expenses by not paying for compute resources when the data is not actively used. This initial iteration is S3 compatible, and official support for other object storage solutions is coming soon.

• Allow tenant FROZEN status REST & gRPC by @moogacs in #5072
• S3 offload tenants module by @moogacs in #5074
• Offload shards to S3 upload/download paths by @moogacs in #5119
• Add offloading tenant states to gRPC API by @tsmith023 in #5224
• Add offload upload/download metric by @moogacs in #5273
• Remove tenant unused UNFROZEN status by @moogacs in #5296
• Add read-only tenant statuses to OpenAPI spec by @tsmith023 in #5289
• Delete frozen tenants in cloud on tenant or class deletion by @moogacs in #5288
• Pass error correctly in index.dropCloudShards by @moogacs in #5303
• Rename S3_ENDPOINT_URL flag to OFFLOAD_S3_ENDPOINT by @moogacs in #5304
• Allow new tenant names as input for Add and Update by @dirkkul in #5309
• Increase offload module timeout default to 120s by @moogacs in #5357
• Prevent tenant freezing when offload module not available by @parkerduckworth in #5354
• Offload tenants stuck on unfreezing by @moogacs in #5361
• Offload auto tenant de/activate by @moogacs in #5366
• Allow concurrent tenants update by @moogacs in #5367
• Update tenants offload tests timeout by @moogacs in #5371
• Offloading unfreeze concurrency loop variable capture by @moogacs in #5376
• Offload module: log level error only for default by @moogacs in #5377
• Improve performance update tenant upload/download process all at once by @moogacs in #5383
• Refactor update tenants validation by @moogacs in #5390
• Make offload bucket auto creation configurable by @moogacs in #5392
• Acceptance test for backup & offload by @moogacs in #5395
• Adjust the WAL recovery log to include on offloaded by @moogacs in #5409
• Delete class will check for frozen tenant before cloud deletion by @moogacs in #5408
• Abort on any shard error during offloading by @moogacs in #5416

Multi-Target Vector Search

Significantly enhances search capabilities by allowing users to perform searches across multiple vectors simultaneously. This feature improves search efficiency and relevance by querying multiple vectors in a single operation, enabling more comprehensive and accurate data retrieval.

• Named vector multi search by @dirkkul in #5099
• Refactor dto.Get struct by @dirkkul in #5173
• Refactor multi target search by @dirkkul in #5190
• Add GQL support for multi target search by @dirkkul in #5172
• More multi target search fixes by @dirkkul in #5192
• Add separate vector per target for multi target search by @dirkkul in #5216
• Fix targets for near vector GQL subsearch by @dirkkul in #5329
• Disable certainty for multi vector search by @dirkkul in #5344
• More multi target tests and fixes by @dirkkul in #5346
• Multi target/fix near object with different length vectors by @tsmith023 in #5389
• Fix multi target search for BQ without cache by @dirkkul in #5391

Scalar Quantization

Optimizes storage and enhances search performance. Scalar Quantization compresses vector data by mapping the vector's floating point values to integers, significantly reducing storage size while maintaining accuracy. This compression allows for faster and more efficient searches by decreasing the amount of data processed.

• Add SIMD l2 distance calculation by @asdine in #5095
• Dot byte implementation by @asdine in #5097
• Scalar Quantization by @abdelr in #5125
• Fix compressor logger for scalar quantization by @trengrj in #5307
• Add vector generator function for BQ and SQ by @robbespo00 in #5399

Async Replication

Ensures data consistency and integrity across replicas by asynchronously replicating changes. The Merkle tree-based implementation allows for efficient comparison and synchronization of data, identifying differences quickly and reducing replication lag. This asynchronous approach minimizes the impact on performance while ensuring that all replicas are up-to-date, providing a robust and reliable HA solution

• Asynchronous Replication by @jeroiraz in #4245
• Do not omit async replication attribute when disabled by @jeroiraz in #5175
• Reduce hashtree sizing by @jeroiraz in #5176
• Fix async replication leaks by @jeroiraz in #5222
• Ensure shardstate is up to date by @jeroiraz in #5233
• Consider the case when the hashtree has been explicitly set to nil by @jeroiraz in #5275
• Support deleted obj during async replication by @jeroiraz in #5314
• Fix enabling async replication on existing class, and multi-tenancy validation by @parkerduckworth in #5343
• Use compact hashtree even when multi-tenancy is not enabled by @jeroiraz in #5333
• Set a limit of objects being propagated per hashbeat iteration by @jeroiraz in #5368
• Improve err handling when fetching shard node name by @jeroiraz in #5415

Improved Range Queries

Introduces a new range filter index type, drastically improving the performance of large scale queries on numeric ranges.

• Range roaring set index by @aliszka in #5128
• Bucket range reader: improved panic handling by @aliszka in #5236
• Range roaring set index tests by @aliszka in #5280
• Range roaring set index - rename indexRangeble to indexRangeFilters by @aliszka in #5299
• Migrate existing class properties so IndexRangeFilters won't be nil by @parkerduckworth in #5347

Other

• Added an environment variable for disabling the go profiler setup by @bennycortese in #4290
• Remove duplicate contextionary url in dev setup by @aminst in #4806
• Deprecate old schema impl. before RAFT by @moogacs in #4943
• Use SafeErrorCompounder in migrator by @moogacs in #5388
• Add reindex endpoint by @asdine in #5305

Module Improvements

• Add API based modules by default by @databyjp in #5002
• Add dynamic generative modules syntax (module system and GraphQL support) by @antas-marcin in #5238
• Fix gRPC handling of dynamic generate result by @antas-marcin in #5261
• Module Generative Anthropic by @cdpierse in #5210
• Add Generative Anthropic e2e tests by @antas-marcin in #5290
• Add multi2vec-palm to the list of API-based modules to enable by @databyjp in #5313

Performance Improvements

• Concurrent batch vectorization requests by @dirkkul in #4827
• Add Dot Byte SIMD implementation for ARM64 by @ajit283 in #4756
• Dot graviton sve by @ajit283 in #4797
• L2 Byte ARM64 by @ajit283 in #5147
• Simpler WAL format for roaringSet by @amourao in #4358
• HNSW: set default max connections to 32 by @asdine in #5248
• Raft: retry leader not found and refactor the retry policy by @moogacs in #5412
• Improve update tenant in raft schema by @reyreaud-l in #5422

Testing Improvements

• Add retry on failure in CI for tests instability by @reyreaud-l in #4848
• Refactor replication acceptance test logic and always use 3 nodes by @moogacs in #4835
• Adds text-to-vector bigram, a rapid built in vectoriser by @donomii in #4468
• SIMD: fix flaky test by @asdine in #5160
• Update expected max connections in tests by @asdine in #5262
• Improve dynamic generative syntax tests by @antas-marcin in #5286
• Add possible vector index types e2e sanity tests by @antas-marcin in #5301
• Further divide acceptance with go client e2e tests by @antas-marcin in #5405
• PyTest import fix by @moogacs in #5425

Fixes

• Don't return shutdown error in case the shard was already shutdow… by @moogacs in #4837
• Fix Dot Graviton SVE by @ajit283 in #5141
• Fix vectorizer client. by @mindaugasrukas in #5171
• Replace if targetVectors != nil with if vectorSearch when extracting metadata.certainty by @tsmith023 in #5217
• Fix panic by add missing wrapper func in RAFT apply and offload grpc client tests by @moogacs in #5255
• Fix error when performing near vector search with no target vector and a single named vector config by @tsmith023 in #5274
• Fix batch updates of empty arrays by @dirkkul in #5418

Docs & Chores

• Remove blankspace from readme by @reyreaud-l in #4917
• Add community experience survey link to README.md by @databyjp in #5034
• Typo fixes; update ai studio error msg by @databyjp in #5037
• Add alternative to skip vulnerabilities in trivy by @jfrancoa in #5179
• Correct reference to text2vec-contextionary in text2vec-transformers by @jfrancoa in #5201
• Fixed references to Haystack and LangChain documentation in README by @kumarlokesh in #4974
• Remove commented out method in repos/sorter package by @kumarlokesh in #5011
• Add missing flags for dev server by @moogacs in #5278
• Fix license compliance issues by @parkerduckworth in #5372
• Upgrade s5cmd to latest release by @parkerduckworth in #5373
• Fix go.mod dependencies by @antas-marcin in #5400
• Adjust pull_requests.yaml to allow quick docker push on prefix 'build' branches by @moogacs in #5016
• Add more parameters to configure sentry by @reyreaud-l in #5424
• Fix ci push docker typo by @reyreaud-l in #5430

New Contributors

• @aminst made their first contribution in #4806
• @mindaugasrukas made their first contribution in #5171
• @kumarlokesh made their first contribution in #4974

Full Changelog: v1.25.8...v1.26.0