v1.21.0 - Contains Operators, pread, backup compression, Import Performance, Inactive Tenants, and much more!

v1.21.0 - Contains Operators, pread, backup compression, Import Performance, Inactive Tenants, and much more!

Breaking Changes

none

New Features

ContainsAny / ContainsAll operators

New filter operators that allow for easier filters with array types or matching one of (or all of) words in tokenized text.

• ContainsAny & ContainsAll Filter Operators by @antas-marcin in #3370

Backup Compression

Backups can now be compressed into pre-configureable chunks and require much fewer file operations. This reduces costs on S3/GCS/etc and improves the speed of backups:

• Enhanced backup efficiency and configurability with compression techniques by @redouan-rhazouani in #3327

Add ability to deactive tenants (experimental)

An inactive tenant does not consume any resources. This allows for hosting more tenants on the same node as long as not all tenants are active at the same time.

• Activated deactivated tenants by @aliszka in #3353
• Activate / deactivate tenant improvements by @aliszka in #3378

Support Pread (as alternative to Mmap) in LSM store

Improves performance and stability on disk-bound setups. Opt-in for now, will become opt-out in a future relase:

• Add option to use pread instead of mmap for LSM store access by @parkerduckworth in #3254

Performance Improvements

• hnsw: add neon simd support to l2 and dot product by @asdine in #3215
• Single cyclemanager of type per class. Shard's cycle callbacks grouped in containers by @aliszka in #3306
• Cycle manager per class p2 by @aliszka in #3326
• Missing callback unregister by @aliszka in #3359
• Reduce indexing time due to lock contention by @abdelr in #3186

New Modules

• Introduce text2vec-gpt4all module by @antas-marcin in #3361
• Introduce multi2vec-bind module by @antas-marcin in #3292

Other

• Batch support in reranker transformers by @antas-marcin in #3279
• Enforce minimum replication factor according to system-wide config by @etiennedi in #3352
• Configurable nested cross reference query limit by @parkerduckworth in #3281
• Add batch queue congestion info to node status by @dirkkul in #3372
• Add missing features to GRPC by @dirkkul in #3323
• Add grpc batching by @dirkkul in #3400
• Enable references without ToClass in object creation by @dirkkul in #3365

Fixes

• Ensure proper vector sync to target nodes during read repair by @redouan-rhazouani in #3280
• fix langchain doc link by @zwpaper in #3257
• Fix missing tenant by @donomii in #3331
• Fix optional unmarshaling of object when dealing with large vector dimensions (up to 65k) by @antas-marcin in #3354
• Fixing bug detected by the chaos pipeline by @abdelr in #3375
• close the indexCounter file before removal by @tsmith023 in #3387
• Catch panics during hnsw delete cleanup by @parkerduckworth in #3389
• pq: implement cache miss logic by @asdine in #3335
• Catch panics during hnsw delete cleanup by @parkerduckworth in #3390
• Fix get requests with classname and include by @dirkkul in #3394
• Prevent deadlock from malformed tx payload in cluster api by @etiennedi in #3402

Internal / Refactoring / Dev Tooling / Other

• Update backup related go dependencies to newest versions by @antas-marcin in #3259
• Upgrade docker client dependency to enable successful tests using go1.20.6 and testcontainers by @antas-marcin in #3299
• Add support for compilation on windows by @dirkkul in #3295
• Update github actions to be able to run with latest go version by @antas-marcin in #3300
• Bump Templum/govulncheck-action from 0.10.1 to 1.0.0 by @dependabot in #3287
• Update testcontainers lib to solve bug with go1.20.6 by @antas-marcin in #3350
• Remove Not operator from where filter by @dirkkul in #3360
• Add class to proppair by @donomii in #3304

New Contributors

• @zwpaper made their first contribution in #3257
• @tsmith023 made their first contribution in #3387

Full Changelog: v1.20.5...v1.21.0