Skip to content

v1.29.0 - RBAC, Async Replication, ACORN improvements, ColBERT, Blazing Fast BM25

Choose a tag to compare

View all tags
@antas-marcin antas-marcin released this 17 Feb 21:14
· 901 commits to stable/v1.29 since this release
v1.29.0
35d800d

Breaking Changes

None

New Features

RBAC

The wait is over—RBAC is now GA! We've refined the user experience, battle-tested security, and made sure managing permissions in Weaviate is as seamless as possible. Whether you're setting up fine-grained access control or keeping it simple, RBAC has you covered. For the smoothest experience, we highly recommend using one of the Weaviate clients. They make configuring and managing permissions a breeze. Try it out, let us know what you think, and keep building amazing things with Weaviate!

  • RBAC: Remove need for metadata access for GRPC query by @dirkkul in #7278
  • RBAC: Relax permission requirements for batching by @dirkkul in #7262
  • Add more verbosity to rbac error messages through wrapping by @tsmith023 in #7260
  • fix(RBAC): add custom matcher to handle collection vs tenants checks by @moogacs in #7265
  • RBAC: Remove need for read config for data methods by @dirkkul in #7231
  • feat(rbac): filter based authz for READ ALL (schema, tenants, roles, getObjects) by @moogacs in #7152
  • Rbac: Correct casing for authz check for update by @dirkkul in #7224
  • feat(rbac): add best effort authorization by @moogacs in #7196
  • RBAC: Flatten manage_roles into CUD by @dirkkul in #7161
  • Add tests for rbac endpoinst with adminlist enabled by @dirkkul in #7151
  • fix(rbac): objects head/get permissions by @moogacs in #7154
  • RBAC: Remove unneeded read collection checks by @dirkkul in #7155
  • RBAC: Skip authz when getting own roles by @dirkkul in #7150
  • feat(rbac): introduce silent Authorize for internal checks without audit logging by @moogacs in #7141
  • rbac(config): prefix readonly group with experimental by @moogacs in #7143
  • flag(rbac): EXPERIMENTAL_AUTHORIZATION_RBAC_READONLY_ROOT_GROUPS by @moogacs in #7142
  • chore(rbac): fix authz acceptance test expectations by @moogacs in #7134
  • refact(rbac): autoSchema and autoTenant error msg by @moogacs in #7133
  • refact(rbac): rename rbac config to be consistent with weaviate configs by @moogacs in #7132
  • refact(rbac): typo builtin to built-in in error msg by @moogacs in #7131
  • refact(rbac): assign & revoke 404 msg by @moogacs in #7130
  • RBAC: add user permissions by @dirkkul in #7118
  • Rename rbacconfig ViewerRootGroups yaml struct tag by @parkerduckworth in #7101
  • feat(rbac): add new EXPERIMENTAL_AUTHORIZATION_READONLY_GROUPS env and fix roots responses to forbidden by @moogacs in #7099
  • feat(rbac): raft migrate roles scope to MATCH as default by @moogacs in #7093
  • RBAC: Add immutable root groups by @dirkkul in #7086
  • feat(rbac): scope based actions for roles permissions by @moogacs in #7079
  • Remove RBAC viewer env var by @dirkkul in #7089
  • RBAC: Add oidc mock acceptance tests by @dirkkul in #7080
  • RBAC: Internal admins are called root and endusers cannot touch them by @dirkkul in #7077
  • feat(rbac): assign and revoke groups by @moogacs in #7070
  • feat(rbac): add groups assignment endpoints by @moogacs in #7067
  • RBAC: Reduce authorizer calls for reference endpoints by @dirkkul in #6991
  • Communicate RBAC authz issue due to GQL introspection in error message by @tsmith023 in #7014
  • Rbac: less calls to authorizer for object endpoint by @dirkkul in #6980
  • RBAC: Reduce amount of authorizer calls for reference batch by @dirkkul in #6976
  • RBAC: Add authZ call count from logs for each method by @dirkkul in #6963
  • Add RBAC gating to queries that invoke auto tenant operations by @tsmith023 in #6799
  • ✍🏼 docs: RBAC auto generation for authorization calls by @moogacs in #6918
  • RBAC: Require collection and not tenant permissions for object+ref endpoints by @dirkkul in #6913
  • RBAC: Remove path validation for printing of paths by @dirkkul in #6887
  • chore(rbac): replace batch enforcer todo with documentation by @moogacs in #6876
  • fix: handle RBAC casbin init for RAFT in case of disable after enable by @moogacs in #6864
  • Anon rbac disallow by @dirkkul in #6862
  • feat: RBAC permission body validation by @moogacs in #6850
  • RBAC: unify env var naming by @dirkkul in #6851
  • feat: rbac seperate tenant and collections inside schema by @moogacs in #6800
  • fix: oidc config check in rbac user existance by @moogacs in #6849
  • Fix RBAC with OIDC and API keys enabled by @dirkkul in #6635
  • feat: rbac sort getRoles, getUsers endpoint response by name by @moogacs in #6815
  • Gate assigning/revoking roles to/from groups behind root users only by @tsmith023 in #7195
  • gRPC Aggregate: fix flaky gRPC group by test by @antas-marcin in #7084
  • gRPC Aggregate: add support for search by @antas-marcin in #7068
  • gRPC Aggregate: add support for property aggregators by @antas-marcin in #7042
  • gRPC Aggregate: add support for meta count queries by @antas-marcin in #6957

Async Replication

Async Replication is here! Proven at scale, it efficiently propagates any missing objects across nodes in your cluster. Each node builds a Merkle Tree, using its root hash to compare dataset views with other nodes. This makes detecting inconsistencies incredibly fast and lightweight. It’s highly configurable, and while we’ve introduced it with sensible defaults, there’s plenty of room to fine-tune it to your needs. Check out the docs for all the ways you can customize Async Replication!

  • fix: multiObjectByID may return nil objects if they are deleted by @jeroiraz in #7266
  • fix: skip corrupted hashtree upon initialization by @jeroiraz in #7251
  • chore: include enqueued segments in normal operations by @jeroiraz in #7218
  • chore: change hashtree path to hashtree_uuid by @jeroiraz in #7241
  • chore: log with specified frequency during hashtree initialization by @jeroiraz in #7236
  • chore: gracefully handle repair conflict when target node was already… by @jeroiraz in #7230
  • chore: concurrent object propagation by @jeroiraz in #7225
  • chore: do not include last read digest in next iteration by @jeroiraz in #7220
  • feat: avoid propagation of too recent objects by async replication by @jeroiraz in #7219
  • fix: include object vectors during obj propagation by @jeroiraz in #7201
  • Async replication: backward data compatibility with uuid-based hash exchange by @jeroiraz in #7128
  • chore: optimize alive host comparison by @jeroiraz in #7111
  • chore: simplified hashbeater by @jeroiraz in #7083
  • chore: log how long it took to initialize the hashtree by @jeroiraz in #7074
  • chore: faster hashtree initialization by @jeroiraz in #7061
  • fix: shard drop race with lazy shard initialization by @jeroiraz in #6922
  • test: replace sleep with eventuallyWithT by @jeroiraz in #6948
  • fix: async replication disabled during init by @jeroiraz in #6878
  • test: fix async rep acceptance test EventuallyWithT context by @jeroiraz in #6894
  • fix: only persist hashtree once fully in sync with persisted objects by @jeroiraz in #6877
  • fix: continue with async replication during backup by @jeroiraz in #6820
  • chore: skip data inserted before async replication by @jeroiraz in #6765
  • fix: release lock upon method finalization by @jeroiraz in #6783
  • fix: async noload shard by @jeroiraz in #6673
  • fix: close hashtree for tenant offloading by @jeroiraz in #6644
  • test: increase deadline for obj propagation by @jeroiraz in #6640
  • test: add start/stop container retry by @jeroiraz in #6649
  • test: use assert inside EventuallyWithT by @jeroiraz in #6638
  • fix(freeze): tenant freeze stops async replication if enabled by @jeroiraz in #6611

ACORN: Smarter Random Re-Entry

ACORN just got an upgrade! Our new random re-entry strategy improves how Weaviate handles updates and deletions, making indexing more efficient and reducing query latency. Instead of blindly re-inserting vectors, ACORN now intelligently decides when and where to place them for optimal search performance. It’s automatic, fast, and ensures your vector index stays in top shape. No tuning needed—just enjoy the speed boost!

  • ACORN: Apply random re-entries to improve positively correlated query performance by @abdelr in #7026
  • Add extra environment variables for ACORN filter strategy by @trengrj in #7256

Multi-Vector (ColBERT) Support (Preview)

Weaviate now supports multi-vector retrieval using ColBERT! Instead of a single vector per object, you can now store and search with multiple vectors per document, unlocking more precise and context-aware retrieval. This is a game-changer for handling long-form text and complex queries. Seamless, powerful, and ready to use—check out the docs to start leveraging multi-vector search today! This is currently a preview release, with stable support coming soon!

BockMax WAND-Based BM25 (Preview)

BM25 in Weaviate just got a serious speed boost! With BockMax WAND, we’ve optimized BM25 retrieval to be even more efficient, skipping unnecessary calculations while still delivering top-notch relevance. This means faster searches without compromising result quality. This preview release is only availble for new data created with this version and newer, but our full production-ready migration path for older data will ready soon!

  • fix(bm25_block): 🐛 Fix segment group strategy for Inverted and MapCollection by @amourao in #7249
  • Perf - Preload and better tombstone merge for inverted segments by @amourao in #7157
  • Perf - Load only objects from ids after merge by @amourao in #7158
  • Refactor: clean up Inverted/BlockMax and map/WAND interactions by @amourao in #7092
  • Faster BlockMax WAND by using SegmentBlockMax directly by @amourao in #7023
  • Propagate the varint encoders and fix buffer reuse by @amourao in #7024
  • refactor(bm25_block): ♻️ Make Bitmap public in allowlist by @amourao in #7022
  • feat/Store average instead of sum for proplen by @amourao in #7040
  • fix(bm25): 🐛 Fix batch index to increment with each batch properly by @amourao in #7012

Modules

Performance & Observability Improvements

  • [storobj] Clean up pointers from map parameters by @faustuzas in #7169
  • [storobj] Remove unnecessary complexity from object unmarshaling by @faustuzas in #7170
  • [aggregator] Unmarshal only required property in non-filtered group by by @faustuzas in #7194
  • performance: bitmaps merged concurrently by @aliszka in #6801
  • performance: buffers pool for cloned NotEqual bitmaps by @aliszka in #6681
  • performance: sort bitmaps before merge to reduce memory allocations by @aliszka in #6552

Fixes

Testing Improvements

Docs & Chores

Security Updates

  • dependencies(update): bump github.com/prometheus/common from 0.61.0 to 0.62.0 by @dependabot[bot] in #7044
  • dependencies(update): bump github.com/minio/minio-go/v7 from 7.0.83 to 7.0.84 by @dependabot[bot] in #7046
  • dependencies(update): bump github.com/aws/aws-sdk-go-v2/config from 1.29.0 to 1.29.2 by @dependabot[bot] in #7048
  • SEC(update): Bump google.golang.org/protobuf from 1.36.1 to 1.36.3 by @dependabot[bot] in #6927
  • SEC(update): Bump github.com/aws/aws-sdk-go-v2/service/bedrockruntime from 1.23.1 to 1.23.3 by @dependabot[bot] in #6924
  • SEC(update): Bump github.com/aws/aws-sdk-go-v2/config from 1.28.10 to 1.29.0 by @dependabot[bot] in #6923
  • SEC(update): Bump github.com/ikawaha/kagome/v2 from 2.9.11 to 2.10.0 by @dependabot[bot] in #6925
  • SEC(update): Bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.52 to 1.17.53 by @dependabot[bot] in #6926
  • SEC(update): Bump github.com/hashicorp/raft-boltdb/v2 from 2.3.0 to 2.3.1 by @dependabot[bot] in #6908
  • SEC(update): Bump google.golang.org/grpc from 1.69.2 to 1.69.4 by @dependabot[bot] in #6906
  • SEC(update): Bump github.com/coreos/go-oidc/v3 from 3.11.0 to 3.12.0 by @dependabot[bot] in #6910
  • SEC(update): Bump cloud.google.com/go/storage from 1.43.0 to 1.50.0 by @dependabot[bot] in #6907
  • SEC(update): Bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.51 to 1.17.52 by @dependabot[bot] in #6909
  • SEC(update): Bump github.com/minio/minio-go/v7 from 7.0.82 to 7.0.83 by @dependabot[bot] in #6858
  • SEC(update): Bump google.golang.org/api from 0.215.0 to 0.216.0 by @dependabot[bot] in #6883
  • SEC(update): Bump github.com/aws/aws-sdk-go-v2/config from 1.28.7 to 1.28.10 by @dependabot[bot] in #6884
  • SEC(update): Bump github.com/hashicorp/memberlist from 0.5.1 to 0.5.2 by @dependabot[bot] in #6882
  • SEC(update): Bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.16.0 to 1.17.0 by @dependabot[bot] in #6885
  • SEC(update): Bump github.com/bmatcuk/doublestar from 1.1.3 to 1.3.4 by @dependabot[bot] in #6841
  • SEC(update): Bump github.com/prometheus/common from 0.60.0 to 0.61.0 by @dependabot[bot] in #6843
  • SEC(update): Bump google.golang.org/api from 0.214.0 to 0.215.0 by @dependabot[bot] in #6844
  • SEC(update): Bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.4.1 to 1.5.0 by @dependabot[bot] in #6845
  • SEC(update): Bump github.com/casbin/casbin/v2 from 2.102.0 to 2.103.0 by @dependabot[bot] in #6791
  • SEC(update): Bump github.com/edsrzf/mmap-go from 1.1.0 to 1.2.0 by @dependabot[bot] in #6792
  • SEC(update): Bump github.com/prometheus/client_golang from 1.20.4 to 1.20.5 by @dependabot[bot] in #6793
  • SEC(update): Bump google.golang.org/grpc from 1.67.1 to 1.69.2 by @dependabot[bot] in #6794
  • SEC(update): Bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.34 to 1.17.48 by @dependabot[bot] in #6795
  • sec(update): Bump github.com/tailor-inc/graphql from 0.4.1 to 0.5.7 i… by @moogacs in #6733
  • SEC(update): Bump github.com/go-openapi/strfmt from 0.21.3 to 0.23.0 in /test/benchmark_bm25 by @dependabot[bot] in #6712
  • SEC(update): Bump github.com/go-openapi/strfmt from 0.21.3 to 0.23.0 in /test/acceptance_with_go_client by @dependabot[bot] in #6717
  • SEC(update): Bump github.com/weaviate/weaviate-go-client/v4 from 4.13.1 to 4.16.1 in /test/acceptance_with_go_client by @dependabot[bot] in #6718
  • SEC(update): Bump github.com/spf13/cobra from 1.7.0 to 1.8.1 in /test/benchmark_bm25 by @dependabot[bot] in #6713
  • SEC(update): Bump github.com/go-openapi/errors from 0.20.3 to 0.22.0 by @dependabot[bot] in #6716
  • SEC(update): Bump github.com/go-openapi/strfmt from 0.21.3 to 0.23.0 by @dependabot[bot] in #6720
  • SEC(update): Bump github.com/KimMachineGun/automemlimit from 0.3.0 to 0.6.1 by @dependabot[bot] in #6721
  • SEC(update): Bump github.com/aws/aws-sdk-go-v2/service/bedrockruntime from 1.17.0 to 1.23.1 by @dependabot[bot] in #6726
  • SEC: upgrade golang.org/x/crypto for CVE-2024-45337 by @moogacs in #6609

New Contributors

Full Changelog: v1.28.5...1.29.0

Contributors

aliszka, jeroiraz, and 19 other contributors
Assets 17
Loading