v1.38.3 - Async replication stability Fixes, BM25 search performance Fixes, Backup, HFresh and Queue improvement Fixes

Breaking Changes

none

New Features

none

Fixes

• fix(backup-azure): use delimiter listing in AllBackups to avoid full object scan by @gkampitakis in #11669
• perf(bm25): defer tombstone checks from advance-time to scoring by @amourao in #11774
• perf(bm25): tighten block-entry scanning in the WAND loop by @amourao in #11775
• perf(bm25): round-6 query-setup path optimization by @amourao in #11776
• perf(bm25): round-7 matched-branch sort elision by @amourao in #11777
• perf(lsmkv): lock-free atomic segment refcount by @amourao in #11771
• Ensure that rf=1 writes go through replicator if active op for the shard by @tsmith023 in #11867
• perf: dense/sorted-pairs property-length representation by @amourao in #11753
• perf: direct-array compaction serialization via EncodePairs by @amourao in #11754
• Fix EC in namespace tests by @dirkkul in #11871
• fix: persist advanced update time when object content is unchanged by @jeroiraz in #11866
• Namespaces: Match colon-containing user/group ids literally for global callers by @dirkkul in #11868
• Return proper error code from shard endpoints when index is not found by @dirkkul in #11875
• Fix more error codes in REST handler by @dirkkul in #11878
• fix: TestIndex_UsageForCollection_MissingShardFiles data race error by @antas-marcin in #11887
• Fix node handler return types by @dirkkul in #11888
• test(replication): wait for shard readiness after restart in TestReadRepairDeleteOnConflict by @jeroiraz in #11886
• [feat] Use hard-links during replica movement to avoid compaction halting by @tsmith023 in #11226
• Delete rfc.md for replica movement with hard-link snapshots by @tsmith023 in #11891
• Fix partial permission removal from role by @dirkkul in #11885
• Fix potential deadlock in backup in big clusters by @dirkkul in #11890
• feat(async-rep): propagate raw on-disk object bytes by @jeroiraz in #11853
• Avoid pausing compaction in ApplyToObjectDigests by @jeroiraz in #11896
• Fix multi-role delete by @dirkkul in #11904
• fix(hnsw): guard insert-path compression-state reads under compressActionLock by @amourao in #11597
• Add gate to disallow global non-operator users by @dirkkul in #11908
• Fix replica-movement restore crashloop and chained-move data loss by @tsmith023 in #11895
• perf(bm25): gate DoBlockMaxWand prune-branch repair on needFullSort by @amourao in #11915
• Fix/hnsw maintenance flag leak by @abdelr in #11898
• Deny operator only surface to namespaced users by @dirkkul in #11922
• gh-7461 use automaxprocs to set GOMAXPROCS (cgroup v2 support) [backport v1.36] by @Joe-Weaviate in #11918
• chore: fix flakiness of TestAuthzReplicationReplicate by @tsmith023 in #11926
• chore: add detailed error logging to TestReplicationTestSuite by @tsmith023 in #11927
• Fix/hnsw search entrypoint prereqs by @abdelr in #11923
• Return 500 instead of nil Responder on getGroups controller error by @dirkkul in #11933
• Fix/hnsw repair global entrypoint by @abdelr in #11910
• chore: add back EventuallyWithT to TestAuthzReplicationReplicate assertions by @tsmith023 in #11945
• Retry go modules downloads in CI by @dirkkul in #11942
• Remove .Error() from test assert to avoid panics in test by @tsmith023 in #11950
• hfresh: increase searchProbe default to 256 by @asdine in #11955
• chore(metric): fix metric name typo by @moogacs in #11953
• perf(replication): share one zstd encoder in REST client instead of sync.Pool by @jeroiraz in #11960
• perf(async-replication): allocate propagation buffers per-hashBeat, not per-range by @jeroiraz in #11961
• Revert "Fix/hnsw search entrypoint prereqs" by @antas-marcin in #11969
• Revert "Fix/hnsw repair global entrypoint" by @antas-marcin in #11971
• Optimize cycle manager for large MT collections by @dirkkul in #11958
• fix(backup): make halt-for-transfer inactivity monitor race-free by @jeroiraz in #11965
• Additional validation on PQ query vector length by @trengrj in #11954
• Incremental backups: Bugfix collection by @dirkkul in #11972
• hfresh: fix quantizer initialization race by @asdine in #12000
• Namespace local roles by @dirkkul in #11873
• Fix RBAC class-getter memo key mismatch in gRPC service by @dirkkul in #12001
• Resolve class only once on GRPC layer by @dirkkul in #12003
• Narrow traverser single-class reads to a class getter by @dirkkul in #12005
• [Reindex Fixes A] fix: idempotent recovery-swap marking after mid-FINALIZING restart by @etiennedi in #11976
• [Reindex Fixes A] test: harden multinode reindex tests against weak-VM timing by @etiennedi in #11538
• [Reindex Fixes A] test: tolerate transient /v1/tasks 500s after node restarts by @etiennedi in #11530
• [Reindex Fixes A] fix: class DELETE racing a draining reindex worker re-created the class dir by @etiennedi in #11539
• fix(queue): recover from corrupt chunk headers instead of failing to load by @asdine in #12006
• Add percentage eager shards loaded to raft waiting for database to restore log by @tsmith023 in #11952
• Add weaviate_ prefix to new metric that was missed by @tsmith023 in #12013
• test(ci): split go-client-group-2 into group-2 (usage) and group-3 by @antas-marcin in #11946
• fix(bm25): dedup duplicate properties (last boost wins) by @etiennedi in #12004
• fix(bm25): propagate combineResults error (was silent empty results) by @etiennedi in #12007
• fix(async-indexing): vector queue stays paused forever after a failed index upgrade by @etiennedi in #11983
• perf(usage): replace per-shard jitter with bounded concurrent shard readers by @antas-marcin in #11902
• fix(lsmkv): pass k1/b in correct order when flushing block-max impacts by @amourao in #12022
• fix(auth-broker): retry on 429 http status code by @gkampitakis in #12025
• perf(async-replication): batched hashtree-root pre-filter for many-tenant clusters by @jeroiraz in #11968
• fix(async-replication): hoist hashtree-init wait out of RLock in merge paths by @jeroiraz in #12027
• fix(async-replication): don't let TimeBased repair clobber a newer local write by @jeroiraz in #12028
• fix(replication): avoid nil-pointer deref in read-repair when a live winner's refetch fails by @jeroiraz in #12030
• fix(queue): salvage and quarantine torn sealed chunks by @asdine in #12012
• feat: implement runtime toggle for GraphQL API by @amourao in #11930
• Reuse consistent view for fetches of multiple objects by @dirkkul in #12010
• fix(queue): validate chunk record count and length against file size by @asdine in #12009
• fix(queue): back off on recoverable task errors, contain task and dispatch panics by @asdine in #12023
• perf(bm25): approximate object count for IDF, reuse query stats on blockmax fallback by @amourao in #12017
• fix(async-replication): serialize hashtree rebuild against shard shutdown by @jeroiraz in #12031
• fix(async-replication): deregister under the write lock and keep replication on while overrides are active by @jeroiraz in #12034
• fix(async-replication): close hashtree-init channel on early goroutine exit to unblock parked merges by @antas-marcin in #12037
• fix(bm25): apply propertyBoost to WAND pivot and block-impact bounds by @amourao in #12018
• fix(async-replication): repair REST root-prefilter JSON and bound request size by @jeroiraz in #12048
• Bound halt-for-transfer backup preparation by @asdine in #11947
• Fix geo queue backup preparation timeout by @asdine in #12055
• feat: introduce /grpc-web endpoint by @tsmith023 in #12024
• fix(replica): harden async-replication and replica read/write paths by @jeroiraz in #12044
• fix(lsmkv): guard nil logger in GetConsistentView slow-lock warning by @amourao in #12060
• Fix RBAC leak in node handler by @dirkkul in #12002
• perf(bm25): cursor tombstone and filter probes in block-max WAND by @amourao in #12046
• chore(wf): update slack notify template by @moogacs in #12064
• fix(hfresh): normalize vectors during reassign by @asdine in #12068
• Optimize shard gauge allocations by @dirkkul in #12067
• Wrap ErrNotFound in GetShardsQueueSize so a non-existing index returns 404, not 500 by @jfrancoa in #12040
• feat: namespace graduation using backup/restore by @tsmith023 in #11924
• perf(shard): lazy load empty multi-tenant shards at startup by @moogacs in #12061
• Avoid target node collisions when creating replication ops in test by @tsmith023 in #12074
• perf(usage): make shard concurrency a usage node wide constraint by @antas-marcin in #12071
• review(usage): guard semaphore capacity, drop FIFO claim, shorten test timeout by @antas-marcin in #12083
• fix(async-repl): respect parent index ctx on deletion by @moogacs in #12079

New Contributors

• @Joe-Weaviate made their first contribution in #11918

Full Changelog: v1.38.2...v1.38.3