Release 4.0.3

What's Changed

• Remove sdk documentation logic by @lferran in #2205

Full Changelog: v4.0.2...v4.0.3

Release 4.0.2

What's Changed

• Compatible with python 3.9+ by @javitonino in #2195
• Supports Python 3.12 by @javitonino in #2176
• Fix test by @javitonino in #2197
• Add vectorset datamanager by @jotare in #2196
• SimSimd with aligned vectors by @javitonino in #2179
• Improve storage interface by @lferran in #2198
• Overwrite vectorsets key, create vectorsets on shard creation and make new kbs have this key by @jotare in #2201
• Include ask endpoint in open api schema by @lferran in #2204
• Ingest consumer retry maindb server errors by @lferran in #2203
• Add ingest gRPC methods to create and delete vectorsets by @jotare in #2202

Full Changelog: v4.0.1...v4.0.2

Release 4.0.1

What's Changed

• Switch to argdantic by @javitonino in #2163
• Single app tag for all node replicas by @javitonino in #2160
• Change app label in node replicas deployment by @javitonino in #2164
• Add date range filtering for catalog by @lferran in #2161
• Add PyNacl dependency (from nkeys which forgot it) by @javitonino in #2166
• K8s selector by @javitonino in #2167
• Remove broker message from pubsub notifications by @lferran in #2165
• Split shard & vector configuration by @javitonino in #2155
• Handle rebalance job being locked by @lferran in #2168
• Add vectorsets support to bindings by @jotare in #2162
• Fix vectorests bindings after merge without rebase by @jotare in #2171
• Fix requests version for docker by @javitonino in #2173
• Fix processing storage driver enum usage by @lferran in #2174
• Synonyms datamanager by @jotare in #2169
• Include necessary files in sdist by @javitonino in #2177
• Replicate vectorsets by @javitonino in #2172
• Restore node services by @javitonino in #2180
• Reprocess external links as well by @javitonino in #2183
• Reprocess external link by @javitonino in #2184
• Clean some ingest dead code by @jotare in #2181
• Extend BrokerMessage with vectorsets by @jotare in #2158
• Sc 9354/idp regional merge nucliadb predict and processing by @drf7 in #2145
• Replicate shards using descriptor by @javitonino in #2185
• Less logs noise by @javitonino in #2186
• Resources orm datamanager cleanup by @jotare in #2182
• Atomic datamanagers by @jotare in #2187
• change kb-usage.audit to kb-usage.nuclia_db by @drf7 in #2189
• Stream file even if it has been deleted by @javitonino in #2190
• Update docker.py by @javitonino in #2191
• Lint protos by @javitonino in #2192
• Update deps (audit) by @javitonino in #2193
• Fix s3 storage driver: missing move method by @lferran in #2194

Full Changelog: v4.0.0...v4.0.1

Release 4.0.0

What's Changed

• back pressure less warnings by @lferran in #2159
• Update to Pydantic 2 by @javitonino in #2142

Full Changelog: v3.2.0...v4.0.0

Release 3.2.0

What's Changed

• Pdm by @javitonino in #2136
• Cleanup index nodes from global state on teardown by @lferran in #2137
• Fix object store s3 building by @lferran in #2139
• Increase maximum message size for index writer grpc service by @javitonino in #2140
• Migration to drain nodes from the cluster by @lferran in #2138
• Drain nodes fix by @lferran in #2141
• Add retries to object store downloads by @lferran in #2143
• Support vectorsets at shard level by @jotare in #2129
• Ask endpoint by @lferran in #2131
• update object_store for better retries by @javitonino in #2146
• message size grpc client by @javitonino in #2147
• Only update changed indexes by @javitonino in #2148
• Merge to prune delete tree when too many deletions by @javitonino in #2151
• Fix local storage file movement by @carlesonielfa in #2153
• Run one migration at a time for onprem by @lferran in #2154
• Add a vectors index resource wrapper to set resources in the proper vectorset by @jotare in #2152
• Tweak logs by @lferran in #2156
• Add empydir sizelimit by @rastut in #2157
• Tweak ask response parsing by @lferran in #2150

New Contributors

• @carlesonielfa made their first contribution in #2153

Full Changelog: v3.1.0...v3.2.0

Release 3.1.0

What's Changed

• Hnsw algorithm by @javitonino in #2066
• Max tokens for LLM by @lferran in #2065
• Do not deserialize empty nodes into RAM hnsw by @javitonino in #2070
• Query shard replicas as primary-secondary by @lferran in #2071
• Add global migration to purge orphan kbslugs keys from maindb by @jotare in #1963
• Support matryoshka embeddings by @jotare in #2063
• Adding AI tables storage by @bloodbare in #2018
• int order retuns only needed results by @hermeGarcia in #2076
• Bump aiohttp and idna to fix vulnerabilities by @lferran in #2078
• Term query set by @javitonino in #2073
• Reuse field information when indexing paragraphs by @javitonino in #2081
• Bump rustls from 0.21.10 to 0.21.11 by @dependabot in #2082
• Fix pagination bug on catalog by @lferran in #2084
• Add origin source-id label alias by @lferran in #2085
• Support keep_markdown in models by @javitonino in #2087
• Validate field ids against regex with fastapi / pydantic by @lferran in #2090
• Migrate to paragraphs v3 by @javitonino in #2089
• Completely remove vectorsets public api by @lferran in #2092
• Do not fail if apt-get update fails in CI by @javitonino in #2093
• Set to pending on field upload for existing resource by @lferran in #2095
• No vector metadata when opening by @hermeGarcia in #2091
• old version names by @hermeGarcia in #2096
• use rename by @hermeGarcia in #2097
• Cleanup ingest gRPC and id/slugs by @jotare in #2080
• Open index readers only need path, not config by @jotare in #2099
• Cleanup optionals from metadata file by @jotare in #2100
• Revert "Bigger k during layer>0 search" by @javitonino in #2101
• Fix identation on rollover by @lferran in #2104
• Ship pingcap cert by @javitonino in #2105
• Added limits to pagination by @lferran in #2108
• Fix fedora e2e by @lferran in #2109
• Be able to index a resource by storage referece by @lferran in #2094
• Set resource from storage tweaks by @lferran in #2111
• Faster HNSW insertion by @javitonino in #2102
• alias not rename by @hermeGarcia in #2115
• Add shard indexes metadata file by @jotare in #2103
• Deprecated unused OpStatus fields and don't compute counters anymore by @jotare in #2114
• Create indexes passsing its config by value by @jotare in #2116
• Less locking in ShardWriter by @jotare in #2117
• Store indexes file by write and rename by @jotare in #2120
• box readers + swap update by @hermeGarcia in #2122
• Concurrency by branch by @javitonino in #2126
• Remove previous ask resource feature by @lferran in #2124
• Stricter grpcio dependency requirements to ease installs by @lferran in #2123
• Add generated files on links by @bloodbare in #2125
• Fix S3 storage delete_object param name by @jotare in #2112
• Get semantic threshold from query endpoint by @lferran in #2127
• Limit the amount of vectorsets allowed by @jotare in #2128
• Update rust deps by @javitonino in #2130
• Onprem migrations on startup by @lferran in #2132
• Do not wait for nodes on onprem deploy by @lferran in #2134
• Set normalize_vectors as optional by @jotare in #2133
• Fix wrong usage of fastapi endpoint docs by @jotare in #2113
• Properly close asyncio loop by @lferran in #2135
• Add test for rollover resilient to modifications and deletions by @lferran in #2106
• prefer_markdown option in chat by @vangheem in #2098

Full Changelog: v3.0.3...v3.1.0

Release 3.0.3

What's Changed

• reader should have a shards dir by @hermeGarcia in #2060
• Simplify kb and shard creation by @jotare in #2052
• madvise by @javitonino in #2050
• Validate that export is compatible when importing by @lferran in #2061
• Bigger k during layer>0 search by @javitonino in #2064
• Allow chat on resource by slug by @lferran in #2067

Full Changelog: v3.0.2...v3.0.3

Release 3.0.2

v3.0.2

chore: autopublish 2024-04-11T16:10:47Z

Release 3.0.1

v3.0.1

chore: autopublish 2024-04-11T13:48:33Z

Release 3.0.0

v3.0.0

Remove previous Vectorsets implementation (#2047)