diff --git a/modules/search/pages/child-field-options-reference.adoc b/modules/search/pages/child-field-options-reference.adoc index c763baf04..c69b168bc 100644 --- a/modules/search/pages/child-field-options-reference.adoc +++ b/modules/search/pages/child-field-options-reference.adoc @@ -23,7 +23,7 @@ include::partial$vector-search-field-descriptions.adoc[tag=dimension] include::partial$vector-search-field-descriptions.adoc[tag=similarity_metric] -|Optimized For (Vector Fields Only) a| +|[[optimized]]Optimized For (Vector Fields Only) a| include::partial$vector-search-field-descriptions.adoc[tag=optimized_for] diff --git a/modules/vector-search/pages/vector-search-index-architecture.adoc b/modules/vector-search/pages/vector-search-index-architecture.adoc new file mode 100644 index 000000000..c29e8a018 --- /dev/null +++ b/modules/vector-search/pages/vector-search-index-architecture.adoc @@ -0,0 +1,152 @@ += Vector Search Index Architecture +:page-topic-type: concept +:description: Vector Search indexes use features from traditional Search indexes, with unique indexing algorithms and features that allow you to compare vectors in nearest neighbor searches. +:page-toclevels: 3 + +[abstract] +{description} + +A Vector Search index still relies on <> and uses <> to manage merging and persisting data to disk in your cluster. +All changes from Database Change Protocol (DCP) and the Data Service are introduced to a Search index in batches, which are further managed by segments. + +[#sync] +== Synchronization with Database Change Protocol (DCP) and the Data Service + +The Search Service uses batches to process data that comes in from xref:server:learn:clusters-and-availability/intra-cluster-replication.adoc#database-change-protocol[DCP] and the xref:server:learn:services-and-indexes:services/data-service.adoc[Data Service]. +DCP and Data Service changes are introduced gradually, based on available memory on Search Service nodes, until reindexing operations for an index are complete. + +The Search Service can merge batches into a single batch before they're sent to the disk write queue, to reduce the resources required for batch processing. + +The Search Service maintains index snapshots on each Search index partition. +These snapshots contain a representation of document mutations on either a write queue, or in storage. + +If the Search Service loses connection to the Data Service, the Search Service compares its rollback sequence numbers in its snapshots with the Data Service when the connection is reestablished. +If the index snapshots on the Search Service are too far ahead, the Search Service performs a full rollback to get back in sync with the Data Service. + +[#segments] +== Search Index Segments + +Search and Vector Search indexes in Couchbase Server are built with segments. + +All Search indexes contain a root segment, which includes all data for the Search index but excludes any segments that might be stale. +Stale segments are eventually removed by the Search Services's persister or merger routines. + +The persister reads in-memory segments from the disk write queue and flushes them to disk, completing batch operations as part of <>. +The merger works with the persister to consolidate flushed files and flush the consolidated results back through the persister - while purging the smaller, older files. + +The persister and merger interact to continuously flush and merge new in-memory segments to disk, and remove stale segments. + +Segments are marked as stale when they're replaced by a new merged segment created by the merger. +Stale segments are deleted when they're no longer used by any new queries. + +As smaller segments are merged together through the merger routine, the Search Service automatically runs any needed retraining for Vector Search indexes. +The segments for a Vector Search index can contain different index types and use a separate indexing pipeline, choosing the appropriate indexing algorithm based on the size of your available documents. + +== Vector Search and FAISS + +Vector Search specifically uses https://faiss.ai/index.html[FAISS^] indexes. +Any vectors inside your documents are indexed using FAISS, to create a new query vector that can be searched for similar vectors inside your Vector Search index. + +Vector Search chooses the best https://github.com/facebookresearch/faiss/wiki/Faiss-indexes[FAISS index class^], or vector search algorithm, for your data, and automatically tunes parameters to provide a balance of recall and latency. +You can choose to prioritize recall, latency, or memory efficiency with the xref:search:child-field-options-reference.adoc#optimized[Optimized For] setting on your index. +You can also choose to xref:fine-tune-vector-search.adoc[fine tune your Vector Search queries] to override the default balancing for your index, and change the number of centroids or probes searched in a query. + +The FAISS indexes created for your vector data can be: + +* <> +* <> + +The specific type of index used depends on the number of vectors in your dataset: + +|==== +| Vector Count | Index Types | Description + +| >=10,000 +| IVF with scalar quantization +a| Vectors are indexed with <> indexes and <>. + +If xref:search:child-field-options-reference.adoc#optimized[Optimized For] is set to *recall* or *latency*, Vector Search uses 8bit scalar quantization. +If set to *memory-efficient*, Vector Search uses 4bit scalar quantization. + +| >=1000 +| IVF with Flat +| Vectors are indexed with <> combined with <>. +Indexes do not use <>. + +| <1000 +| Flat +| Vectors are indexed with <>. +Indexes do not use <>. +|==== + +[#flat] +=== FLAT Indexes + +The most basic kind of index that Vector Search can use for your vectors is a flat index. + +Vector Search uses flat indexes for data that contains less than 1000 vectors. + +Flat indexes are a list of vectors. +Searches run on a nearest neighbor process, based on examining the query vector against each vector in the index and calculating the distance. +Results for flat indexes are very accurate, but performance does not scale well as a dataset grows. + +If a Vector Search index uses only flat indexes, no training is required - IDs are mapped directly to vectors with exact vector comparisons, with no need for preprocessing or learning on the data. + +[#ivf] +=== Inverted File Index (IVF) + +For reduced latency, Vector Search can also use Inverted File Indexes (IVF). + +Vector Search uses a combination of IVF and flat indexes for data that contains between 1000 and 9999 vectors. +For even larger datasets, Vector Search uses IVF indexes with <>. + +IVF creates partitions called Voronoi cells in an index. +The total number of cells is the *nlist* parameter. + +Every cell has a centroid. +Every vector in the processed dataset is assigned to a cell that corresponds to its nearest centroid. + +In an IVF index, Vector Search first tries to find a centroid vector closest to the query vector. +After finding this closest centroid vector, Vector Search uses the default `nprobe` and `max_codes` values to search over adjoining cells to the closest centroid and finds the top `k` number of vectors. + +IVF index searches are not exhaustive searches. +You can increase accuracy by changing the `max_nprobe_pct` parameter or `max_codes_pct` when you xref:fine-tune-vector-search.adoc[fine tune your Vector Search queries]. + +The Search Service automatically trains larger IVF indexes to learn the data distribution of your vectors, and the centroids of cells in your dataset. +The training data helps to encode and compress the vectors in your index with <>. +All training occurs during building and merging <>. + +IVF indexes that also use flat indexing automatically train to determine the centroids of cells, but still use exact vector comparisons within each cell. +Training still occurs while building and merging <>. + +[#scalar-quant] +==== Scalar Quantization + +Vector Search uses scalar quantization on large datasets to reduce the size of your indexes. + +Scalar quantization is an important data compression technique that turns the floating point values that could be present in a large vector into low-dimensional integers. +For example, a float32 value could be reduced to an int8 value. + +Scalar quantization in Vector Search does not have a significant effect on the recall, or accuracy, of query results on large datasets. + +Vector Search uses both 8bit and 4bit scalar quantization for indexes, based on your xref:search:child-field-options-reference.adoc#optimized[Optimized For] setting. + +== Search Request Processing + +The Search Service uses a scatter-gather process for running all Search queries, when there are multiple nodes in the cluster running the Search Service. + +The Search Service node that receives the Search request is assigned as the coordinating node. +Using https://grpc.io/[gRPC^], the coordinating node scatters the request to all other partitions for the Search or Vector Search index in the request across other nodes. +The coordinating node applies filters to the results received from the other partitions, and returns the final result set. + +Results are scored, and based on the xref:search:search-request-params.adoc#sort[Sort Object] provided in the Search request, returned in a list. + +For a Vector Search query, search results include the top `k` nearest neighbor vectors to the vector in the Search query. +For more information about how results are scored and returned for Search requests, see xref:search:run-searches.adoc#scoring[Scoring for Search Queries]. + +== See Also + +* xref:fine-tune-vector-search.adoc[] +* xref:search:search-request-params.adoc[] +* xref:create-vector-search-index-rest-api.adoc[] +* xref:create-vector-search-index-ui.adoc[] \ No newline at end of file diff --git a/modules/vector-search/partials/nav.adoc b/modules/vector-search/partials/nav.adoc index cbfb273a5..8ebb2659e 100644 --- a/modules/vector-search/partials/nav.adoc +++ b/modules/vector-search/partials/nav.adoc @@ -1,6 +1,7 @@ * xref:7.6@server:vector-search:vector-search.adoc[] ** xref:7.6@server:vector-search:create-vector-search-index-ui.adoc[] ** xref:7.6@server:vector-search:create-vector-search-index-rest-api.adoc[] +** xref:7.6@server:vector-search:vector-search-index-architecture.adoc[] ** xref:7.6@server:vector-search:pre-filtering-vector-search.adoc[] ** xref:7.6@server:vector-search:run-vector-search-ui.adoc[] ** xref:7.6@server:vector-search:run-vector-search-rest-api.adoc[]