diff --git a/docs/docs/examples/examples/docs_to_knowledge_graph.md b/docs/docs/examples/examples/docs_to_knowledge_graph.md index 80a4808f..b3c5648e 100644 --- a/docs/docs/examples/examples/docs_to_knowledge_graph.md +++ b/docs/docs/examples/examples/docs_to_knowledge_graph.md @@ -37,7 +37,7 @@ and then build a knowledge graph. ## Setup * [Install PostgreSQL](https://cocoindex.io/docs/getting_started/installation#-install-postgres). CocoIndex uses PostgreSQL internally for incremental processing. -* [Install Neo4j](https://cocoindex.io/docs/ops/targets#neo4j-dev-instance), a graph database. +* [Install Neo4j](https://cocoindex.io/docs/targets/neo4j#neo4j-dev-instance), a graph database. * [Configure your OpenAI API key](https://cocoindex.io/docs/ai/llm#openai). Alternatively, we have native support for Gemini, Ollama, LiteLLM. You can choose your favorite LLM provider and work completely on-premises. diff --git a/docs/docs/examples/examples/photo_search.md b/docs/docs/examples/examples/photo_search.md index b267a3fb..1474b6a2 100644 --- a/docs/docs/examples/examples/photo_search.md +++ b/docs/docs/examples/examples/photo_search.md @@ -186,7 +186,7 @@ face_embeddings.export( Now you can run cosine similarity queries over facial vectors. CocoIndex supports 1-line switch with other vector databases. - + ## Query the Index diff --git a/docs/docs/examples/examples/product_recommendation.md b/docs/docs/examples/examples/product_recommendation.md index 6f5e4a18..110201b2 100644 --- a/docs/docs/examples/examples/product_recommendation.md +++ b/docs/docs/examples/examples/product_recommendation.md @@ -39,7 +39,7 @@ Alternatively, we have native support for Gemini, Ollama, LiteLLM. You can choos ## Documentation - + ## Flow Overview diff --git a/docs/docs/examples/examples/simple_vector_index.md b/docs/docs/examples/examples/simple_vector_index.md index bb2fc580..e66dbc15 100644 --- a/docs/docs/examples/examples/simple_vector_index.md +++ b/docs/docs/examples/examples/simple_vector_index.md @@ -140,7 +140,7 @@ This decorator marks this as a reusable transformation flow that can be called o CocoIndex doesn't provide additional query interface at the moment. We can write SQL or rely on the query engine by the target storage, if any. - + ```python diff --git a/docs/docs/targets/index.md b/docs/docs/targets/index.md new file mode 100644 index 00000000..7915fe7f --- /dev/null +++ b/docs/docs/targets/index.md @@ -0,0 +1,338 @@ +--- +title: Targets +description: CocoIndex Built-in Targets +toc_max_heading_level: 4 +--- +import { ExampleButton } from '../../src/components/GitHubButton'; + +# CocoIndex Built-in Targets + +For each target, data are exported from a data collector, containing data of multiple entries, each with multiple fields. +The way to map data from a data collector to a target depends on data model of the target. + +## Targets Overview + +| Target Type | See Also | +|------------------|-------------------------| +| [Postgres](/docs/targets/postgres) | Relational Database, Vector Search (PGVector) | +| [Qdrant](/docs/targets/qdrant) | Vector Database, Keyword Search | +| [LanceDB](/docs/targets/lancedb) | Vector Database, Keyword Search | +| [Neo4j](/docs/targets/neo4j) | [Property graph](#property-graph-targets) | +| [Kuzu](/docs/targets/kuzu) | [Property graph](#property-graph-targets) | + + +## Property Graph Targets +Property graph is a widely-adopted model for knowledge graphs, where both nodes and relationships can have properties. + +[Graph database concepts](https://neo4j.com/docs/getting-started/appendix/graphdb-concepts/) has a good introduction to basic concepts of property graphs. + +The following concepts will be used in the following sections: +* [Node](https://neo4j.com/docs/getting-started/appendix/graphdb-concepts/#graphdb-node) + * [Node label](https://neo4j.com/docs/getting-started/appendix/graphdb-concepts/#graphdb-labels), which represents a type of nodes. +* [Relationship](https://neo4j.com/docs/getting-started/appendix/graphdb-concepts/#graphdb-relationship), which describes a connection between two nodes. + * [Relationship type](https://neo4j.com/docs/getting-started/appendix/graphdb-concepts/#graphdb-relationship-type) +* [Properties](https://neo4j.com/docs/getting-started/appendix/graphdb-concepts/#graphdb-properties), which are key-value pairs associated with nodes and relationships. + +### Data Mapping + +Data from collectors are mapped to graph elements in various types: + +1. Rows from collectors → Nodes in the graph +2. Rows from collectors → Relationships in the graph (including source and target nodes of the relationship) + +This is what you need to provide to define these mappings: + +* Specify [nodes to export](#nodes-to-export). +* [Declare extra node labels](#declare-extra-node-labels), for labels to appear as source/target nodes of relationships but not exported as nodes. +* Specify [relationships to export](#relationships-to-export). + +In addition, the same node may appear multiple times, from exported nodes and various relationships. +They should appear as the same node in the target graph database. +CocoIndex automatically [matches and deduplicates nodes](#nodes-matching-and-deduplicating) based on their primary key values. + +### Nodes to Export + +Here's how CocoIndex data elements map to nodes in the graph: + +| CocoIndex Element | Graph Element | +|-------------------|------------------| +| an export target | nodes with a unique label | +| a collected row | a node | +| a field | a property of node | + +Note that the label used in different `Nodes`s should be unique. + +`cocoindex.targets.Nodes` is to describe mapping to nodes. It has the following fields: + +* `label` (`str`): The label of the node. + +For example, consider we have collected the following rows: + + + +| filename | summary | +|----------|---------| +| chapter1.md | At the beginning, ... | +| chapter2.md | In the second day, ... | + + + +We can export them to nodes under label `Document` like this: + +```python +document_collector.export( + ... + cocoindex.targets.Neo4j( + ... + mapping=cocoindex.targets.Nodes(label="Document"), + ), + primary_key_fields=["filename"], +) +``` + +The collected rows will be mapped to nodes in knowledge database like this: + +```mermaid +graph TD + Doc_Chapter1@{ + shape: rounded + label: "**[Document]** + **filename\\*: chapter1.md** + summary: At the beginning, ..." + classDef: node + } + + Doc_Chapter2@{ + shape: rounded + label: "**[Document]** + **filename\\*: chapter2.md** + summary: In the second day, ..." + classDef: node + } + + classDef node font-size:8pt,text-align:left,stroke-width:2; +``` + +### Declare Extra Node Labels + +If a node label needs to appear as source or target of a relationship, but not exported as a node, you need to [declare](/docs/core/flow_def#target-declarations) the label with necessary configuration. + +The dataclass to describe the declaration is specific to each target (e.g. `cocoindex.targets.Neo4jDeclarations`), +while they share the following common fields: + +* `nodes_label` (required): The label of the node. +* Options for [storage indexes](/docs/core/flow_def#storage-indexes). + * `primary_key_fields` (required) + * `vector_indexes` (optional) + +Continuing the same example above. +Considering we want to extract relationships from `Document` to `Place` later (i.e. a document mentions a place), but the `Place` label isn't exported as a node, we need to declare it: + +```python +flow_builder.declare( + cocoindex.targets.Neo4jDeclarations( + connection = ..., + nodes_label="Place", + primary_key_fields=["name"], + ), +) +``` + +### Relationships to Export + +Here's how CocoIndex data elements map to relationships in the graph: + +| CocoIndex Element | Graph Element | +|-------------------|------------------| +| an export target | relationships with a unique type | +| a collected row | a relationship | +| a field | a property of relationship, or a property of source/target node, based on configuration | + +Note that the type used in different `Relationships`s should be unique. + +`cocoindex.targets.Relationships` is to describe mapping to relationships. It has the following fields: + +* `rel_type` (`str`): The type of the relationship. +* `source`/`target` (`cocoindex.targets.NodeFromFields`): Specify how to extract source/target node information from specific fields in the collected row. It has the following fields: + * `label` (`str`): The label of the node. + * `fields` (`Sequence[cocoindex.targets.TargetFieldMapping]`): Specify field mappings from the collected rows to node properties, with the following fields: + * `source` (`str`): The name of the field in the collected row. + * `target` (`str`, optional): The name of the field to use as the node field. If unspecified, will use the same as `source`. + + :::note Map necessary fields for nodes of relationships + + You need to map the following fields for nodes of each relationship: + + * Make sure all primary key fields for the label are mapped. + * Optionally, you can also map non-key fields. If you do so, please make sure all value fields are mapped. + + ::: + +All fields in the collector that are not used in mappings for source or target node fields will be mapped to relationship properties. + +For example, consider we have collected the following rows, to describe places mentioned in each file, along with embeddings of the places: + + + +| doc_filename | place_name | place_embedding | location | +|----------|-------|-----------------|-----------------| +| chapter1.md | Crystal Palace | [0.1, 0.5, ...] | 12 | +| chapter2.md | Magic Forest | [0.4, 0.2, ...] | 23 | +| chapter2.md | Crystal Palace | [0.1, 0.5, ...] | 56 | + + + +We can export them to relationships under type `MENTION` like this: + +```python +doc_place_collector.export( + ... + cocoindex.targets.Neo4j( + ... + mapping=cocoindex.targets.Relationships( + rel_type="MENTION", + source=cocoindex.targets.NodeFromFields( + label="Document", + fields=[cocoindex.targets.TargetFieldMapping(source="doc_filename", target="filename")], + ), + target=cocoindex.targets.NodeFromFields( + label="Place", + fields=[ + cocoindex.targets.TargetFieldMapping(source="place_name", target="name"), + cocoindex.targets.TargetFieldMapping(source="place_embedding", target="embedding"), + ], + ), + ), + ), + ... +) +``` + +The `doc_filename` field is mapped to `Document.filename` property for the source node, while `place_name` and `place_embedding` are mapped to `Place.name` and `Place.embedding` properties for the target node. +The remaining field `location` becomes a property of the relationship. +For the data above, we get a bunch of relationships like this: + +```mermaid +graph TD + Doc_Chapter1@{ + shape: rounded + label: "**[Document]** + **filename\\*: chapter1.md**" + classDef: nodeRef + } + + Doc_Chapter2_a@{ + shape: rounded + label: "**[Document]** + **filename\\*: chapter2.md**" + classDef: nodeRef + } + + Doc_Chapter2_b@{ + shape: rounded + label: "**[Document]** + **filename\\*: chapter2.md**" + classDef: nodeRef + } + + Place_CrystalPalace_a@{ + shape: rounded + label: "**[Place]** + **name\\*: Crystal Palace** + embedding: [0.1, 0.5, ...]" + classDef: node + } + + Place_MagicForest@{ + shape: rounded + label: "**[Place]** + **name\\*: Magic Forest** + embedding: [0.4, 0.2, ...]" + classDef: node + } + + Place_CrystalPalace_b@{ + shape: rounded + label: "**[Place]** + **name\\*: Crystal Palace** + embedding: [0.1, 0.5, ...]" + classDef: node + } + + + Doc_Chapter1:::nodeRef -- **:MENTION** (location:12) --> Place_CrystalPalace_a:::node + Doc_Chapter2_a:::nodeRef -- **:MENTION** (location:23) --> Place_MagicForest:::node + Doc_Chapter2_b:::nodeRef -- **:MENTION** (location:56) --> Place_CrystalPalace_b:::node + + classDef nodeRef font-size:8pt,text-align:left,fill:transparent,stroke-width:1,stroke-dasharray:5 5; + classDef node font-size:8pt,text-align:left,stroke-width:2; + +``` + +### Nodes Matching and Deduplicating + +The nodes and relationships we got above are discrete elements. +To fit them into a connected property graph, CocoIndex will match and deduplicate nodes automatically: + +* Match nodes based on their primary key values. Nodes with the same primary key values are considered as the same node. +* For non-primary key fields (a.k.a. value fields), CocoIndex will pick the values from an arbitrary one. + If multiple nodes (before deduplication) with the same primary key provide value fields, an arbitrary one will be picked. + +:::note + +The best practice is to make the value fields consistent across different appearances of the same node, to avoid non-determinism in the exported graph. + +::: + +After matching and deduplication, we get the final graph: + +```mermaid +graph TD + Doc_Chapter1@{ + shape: rounded + label: "**[Document]** + **filename\\*: chapter1.md** + summary: At the beginning, ..." + classDef: node + } + + Doc_Chapter2@{ + shape: rounded + label: "**[Document]** + **filename\\*: chapter2.md** + summary: In the second day, ..." + classDef: node + } + + Place_CrystalPalace@{ + shape: rounded + label: "**[Place]** + **name\\*: Crystal Palace** + embedding: [0.1, 0.5, ...]" + classDef: node + } + + Place_MagicForest@{ + shape: rounded + label: "**[Place]** + **name\\*: Magic Forest** + embedding: [0.4, 0.2, ...]" + classDef: node + } + + Doc_Chapter1:::node -- **:MENTION** (location:12) --> Place_CrystalPalace:::node + Doc_Chapter2:::node -- **:MENTION** (location:23) --> Place_MagicForest:::node + Doc_Chapter2:::node -- **:MENTION** (location:56) --> Place_CrystalPalace:::node + + classDef node font-size:8pt,text-align:left,stroke-width:2; +``` + +### Examples + +You can find end-to-end examples fitting into any of supported property graphs in the following directories: +* + +* + + + diff --git a/docs/docs/targets/kuzu.md b/docs/docs/targets/kuzu.md new file mode 100644 index 00000000..b8fcb586 --- /dev/null +++ b/docs/docs/targets/kuzu.md @@ -0,0 +1,60 @@ +--- +title: Kuzu +description: CocoIndex Kuzu Target +toc_max_heading_level: 4 +--- +import { ExampleButton } from '../../src/components/GitHubButton'; + +# Kuzu + +Exports data to a [Kuzu](https://kuzu.com/) graph database. + +## Get Started + +Read [Property Graph Targets](./index.md#property-graph-targets) for more information to get started on how it works in CocoIndex. + +## Spec + +CocoIndex supports talking to Kuzu through its [API server](https://github.com/kuzudb/api-server). + +The `Kuzu` target spec takes the following fields: + +* `connection` ([auth reference](/docs/core/flow_def#auth-registry) to `KuzuConnectionSpec`): The connection to the Kuzu database. `KuzuConnectionSpec` has the following fields: + * `api_server_url` (`str`): The URL of the Kuzu API server, e.g. `http://localhost:8123`. +* `mapping` (`Nodes | Relationships`): The mapping from collected row to nodes or relationships of the graph. For either [nodes to export](#nodes-to-export) or [relationships to export](#relationships-to-export). + +Kuzu also provides a declaration spec `KuzuDeclaration`, to configure indexing options for nodes only referenced by relationships. It has the following fields: + +* `connection` (auth reference to `KuzuConnectionSpec`) +* Fields for [nodes to declare](#declare-extra-node-labels), including + * `nodes_label` (required) + * `primary_key_fields` (required) + +## Kuzu dev instance + +If you don't have a Kuzu instance yet, you can bring up a Kuzu API server locally by running: + +```bash +KUZU_DB_DIR=$HOME/.kuzudb +KUZU_PORT=8123 +docker run -d --name kuzu -p ${KUZU_PORT}:8000 -v ${KUZU_DB_DIR}:/database kuzudb/api-server:latest +``` + +To explore the graph you built with Kuzu, you can use the [Kuzu Explorer](https://github.com/kuzudb/explorer). +Currently Kuzu API server and the explorer cannot be up at the same time. So you need to stop the API server before running the explorer. + +To start the instance of the explorer, run: + +```bash +KUZU_EXPLORER_PORT=8124 +docker run -d --name kuzu-explorer -p ${KUZU_EXPLORER_PORT}:8000 -v ${KUZU_DB_DIR}:/database -e MODE=READ_ONLY kuzudb/explorer:latest +``` + +You can then access the explorer at [http://localhost:8124](http://localhost:8124). + +## Example + \ No newline at end of file diff --git a/docs/docs/targets/lancedb.md b/docs/docs/targets/lancedb.md new file mode 100644 index 00000000..cda7fdf8 --- /dev/null +++ b/docs/docs/targets/lancedb.md @@ -0,0 +1,92 @@ +--- +title: LanceDB +description: CocoIndex LanceDB Target +toc_max_heading_level: 4 +--- + +import { ExampleButton } from '../../src/components/GitHubButton'; + +# LanceDB + +Exports data to a [LanceDB](https://lancedb.github.io/lancedb/) table. + +## Data Mapping + +Here's how CocoIndex data elements map to LanceDB elements during export: + +| CocoIndex Element | LanceDB Element | +|-------------------|-----------------| +| an export target | a unique table | +| a collected row | a row | +| a field | a column | + + +::::info Installation and import + +This target is provided via an optional dependency `[lancedb]`: + +```sh +pip install "cocoindex[lancedb]" +``` + +To use it, you need to import the submodule `cocoindex.targets.lancedb`: + +```python +import cocoindex.targets.lancedb as coco_lancedb +``` + +:::: + +## Spec + +The spec `coco_lancedb.LanceDB` takes the following fields: + +* `db_uri` (`str`, required): The LanceDB database location (e.g. `./lancedb_data`). +* `table_name` (`str`, required): The name of the table to export the data to. +* `db_options` (`coco_lancedb.DatabaseOptions`, optional): Advanced database options. + * `storage_options` (`dict[str, Any]`, optional): Passed through to LanceDB when connecting. + +Additional notes: + +* Exactly one primary key field is required for LanceDB targets. We create B-Tree index on this key column. + +:::info + +LanceDB has a limitation that it cannot build a vector index on an empty table (see [LanceDB issue #4034](https://github.com/lancedb/lance/issues/4034)). +If you want to use vector indexes, you can run the flow once to populate the target table with data, and then create the vector indexes. + +::: + +You can find an end-to-end example here: [examples/text_embedding_lancedb](https://github.com/cocoindex-io/cocoindex/tree/main/examples/text_embedding_lancedb). + +## `connect_async()` helper + +We provide a helper to obtain a shared `AsyncConnection` that is reused across your process and shared with CocoIndex's writer for strong read-after-write consistency: + +```python +from cocoindex.targets import lancedb as coco_lancedb + +db = await coco_lancedb.connect_async("./lancedb_data") +table = await db.open_table("TextEmbedding") +``` + +Signature: + +```python +def connect_async( + db_uri: str, + *, + db_options: coco_lancedb.DatabaseOptions | None = None, + read_consistency_interval: datetime.timedelta | None = None +) -> lancedb.AsyncConnection +``` + +Once `db_uri` matches, it automatically reuses the same connection instance without re-establishing a new connection. +This achieves strong consistency between your indexing and querying logic, if they run in the same process. + +## Example + diff --git a/docs/docs/targets/neo4j.md b/docs/docs/targets/neo4j.md new file mode 100644 index 00000000..5b9ee8bb --- /dev/null +++ b/docs/docs/targets/neo4j.md @@ -0,0 +1,52 @@ +--- +title: Neo4j +description: CocoIndex Neo4j Target +toc_max_heading_level: 4 +--- +import { ExampleButton } from '../../src/components/GitHubButton'; + +# Neo4j + +**Exports data to a [Neo4j](https://neo4j.com/) graph database.** + + +## Get Started +Read [Property Graph Targets](./index.md#property-graph-targets) for more information to get started on how it works in CocoIndex. + + +## Spec + +The `Neo4j` target spec takes the following fields: + +* `connection` ([auth reference](/docs/core/flow_def#auth-registry) to `Neo4jConnectionSpec`): The connection to the Neo4j database. `Neo4jConnectionSpec` has the following fields: + * `url` (`str`): The URI of the Neo4j database to use as the internal storage, e.g. `bolt://localhost:7687`. + * `user` (`str`): Username for the Neo4j database. + * `password` (`str`): Password for the Neo4j database. + * `db` (`str`, optional): The name of the Neo4j database to use as the internal storage, e.g. `neo4j`. +* `mapping` (`Nodes | Relationships`): The mapping from collected row to nodes or relationships of the graph. For either [nodes to export](#nodes-to-export) or [relationships to export](#relationships-to-export). + +Neo4j also provides a declaration spec `Neo4jDeclaration`, to configure indexing options for nodes only referenced by relationships. It has the following fields: + +* `connection` (auth reference to `Neo4jConnectionSpec`) +* Fields for [nodes to declare](#declare-extra-node-labels), including + * `nodes_label` (required) + * `primary_key_fields` (required) + * `vector_indexes` (optional) + +## Neo4j dev instance + +If you don't have a Neo4j database, you can start a Neo4j database using our docker compose config: + +```bash +docker compose -f <(curl -L https://raw.githubusercontent.com/cocoindex-io/cocoindex/refs/heads/main/dev/neo4j.yaml) up -d +``` + +This will bring up a Neo4j instance, which can be accessed by username `neo4j` and password `cocoindex`. +You can access the Neo4j browser at [http://localhost:7474](http://localhost:7474). + +## Example + \ No newline at end of file diff --git a/docs/docs/targets/postgres.md b/docs/docs/targets/postgres.md new file mode 100644 index 00000000..0a748681 --- /dev/null +++ b/docs/docs/targets/postgres.md @@ -0,0 +1,55 @@ +--- +title: Postgres +description: CocoIndex Postgres Target +toc_max_heading_level: 4 +--- + +import { ExampleButton } from '../../src/components/GitHubButton'; + +# Postgres + +Exports data to Postgres database (with pgvector extension). + +## Data Mapping + +Here's how CocoIndex data elements map to Postgres elements during export: + +| CocoIndex Element | Postgres Element | +|-------------------|------------------| +| an export target | a unique table | +| a collected row | a row | +| a field | a column | + +For example, if you have a data collector that collects rows with fields `id`, `title`, and `embedding`, it will be exported to a Postgres table with corresponding columns. +It should be a unique table, meaning that no other export target should export to the same table. + +:::warning vector type mapping to Postgres + +Since vectors in pgvector must have fixed dimension, we only map vectors of number types with fixed dimension (i.e. *Vector[cocoindex.Float32, N]*, *Vector[cocoindex.Float64, N]*, and *Vector[cocoindex.Int64, N]*) to `vector(N)` columns. +For all other vector types, we map them to `jsonb` columns. + +::: + +:::info U+0000 (NUL) characters in strings + +U+0000 (NUL) is a valid character in Unicode, but Postgres has a limitation that strings (including `text`-like types and strings in `jsonb`) cannot contain them. +CocoIndex automatically strips U+0000 (NUL) characters from strings before exporting to Postgres. For example, if you have a string `"Hello\0World"`, it will be exported as `"HelloWorld"`. + +::: + +## Spec + +The spec takes the following fields: + +* `database` ([auth reference](/docs/core/flow_def#auth-registry) to `DatabaseConnectionSpec`, optional): The connection to the Postgres database. + See [DatabaseConnectionSpec](/docs/core/settings#databaseconnectionspec) for its specific fields. + If not provided, will use the same database as the [internal storage](/docs/core/basics#internal-storage). + +* `table_name` (`str`, optional): The name of the table to store to. If unspecified, will use the table name `[${AppNamespace}__]${FlowName}__${TargetName}`, e.g. `DemoFlow__doc_embeddings` or `Staging__DemoFlow__doc_embeddings`. + +## Example + diff --git a/docs/docs/targets/qdrant.md b/docs/docs/targets/qdrant.md new file mode 100644 index 00000000..26ddf54e --- /dev/null +++ b/docs/docs/targets/qdrant.md @@ -0,0 +1,56 @@ +--- +title: Qdrant +description: CocoIndex Qdrant Target +toc_max_heading_level: 4 +--- + +import { ExampleButton } from '../../src/components/GitHubButton'; + +# Qdrant + +Exports data to a [Qdrant](https://qdrant.tech/) collection. + +## Data Mapping + +Here's how CocoIndex data elements map to Qdrant elements during export: + +| CocoIndex Element | Qdrant Element | +|-------------------|------------------| +| an export target | a unique collection | +| a collected row | a point | +| a field | a named vector, if fits into Qdrant vector; or a field within payload otherwise | + +The following vector types fit into Qdrant vector: +* One-dimensional vectors with fixed dimension, e.g. *Vector[Float32, N]*, *Vector[Float64, N]* and *Vector[Int64, N]*. + We map them to [dense vectors](https://qdrant.tech/documentation/concepts/vectors/#dense-vectors). +* Two-dimensional vectors whose inner layer is a one-dimensional vector with fixed dimension, e.g. *Vector[Vector[Float32, N]]*, *Vector[Vector[Int64, N]]*, *Vector[Vector[Float64, N]]*. The outer layer may or may not have a fixed dimension. + We map them to [multivectors](https://qdrant.tech/documentation/concepts/vectors/#multivectors). + + +:::warning vector type mapping to Qdrant + +Since vectors in Qdrant must have fixed dimension, we only map vectors of number types with fixed dimension to Qdrant vectors. +For all other vector types, we map to Qdrant payload as JSON arrays. + +::: + +## Spec + +The spec takes the following fields: + +* `connection` ([auth reference](/docs/core/flow_def#auth-registry) to `QdrantConnection`, optional): The connection to the Qdrant instance. `QdrantConnection` has the following fields: + * `grpc_url` (`str`): The [gRPC URL](https://qdrant.tech/documentation/interfaces/#grpc-interface) of the Qdrant instance, e.g. `http://localhost:6334/`. + * `api_key` (`str`, optional). API key to authenticate requests with. + + If `connection` is not provided, will use local Qdrant instance at `http://localhost:6334/` by default. + +* `collection_name` (`str`, required): The name of the collection to export the data to. + +You can find an end-to-end example [here](https://github.com/cocoindex-io/cocoindex/tree/main/examples/text_embedding_qdrant). + +## Example + diff --git a/docs/docusaurus.config.ts b/docs/docusaurus.config.ts index 01ff5cc7..2cd1e313 100644 --- a/docs/docusaurus.config.ts +++ b/docs/docusaurus.config.ts @@ -69,6 +69,10 @@ const config: Config = { from: '/about/contributing', to: '/contributing/guide', }, + { + from: '/ops/targets', + to: '/targets', + }, ], }, ], diff --git a/docs/sidebars.ts b/docs/sidebars.ts index 17a85ad1..f10cdf9a 100644 --- a/docs/sidebars.ts +++ b/docs/sidebars.ts @@ -41,7 +41,20 @@ const sidebars: SidebarsConfig = { items: [ 'ops/sources', 'ops/functions', - 'ops/targets', + ], + }, + { + type: 'category', + label: 'Built-in Targets', + link: { type: 'doc', id: 'targets/index' }, + collapsed: true, + items: [ + + 'targets/postgres', + 'targets/qdrant', + 'targets/lancedb', + 'targets/neo4j', + 'targets/kuzu', ], }, {