Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 44 additions & 19 deletions docs/docs/ops/storages.md
Original file line number Diff line number Diff line change
Expand Up @@ -386,19 +386,7 @@ You can find end-to-end examples fitting into any of supported property graphs i

### Neo4j

If you don't have a Neo4j database, you can start a Neo4j database using our docker compose config:

```bash
docker compose -f <(curl -L https://raw.githubusercontent.com/cocoindex-io/cocoindex/refs/heads/main/dev/neo4j.yaml) up -d
```

:::warning

The docker compose config above will start a Neo4j Enterprise instance under the [Evaluation License](https://neo4j.com/terms/enterprise_us/),
with 30 days trial period.
Please read and agree the license before starting the instance.

:::
#### Spec

The `Neo4j` target spec takes the following fields:

Expand All @@ -417,17 +405,32 @@ Neo4j also provides a declaration spec `Neo4jDeclaration`, to configure indexing
* `primary_key_fields` (required)
* `vector_indexes` (optional)

### Kuzu
#### Neo4j dev instance

CocoIndex supports talking to Kuzu through its [API server](https://github.com/kuzudb/api-server).
You can bring up a Kuzu API server locally by running:
If you don't have a Neo4j database, you can start a Neo4j database using our docker compose config:

```bash
KUZU_DB_DIR=$HOME/.kuzudb
KUZU_PORT=8123
docker run -d --name kuzu -p ${KUZU_PORT}:8000 -v ${KUZU_DB_DIR}:/database kuzudb/api-server:latest
docker compose -f <(curl -L https://raw.githubusercontent.com/cocoindex-io/cocoindex/refs/heads/main/dev/neo4j.yaml) up -d
```

If will bring up a Neo4j instance, which can be accessed by username `neo4j` and password `cocoindex`.
You can access the Neo4j browser at [http://localhost:7474](http://localhost:7474).

:::warning

The docker compose config above will start a Neo4j Enterprise instance under the [Evaluation License](https://neo4j.com/terms/enterprise_us/),
with 30 days trial period.
Please read and agree the license before starting the instance.

:::


### Kuzu

#### Spec

CocoIndex supports talking to Kuzu through its [API server](https://github.com/kuzudb/api-server).

The `Kuzu` target spec takes the following fields:

* `connection` (type: [auth reference](../core/flow_def#auth-registry) to `KuzuConnectionSpec`): The connection to the Kuzu database. `KuzuConnectionSpec` has the following fields:
Expand All @@ -440,3 +443,25 @@ Kuzu also provides a declaration spec `KuzuDeclaration`, to configure indexing o
* Fields for [nodes to declare](#declare-extra-node-labels), including
* `nodes_label` (required)
* `primary_key_fields` (required)

#### Kuzu dev instance

If you don't have a Kuzu instance yet, you can bring up a Kuzu API server locally by running:

```bash
KUZU_DB_DIR=$HOME/.kuzudb
KUZU_PORT=8123
docker run -d --name kuzu -p ${KUZU_PORT}:8000 -v ${KUZU_DB_DIR}:/database kuzudb/api-server:latest
```

To explore the graph you built with Kuzu, you can use the [Kuzu Explorer](https://github.com/kuzudb/explorer).
Currently Kuzu API server and the explorer cannot be up at the same time. So you need to stop the API server before running the explorer.

To start the instance of the explorer, run:

```bash
KUZU_EXPLORER_PORT=8124
docker run -d --name kuzu-explorer -p ${KUZU_EXPLORER_PORT}:8000 -v ${KUZU_DB_DIR}:/database -e MODE=READ_ONLY kuzudb/explorer:latest
```

You can then access the explorer at [http://localhost:8124](http://localhost:8124).
17 changes: 7 additions & 10 deletions examples/docs_to_knowledge_graph/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ Please drop [Cocoindex on Github](https://github.com/cocoindex-io/cocoindex) a s

![example-explanation](https://github.com/user-attachments/assets/07ddbd60-106f-427f-b7cc-16b73b142d27)


## Prerequisite
* [Install Postgres](https://cocoindex.io/docs/getting_started/installation#-install-postgres) if you don't have one.
* [Install Neo4j](https://cocoindex.io/docs/ops/storages#neo4j) if you don't have one.
* Install [Neo4j](https://cocoindex.io/docs/ops/storages#neo4j-dev-instance) or [Kuzu](https://cocoindex.io/docs/ops/storages#kuzu-dev-instance) if you don't have one.
* The example uses Neo4j by default for now. If you want to use Kuzu, find out the "SELECT ONE GRAPH DATABASE TO USE" section and switch the active branch.
* [Configure your OpenAI API key](https://cocoindex.io/docs/ai/llm#openai).

## Documentation
Expand Down Expand Up @@ -45,21 +45,18 @@ cocoindex update main.py

### Browse the knowledge graph

After the knowledge graph is build, you can explore the knowledge graph you built in Neo4j Browser.
After the knowledge graph is built, you can explore the knowledge graph.

For the dev enviroment, you can connect neo4j browser using credentials:
- username: `neo4j`
- password: `cocoindex`
which is pre-configured in the our docker compose [config.yaml](https://raw.githubusercontent.com/cocoindex-io/cocoindex/refs/heads/main/dev/neo4j.yaml).
* If you're using Neo4j, you can open the explorer at [http://localhost:7474](http://localhost:7474), with username `neo4j` and password `cocoindex`.
* If you're using Kuzu, you can start a Kuzu explorer locally. See [Kuzu dev instance](https://cocoindex.io/docs/ops/storages#kuzu-dev-instance) for more details.

You can open it at [http://localhost:7474](http://localhost:7474), and run the following Cypher query to get all relationships:
You can run the following Cypher query to get all relationships:

```cypher
MATCH p=()-->() RETURN p
```
<img width="1366" alt="neo4j-for-coco-docs" src="https://github.com/user-attachments/assets/3c8b6329-6fee-4533-9480-571399b57e57" />


<img width="1366" alt="neo4j-for-coco-docs" src="https://github.com/user-attachments/assets/3c8b6329-6fee-4533-9480-571399b57e57" />

## CocoInsight
I used CocoInsight (Free beta now) to troubleshoot the index generation and understand the data lineage of the pipeline.
Expand Down
49 changes: 26 additions & 23 deletions examples/docs_to_knowledge_graph/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,27 +5,6 @@
import dataclasses
import cocoindex


@dataclasses.dataclass
class DocumentSummary:
"""Describe a summary of a document."""

title: str
summary: str


@dataclasses.dataclass
class Relationship:
"""
Describe a relationship between two entities.
Subject and object should be Core CocoIndex concepts only, should be nouns. For example, `CocoIndex`, `Incremental Processing`, `ETL`, `Data` etc.
"""

subject: str
predicate: str
object: str


neo4j_conn_spec = cocoindex.add_auth_entry(
"Neo4jConnection",
cocoindex.storages.Neo4jConnection(
Expand All @@ -41,19 +20,43 @@ class Relationship:
),
)

# Use Neo4j as the graph database
# SELECT ONE GRAPH DATABASE TO USE
# This example can use either Neo4j or Kuzu as the graph database.
# Please make sure only one branch is live and others are commented out.

# Use Neo4j
GraphDbSpec = cocoindex.storages.Neo4j
GraphDbConnection = cocoindex.storages.Neo4jConnection
GraphDbDeclaration = cocoindex.storages.Neo4jDeclaration
conn_spec = neo4j_conn_spec

# Use Kuzu as the graph database
# Use Kuzu
# GraphDbSpec = cocoindex.storages.Kuzu
# GraphDbConnection = cocoindex.storages.KuzuConnection
# GraphDbDeclaration = cocoindex.storages.KuzuDeclaration
# conn_spec = kuzu_conn_spec


@dataclasses.dataclass
class DocumentSummary:
"""Describe a summary of a document."""

title: str
summary: str


@dataclasses.dataclass
class Relationship:
"""
Describe a relationship between two entities.
Subject and object should be Core CocoIndex concepts only, should be nouns. For example, `CocoIndex`, `Incremental Processing`, `ETL`, `Data` etc.
"""

subject: str
predicate: str
object: str


@cocoindex.flow_def(name="DocsToKG")
def docs_to_kg_flow(
flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope
Expand Down
14 changes: 7 additions & 7 deletions examples/product_recommendation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ Please drop [CocoIndex on Github](https://github.com/cocoindex-io/cocoindex) a s

## Prerequisite
* [Install Postgres](https://cocoindex.io/docs/getting_started/installation#-install-postgres) if you don't have one.
* [Install Neo4j](https://cocoindex.io/docs/ops/storages#neo4j) if you don't have one.
* Install [Neo4j](https://cocoindex.io/docs/ops/storages#neo4j-dev-instance) or [Kuzu](https://cocoindex.io/docs/ops/storages#kuzu-dev-instance) if you don't have one.
* The example uses Neo4j by default for now. If you want to use Kuzu, find out the "SELECT ONE GRAPH DATABASE TO USE" section and switch the active branch.
* [Configure your OpenAI API key](https://cocoindex.io/docs/ai/llm#openai).

## Documentation
Expand Down Expand Up @@ -39,18 +40,17 @@ cocoindex update main.py

### Browse the knowledge graph

After the knowledge graph is built, you can explore the knowledge graph you built in Neo4j Browser.
After the knowledge graph is built, you can explore the knowledge graph.

For the dev enviroment, you can connect neo4j browser using credentials:
- username: `neo4j`
- password: `cocoindex`
which is pre-configured in the our docker compose [config.yaml](https://raw.githubusercontent.com/cocoindex-io/cocoindex/refs/heads/main/dev/neo4j.yaml).
* If you're using Neo4j, you can open the explorer at [http://localhost:7474](http://localhost:7474), with username `neo4j` and password `cocoindex`.
* If you're using Kuzu, you can start a Kuzu explorer locally. See [Kuzu dev instance](https://cocoindex.io/docs/ops/storages#kuzu-dev-instance) for more details.

You can open it at [http://localhost:7474](http://localhost:7474), and run the following Cypher query to get all relationships:
You can run the following Cypher query to get all relationships:

```cypher
MATCH p=()-->() RETURN p
```

![Neo4j Browser Screenshot](img/neo4j.png)

## CocoInsight
Expand Down
60 changes: 32 additions & 28 deletions examples/product_recommendation/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,38 @@
import cocoindex
from jinja2 import Template

neo4j_conn_spec = cocoindex.add_auth_entry(
"Neo4jConnection",
cocoindex.storages.Neo4jConnection(
uri="bolt://localhost:7687",
user="neo4j",
password="cocoindex",
),
)
kuzu_conn_spec = cocoindex.add_auth_entry(
"KuzuConnection",
cocoindex.storages.KuzuConnection(
api_server_url="http://localhost:8123",
),
)

# SELECT ONE GRAPH DATABASE TO USE
# This example can use either Neo4j or Kuzu as the graph database.
# Please make sure only one branch is live and others are commented out.

# Use Neo4j
GraphDbSpec = cocoindex.storages.Neo4j
GraphDbConnection = cocoindex.storages.Neo4jConnection
GraphDbDeclaration = cocoindex.storages.Neo4jDeclaration
conn_spec = neo4j_conn_spec

# Use Kuzu
# GraphDbSpec = cocoindex.storages.Kuzu
# GraphDbConnection = cocoindex.storages.KuzuConnection
# GraphDbDeclaration = cocoindex.storages.KuzuDeclaration
# conn_spec = kuzu_conn_spec


# Template for rendering product information as markdown to provide information to LLMs
PRODUCT_TEMPLATE = """
# {{ title }}
Expand Down Expand Up @@ -77,34 +109,6 @@ def extract_product_info(product: cocoindex.Json, filename: str) -> ProductInfo:
)


neo4j_conn_spec = cocoindex.add_auth_entry(
"Neo4jConnection",
cocoindex.storages.Neo4jConnection(
uri="bolt://localhost:7687",
user="neo4j",
password="cocoindex",
),
)
kuzu_conn_spec = cocoindex.add_auth_entry(
"KuzuConnection",
cocoindex.storages.KuzuConnection(
api_server_url="http://localhost:8123",
),
)

# Use Neo4j as the graph database
GraphDbSpec = cocoindex.storages.Neo4j
GraphDbConnection = cocoindex.storages.Neo4jConnection
GraphDbDeclaration = cocoindex.storages.Neo4jDeclaration
conn_spec = neo4j_conn_spec

# Use Kuzu as the graph database
# GraphDbSpec = cocoindex.storages.Kuzu
# GraphDbConnection = cocoindex.storages.KuzuConnection
# GraphDbDeclaration = cocoindex.storages.KuzuDeclaration
# conn_spec = kuzu_conn_spec


@cocoindex.flow_def(name="StoreProduct")
def store_product_flow(
flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope
Expand Down