AskBend is a Rust project that utilizes the power of Databend and OpenAI to create a SQL-based knowledge base from Markdown files.
Databend is a cloud-native data warehouse adept at storing and performing vector computations, making it suitable for this use case.
Databend Cloud seamlessly integrates with OpenAI's capabilities, such as embedding generation, cosine distance calculation, and text completion. This integration means you don't need to interact with OpenAI directly; Databend Cloud manages everything.
The project automatically generates document embeddings from the content, enabling users to search and retrieve the most relevant information to their queries using SQL.
SQL-Based means you don't need any OpenAI API knowledge. With the Databend Cloud platform, you can perform these tasks using SQL. Some SQL AI functions of Databend Cloud include:
- ai_embedding_vector: Get the vector from OpenAI API
- ai_text_completion: Get the completion of a text
- cosine_distance: Calculate the distance between embedding vectors
The project follows this general process:
- Read and parse Markdown files from a directory.
- Extract the content and store it in the askbend.doc table.
- Compute embeddings for the content using Databend Cloud's built-in AI capabilities, including OpenAI's embedding generation, all through SQL.
- When a user queries, generate the query embedding using Databend Cloud's SQL-based
ai_embedding_vector
function. - Perform a vector calculation to find the most relevant doc.content using Databend Cloud's SQL-based
cosine_distance
function. - Concatenate the retrieved content and use OpenAI's completion capabilities with Databend Cloud's SQL-based
ai_text_completion
function. - Output the completion result in Markdown format.
git clone https://github.com/datafuselabs/askbend
cd askbend
make setup
make build
CREATE DATABASE askbend;
USE askbend;
CREATE TABLE doc (path VARCHAR, content VARCHAR, embedding ARRAY(FLOAT32));
4. Modify the configuration file conf/askbend.toml
# Usage:
# askbend -c askbend.toml
[data]
# Path to the directory containing your markdown documents
path = "data/"
[database]
database = "askbend"
table = "doc"
# Data source name (DSN) for connecting to your Databend cloud warehouse
# https://docs.databend.com/using-databend-cloud/warehouses/connecting-a-warehouse
dsn = "databend://<sql-user>:<sql-password>@<your-databend-cloud-warehouse>/default"
[server]
host = "0.0.0.0"
port = 8081
[query]
top = 3
prompt = '''
<your prompt> ...
Documentation sections:
{{context}}
Question:
{{query}}
'''
./target/release/askbend -c conf/askbend.toml --rebuild
[2023-04-01T07:17:13Z INFO ] Step-1: begin parser all markdown files
[2023-04-01T07:17:14Z INFO ] Step-1: finish parser all markdown files:397, sections:969, tokens:117758
[2023-04-01T07:17:14Z INFO ] Step-2: begin insert to table
[2023-04-01T07:17:14Z INFO ] Step-2: finish insert to table
[2023-04-01T07:17:14Z INFO ] Step-3: begin generate embedding, may take some minutes
[2023-04-01T07:26:03Z INFO ] Step-3: finish generate embedding
... ...
The --rebuild
flag rebuilds all the embeddings for the data directory. This process may take a few minutes, depending on the number of Markdown files.
./target/release/askbend -c conf/askbend.toml
curl -X POST -H "Content-Type: application/json" -d '{"query": "tell me how to do copy"}' http://localhost:8081/query
Response:
{"result":["\n\nYou can use the `COPY INTO <table>` command to copy data from an internal stage, Amazon S3 bucket, or a remote file into a table in Databend. \n\nFor example, to copy data from an internal stage, you can use the following command:\n\n```\nCOPY INTO <table>\nFROM (\n SELECT <columns>\n FROM @<stage>\n FILE_FORMAT = (TYPE = PARQUET)\n)\n```\n\nFor more information, please refer to the [Tutorial: Load from an internal stage](../../12-load-data/00-stage.md) and [Tutorial: Load from an Amazon S3 bucket](../../12-load-data/01-s3.md) sections in the Databend documentation."]}
This API document describes how to use the Databend query API to submit queries and receive results.
http://:8081/query
The request body should be a JSON object containing a single field query
, which is the query string.
Example:
{
"query": "whats the fast way to load data to databend"
}
On successful query execution, the API will return a 200 OK status code, along with a JSON object containing the field result.
The result field is an array of strings. However, we only need to consider the first string in the array as the final result.
The API assumes that if the query was successful, the first item in the result array is the most relevant answer.