Copy the .env.sample file to .env and set an OpenAI API key.
cp .env.sample .envmake build
# or
docker compose buildmake start
# or
docker compose up -dmake migrate
# or
docker compose exec server bun run migrate --skip-generateOpen Swagger on http://localhost:8080 to interact with the API.
Upload any TXT or PDF file to the /upload endpoint.
Send queries to the /search endpoint.
Set Bearer dev-secret as the authorization token.
Upload each example file from the examples directory.
Examples questions to ask:
- What is the Great Filter?
- It should return page 7 of
the-fermi-paradox.pdfas the first result.
- It should return page 7 of
- Page 10 of the Fermi Paradox post
- It should return 3 items from page 10 of
the-fermi-paradox.pdf.
- It should return 3 items from page 10 of
- How old is the lost ark?
- It should return page 1 of
test.pdfcontaining the info that if the ark is real, it would be ~3000 years old.
- It should return page 1 of
The application is an Express server that uses PostgreSQL with the pgvector extension to store and search document embeddings.
To simplify the application and user interface, metadata-based search keys are extracted from the user’s query instead of using a separate metadata search.
First, a model extracts file names and page numbers if present in the user’s query. These are then used to narrow down and speed up the similarity search.
When storing the vectors, the metadata is extracted from the document and stored in the database along with the vector.
The application computes relevance using the inner product of embeddings. Since the embeddings are normalized, the inner product is equivalent to the cosine similarity which is the suggested similarity metric for OpenAI embeddings.
As more and more documents are added to the index, the search time increases linearly. To address this, the application uses a Hierarchical Navigable Small World (HNSW) index to speed up the search while maintaining high retrieval quality.
Since I was focusing on speed, I chose HNSW over IVFFlat, sacrificing size for speed.
I arbitrarily chose 1000 characters as the maximum chunk size with a 200 character overlap. Multiple blog posts used the same or similar values, and this configuration yielded good results while manually testing the application.
The application uses a RecursiveCharacterTextSplitter to split the text into chunks. The default separators for the splitter are ["\n\n", "\n", " ", ""]. This means that the app will first try to split the text by paragraphs, then by sentences, and finally by words, ensuring the highest possible semantic meaning in each chunk.