Skip to content

Vector Database implemented in Golang with support for full-text and vector search as well as fault tolerance via Raft.

Notifications You must be signed in to change notification settings

farouqzaib/mettis

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

26 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

mettisDB: Distributed lite vector database built from *scratch.

What it does:

  • full-text search using proximity ranking
  • semantic search via HNSW + Cosine distance
  • integrated basic text embedding service (Python HTTP API around a sentence transformer)
  • Reciprocal Rank Fusion for merging full-text + semantic search results
  • in-memory serving + disk persistence
  • fault-tolerance with segment replication using Raft

What it's not:

  • production-ready (code is pretty sus right now)
  • perfect (because enemy of good)

Architecture

Getting started

Install the dependencies for the basic text embedding service in third_party folder using pip.

pip install -r requirements.txt

Then start the service like so:

uvicorn main:app

Set an environment variable EmbeddingHost which points to the address of the embedding service

export EmbeddingHost="http://127.0.0.1:8000/embeddings"

Proceed to start instance(s) of the vector db

flags
  • httpAddr: address of HTTP API service
  • joinAddr: HTTP API service address of primary node to join
  • nodeId: unique identifier for node
  • raftAddr: raft address for node
Run single-node
go run cmd/server/main.go -httpAddr 127.0.0.1:8111 -nodeId 0 -raftAddr 127.0.0.1:9000

API

POST /index

index a document

curl --location '127.0.0.1:8111/index' --header 'Content-Type: application/json' --data '{"text": "some text"}'
GET /search

do a search

curl --location --request GET '127.0.0.1:8111/search' \
--header 'Content-Type: application/json' \
--data '{"query": "some text"}'
Run 3-node cluster

Run the commands below on different machines (at least different instances of the project to simulate)

go run cmd/server/main.go -httpAddr 127.0.0.1:8111 -nodeId 0 -raftAddr 127.0.0.1:9000

Replicas join the primary on 127.0.0.1:9000

go run cmd/server/main.go -httpAddr 127.0.0.1:8112 -nodeId 1 -raftAddr 127.0.0.1:9001 -joinAddr 127.0.0.1:8111
go run cmd/server/main.go -httpAddr 127.0.0.1:8113 -nodeId 2 -raftAddr 127.0.0.1:9002 -joinAddr 127.0.0.1:8111

TODO

  • Indexing
    • Concurrent indexing using goroutines to process terms
  • Retrieval
    • Boolean queries
    • Concurrent memtable search
  • Ranking
  • API
    • Bulk index
    • Document deletion
  • Storage
    • Segment compaction
  • Replication
    • Snapshot working?
  • Deployment
    • Containerisation
  • Code quality
    • Penance for all the atrocities I committed.

*Would not have been possible without these resources:

About

Vector Database implemented in Golang with support for full-text and vector search as well as fault tolerance via Raft.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages