semsearch

Current status: PoC, towards prod

Goal

Semsearch is an embedding-focused indexing and search engine (ideas heavily borrowed from agentic AI RAG architecture) that aims to aggregate and promote indie blogs.

Implementation

FastAPI frontend & pgvector database

Refer to search(...) from semsearch.web.search.pipeline. Pretty self-explanatory code, hopefully.

Structure

src/semsearch/
|-- share/  # configuration, database pool, embeddings, shared utilities
|-- cli/    # Typer commands, site administration, crawling, and ingestion
`-- web/    # FastAPI application, search pipeline, and templates

Development

Running db in container and python apps on host would be easier:

uv sync
docker compose up -d db
cp .env.example .env  # set EMBEDDING_API_KEY before indexing
uv run semsearch init-db
# manually add a site. note that feed is mandatory
uv run semsearch site add https://some.blog/ --sitemap auto --feed auto
uv run semsearch site remove https://some.blog/
uv run semsearch daemon &  # long-running polling and ingestion process
uv run uvicorn semsearch.web.app:app --reload

Check with pyright, pytest, ruff and ty.

Deployment

cp .env.example .env  # See .env.example for config keys
docker compose up -d --build
docker compose exec app /app/.venv/bin/semsearch init-db  # first run only

For an embedding server on the host, use http://host.docker.internal:<port>/some-api-endpoint in .env.

Run admin commands inside the container:

docker compose exec app /app/.venv/bin/semsearch status
# if you decide to use it
docker compose exec app /app/.venv/bin/python scripts/import_indieblog_feeds.py --dry-run

Changing the chunking algorithm, embedding dimension, or model requires re-indexing. TODO