Skip to content

v0.3.0 — Darimi, embeddings, semantic search, dedup

Latest
Latest

Choose a tag to compare

View all tags
@affanhamid affanhamid released this 27 Apr 04:29
v0.3.0
d6c66d5

What's new since v0.2.0

New collection

  • Sunan ad-Darimi — 2,757 hadiths scraped from sunnah.com, machine-translated to English via Gemini Flash (~30k → 39k total hadiths across 8 collections).

Vector search

  • gemini-embedding-001 (768-dim) embeddings for every hadith, indexed via pgvector HNSW.
  • /api/search/semantic?q=... — cosine similarity search; finds hadiths by meaning ("fasting on Mondays" surfaces relevant Ramadan/voluntary-fast hadiths even without exact word matches).
  • Keyword/Semantic toggle in the UI with per-result relevance score.

Narrator data quality

  • Dedup pass: 84 strict-match narrator merges with audit table for rollback.
  • Ibn Abbas / al-Abbas fix: 2,313 chain rows re-pointed to the correct sahabi. Top-10 sahaba now matches classical history (Abu Hurayra → Aisha → Ibn Abbas).
  • Reliability scoring: 10-rank Ibn Hajar marātib via Arabic keyword matching; per-narrator aggregate 0–1 score shown as HSL gradient.

Hadith classification

  • All 39,058 hadiths now tagged with hadith_type (marfu / mawquf / maqtu / mursal / balagh / fatwa) via heuristic Arabic-text scanner.
  • Edit UI: click the type badge in the detail panel to correct misclassifications.

Schema additions

  • hadiths: hadith_type, slug (e.g. Bu.1.1, Ma.4.2, Da.1.650), external_ref, embedding vector(768)
  • narrators: reliability_score
  • jarh_tadil.evaluation_type expanded from 3 → 10 ranks
  • New tables: narrator_merges, chain_refixes (audit logs)

Database asset

hadith_db.sql.gz (136 MB) — full Postgres dump. Restore with:

createdb hadith
gunzip -c hadith_db.sql.gz | psql -d hadith

Note: the embedding column requires the pgvector extension installed beforehand:

sudo apt install postgresql-16-pgvector
sudo -u postgres psql -d hadith -c "CREATE EXTENSION vector;"

Scale

  • 11,479 narrators (Tahdhib al-Kamal)
  • 39,058 hadiths across 8 books (Kutub al-Sittah + Muwatta + Darimi)
  • 175,818 chain links, ~95% matched to narrators
  • 28,159 jarh/ta'dil evaluations across 7,528 narrators
  • 39,058 hadiths × 768-dim embeddings

Co-Authored-By: Claude Opus 4.7 (1M context)

Assets 3
Loading