ARCHIVED: The search package is inlined into ftmq now

ftmq-search

Search stores logic for FollowTheMoney data.

The aim is to experiment around with different full-text search backends for efficient shallow search of entities.

Currently supported backends:

• Sqlite FTS5
• Tantivy (persistent or in-memory)

For Elasticsearch, use either openaleph-search that can be used standalone outside OpenAleph or yente.

Install

Python 3.11 or later.

pip install ftmq-search

Generate search documents

ftmqs transform -i entities.ftm.json > entities.transformed.json

Speed it up via GNU Parallel

cat entities.ftm.json | parallel -j8 --pipe --roundrobin ftmqs transform > entities.transformed.json

Index transformed documents

Sqlite FTS

ftmqs --uri sqlite:///ftmqs.store index -i entities.transformed.json

Tantivy

ftmqs --uri tantivy://tantivy.db index -i entities.transformed.json

Search

ftmqs search <query>

Autocomplete

ftmqs autocomplete <query>

Python

from ftmq import Query, smart_read_proxies

from ftmqs import get_store
from ftmqs.logic import index_proxies

# sqlite
store = get_store("sqlite:///ftmqs.db")

# tantivy
store = get_store("tantivy://tantivy.db")

# tantivy in-memory
store = get_store("memory://")

# index entity data
proxies = smart_read_proxies("./entities.ftm.json")
index_proxies(proxies, store)

# search
store.search("jane doe")

# filter for country and schema
q = Query().where(country="de", schema="Person")
store.search("jane doe", q)