Skip to content

v0.1.0 — first public cut

Choose a tag to compare

View all tags
@nazmiefearmutcu nazmiefearmutcu released this 27 May 13:04
v0.1.0
dfe9228

awareness ingests the public web (Common Crawl + HF FineWeb + RSS + GDELT) onto your laptop and makes it queryable from DuckDB. One Python process — no Spark, no cloud.

Install (Python 3.11+)

pip install awareness-0.1.0-py3-none-any.whl       # plain
pipx install awareness-0.1.0-py3-none-any.whl       # isolated CLI
uv tool install awareness-0.1.0-py3-none-any.whl    # with uv

Quickstart

awareness --help        # all commands
awareness-api           # local web UI + API
awareness-worker        # run the ingest worker

Ingested rows land in a local DuckDB file you can query directly with SQL.

Assets 3
Loading