Pyoogle

Pyoogle is a Python-specific search engine for discovering high-quality articles, tutorials, release notes, and technical writing from the Planet Python ecosystem. It combines live source feeds, resumable web crawling, Common Crawl history, full-text ranking, and semantic retrieval in a self-hostable application.

Built with Codex and GPT-5.6

Codex Hackathon submission: Pyoogle was built primarily in one long-running OpenAI Codex thread powered by GPT-5.6.

Codex was the primary engineering agent for the project. I used /goal to keep the central objective active across a long implementation cycle: analyze, build, measure, and continuously optimize Pyoogle's initial ingestion pipeline and background-job system. Keeping that work in mostly one thread gave Codex the architectural and operational context needed to improve the system without repeatedly rediscovering earlier decisions.

How Codex was used

• End-to-end implementation: Codex wrote all code in this repository, including the FastAPI backend, crawler and ingestion pipelines, SQL migrations, search ranking, worker orchestration, frontend, deployment configuration, maintenance commands, and automated tests.
• Goal-driven iteration: /goal was used to sustain the multi-stage effort around initial ingestion and background jobs instead of treating each optimization as an isolated prompt.
• Performance analysis: Codex inspected query plans, job leases, Common Crawl range reads, CockroachDB transaction behavior, R2 uploads, embedding work, and worker concurrency to find the actual throughput and reliability bottlenecks.
• Resumability and production hardening: Codex designed checkpointed jobs, retryable transactions, lease recovery, bounded concurrency, graceful shutdown, approval gates, and safe fallbacks so multi-hour ingestion work survives failures and deployments.
• Continuous verification: Codex repeatedly reviewed the repository, added regression tests, ran the full suite, and tightened security and deployment behavior as the system evolved.

My role was to define the product direction, provide operational constraints and live benchmark feedback, make trade-off decisions, and steer the goal. Codex and GPT-5.6 turned that direction into the working application.

Architecture at a Glance

• FastAPI backend.
• Plain HTML/CSS/JS frontend.
• pydantic-settings for all env management through pyoogle.config.Settings.
• CockroachDB for canonical SQL data, resumable jobs, full-text search, and vectors.
• Cloudflare R2/S3-compatible storage for artifacts.
• Real embeddings with fastembed.

Quick Start

You need Python 3.11 or newer, a CockroachDB database, and an S3-compatible bucket such as Cloudflare R2.

1. Install

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
cp sample.env .env

2. Configure

Fill these values in .env:

• DATABASE_URL
• S3_ENDPOINT_URL
• S3_BUCKET
• S3_ACCESS_KEY_ID
• S3_SECRET_ACCESS_KEY

The remaining settings have local defaults in sample.env. HF_TOKEN is optional but helps avoid anonymous model-download rate limits on a fresh machine.

3. Migrate and run

pyoogle migrate
pyoogle start

Open http://localhost:8000. pyoogle start runs the web app and one worker in the same process. Migrations and ingestion jobs are resumable after interruption.

The service does not enqueue crawl work on startup.

Index Your First Content

Planet Python defines Pyoogle's accepted source list. Crawl flags can narrow a test run, but they do not expand that source list.

For the full initial bootstrap:

pyoogle enqueue-initial-all

For a small, single-domain trial:

pyoogle enqueue-planet --domain blog.python.org
pyoogle enqueue-source-feeds --domain blog.python.org
pyoogle enqueue-domain-crawl --domain blog.python.org
pyoogle enqueue-commoncrawl --domain blog.python.org --limit 25

If pyoogle start is not running, process one queued batch with:

pyoogle worker-once

Unbounded Common Crawl imports estimate each domain first. Small domains are automatically allowed; high-volume shared domains wait for operator review before any broad historical fetch. See Crawling for the job flow, source policy, and review gate.

Runtime Commands

Use case	Command
Local web app and worker	pyoogle start
Production web process	pyoogle-web
Production worker process	pyoogle-worker
One finite worker batch	pyoogle worker-once
Inspect resolved settings	pyoogle config

Worker leases, retries, and checkpoints live in CockroachDB. A clean shutdown releases active leases so another worker can resume immediately. For concurrency, queue draining, maintenance, alerts, and profiling, use the Operations guide.

Search

• CockroachDB is the canonical lexical and vector search backend.
• Lexical ranking weights title, URL, lead text, and document body separately.
• Semantic retrieval searches document chunks and combines results with lexical matches.
• AI overview mode sends bounded hybrid context to the OpenAI Responses API, validates every source-number citation, and caches successful answers by query, exact sources, model, and prompt version. Leave AI_API_KEY empty to disable it.
• Manticore is an optional, rebuildable lexical serving index; search falls back to CockroachDB whenever it is incomplete or unavailable.

Implementation and ranking details live in Architecture: Search.

Docker and Deployment

Start the web app and worker with Docker Compose:

docker compose -f infra/compose.yml up --build

CockroachDB and R2 remain external services configured through .env. Production deployments normally run pyoogle-web and pyoogle-worker separately.

• Deployment guide
• Provider and service notes
• Minimum startup path
• systemd units

Documentation

• Architecture diagram — components and data flow.
• Architecture — system boundaries, persistence, and search.
• Crawling — Planet source policy, crawl jobs, and Common Crawl behavior.
• Operations — queueing, worker tuning, repairs, cleanup, and monitoring.
• Deployment — Docker, Coolify, VM setup, and production configuration.
• Contributing — local development, checks, and crawl-policy changes.