AI Opportunity Discovery System

AODS extracts, clusters, and ranks recurring monetizable problems from public sources using local ML for scale, local-first LLMs for audited semantic enrichment, and deterministic scoring for decisions. The current V1 stack supports Reddit, Hacker News, Indie Hackers, and Product Hunt ingestion with inspectable intermediate artifacts.

The repo now uses a local-first LLM contract for semantic tasks. The default runtime decision is Ollama, with qwen3:8b as the primary model and gemma3:4b retained as the lighter fallback profile.

Local Run

One-Command Start

From the repository root:

npm start

This prepares missing .env files from the checked-in examples, starts Ollama when the local provider is configured and not already reachable, verifies configured Ollama models, starts the Docker Compose stack, builds missing images only when needed, applies Alembic migrations, waits for backend and frontend readiness, and prints the local URLs.

Prerequisites:

• Docker Desktop installed. The start script attempts to launch Docker Desktop when the engine is not already running.
• Node/npm available for the root script runner.
• Ollama installed on the host. The start script attempts to launch ollama serve when LLM_PROVIDER=ollama and http://127.0.0.1:11434 is not already reachable. Configured models such as qwen3:8b and gemma3:4b must already be pulled.

The app will be ready at http://127.0.0.1:5173.

Useful root commands:

npm start
npm run rebuild
npm run status
npm stop

Use npm run rebuild after Dockerfile, dependency, or Python service changes. Backend, worker, and scheduler all use one shared infra-python-app:latest image so the large Python/ML runtime is stored once. Normal npm start intentionally avoids --build when images already exist so DockerData does not accumulate large BuildKit cache on every app start.

Using the App

Open http://127.0.0.1:5173 after npm start reports that AODS is ready. The main screen is an operator work surface with source controls on the left and an Opportunity inbox on the Dashboard.

Basic workflow:

1. Choose an Opportunity area from the dropdown in Source controls. The presets fill in starter source queries; use Custom only when you already know the exact scope you want.
2. Click Discover seeds to have AODS create and score source-registry seed candidates for the selected area across the supported ingestion sources.
3. Review discovered seeds in the registry tab. Suggested seeds stay inspectable before they are used by registry-backed ingestion.
4. Click a source ingest button:
   • Ingest for Reddit with the comma-separated subreddit list.
   • HN ingest, IH ingest, or PH ingest for Hacker News, Indie Hackers, or Product Hunt.
   • Reddit registry ingest for Reddit seeds from the active source registry snapshot.
5. Wait for the new pipeline run to appear in Dashboard or Admin run history.
6. Click Run M1 on the latest run to execute embeddings, dedupe, sampling, clustering, deterministic scoring, and output stages.
7. Inspect results in the right-side tabs:
   • dashboard shows the workflow, metrics, and a sortable Opportunity inbox. Click a row to expand AI descriptions, score details, and evidence posts. Rows marked Writeup have audited final LLM output, rows marked Summary have an audited cluster summary but no final writeup yet, and rows marked Pending are deterministically scored clusters waiting for semantic output.
   • clusters shows scored clusters and score breakdowns.
   • raw data pages through persisted source records for the selected opportunity area and supports sorting by source, date, engagement, and duplicate status.
   • registry bootstraps/reviews source seeds, queues registry-backed ingestion, inspects seed/source quality history, and supports sorting seed rows.
   • evaluations creates and compares persisted evaluation runs and label sets.
   • admin shows runtime config, score distributions, threshold profiles, backtests, and pipeline run history.

The API is available at http://127.0.0.1:8000. Use http://127.0.0.1:8000/health for a quick backend readiness check.

Opportunity rows come from scored clusters first. The frequency value is the count of deduplicated raw posts assigned to that cluster in the selected run; AI writeups and summaries are attached when present through ai_status, but missing AI output no longer hides a scored opportunity from /api/opportunities.

Command Reference

Command	What it does
npm start	Starts Docker Desktop and host Ollama when needed, verifies configured Ollama models, starts the AODS stack, creates missing env files, builds only missing Docker images, applies Alembic migrations, waits for backend/frontend readiness, and prints URLs.
npm run rebuild	Forces a Docker Compose rebuild before startup; use after Dockerfile, dependency, or service-image changes.
npm run status	Shows current Docker Compose service status for the AODS stack.
npm run storage	Reports DockerData VHDX size plus visible Docker image/container/volume/cache usage.
npm stop	Stops and removes AODS Compose containers and network while preserving volumes such as infra_postgres_data.
npm run cleanup	Stops the AODS stack, prunes Docker build cache while preserving images/volumes, enables sparse DockerData reclamation when supported, trims free blocks inside DockerData, shuts down Docker Desktop/WSL, and compacts Docker's WSL VHDX.
docker compose -f infra\docker-compose.yml up -d	Starts Compose services using existing images without forcing a rebuild.
docker compose -f infra\docker-compose.yml up -d --build	Starts Compose services and forces image builds; prefer npm run rebuild for the standard workflow.
docker compose -f infra\docker-compose.yml down	Stops/removes Compose containers and network; does not delete named volumes.
docker compose -f infra\docker-compose.yml ps	Lists AODS service containers and health/status.
docker compose -f infra\docker-compose.yml exec backend alembic -c alembic.ini upgrade head	Applies database migrations inside the backend container.
docker builder prune -af	Removes Docker BuildKit cache; keeps images, containers, and volumes. Useful after rebuilds if DockerData grew.
docker image prune -af	Removes unused images; next start may need to rebuild/pull images. Does not delete volumes.
.\scripts\compact_dockerdata.ps1	Direct script form of npm run cleanup; pass -PruneImages only when intentionally deleting unused images too.
.\scripts\docker_storage_status.ps1	Direct script form of npm run storage.

Manual Docker

1. Copy environment examples:

Copy-Item infra\.env.example .env
Copy-Item frontend\.env.example frontend\.env

2. Start services:

docker compose -f infra\docker-compose.yml up --build

For routine starts after images already exist, prefer:

docker compose -f infra\docker-compose.yml up -d

The local worker runs Celery with --concurrency=1 so the Docker stack can share host memory with Ollama running qwen3:8b.

3. Apply migrations:

docker compose -f infra\docker-compose.yml exec backend alembic upgrade head

4. Open the app at http://localhost:5173.

Useful checks:

docker compose -f infra\docker-compose.yml ps
Invoke-RestMethod -Uri http://127.0.0.1:8000/health
Invoke-RestMethod -Uri http://127.0.0.1:8000/api/admin/debug-summary
Invoke-RestMethod -Uri http://127.0.0.1:8000/api/admin/runtime-config
Invoke-RestMethod -Uri "http://127.0.0.1:8000/api/admin/score-distribution?niche=founder%20operations"
Invoke-RestMethod -Uri "http://127.0.0.1:8000/api/admin/score-distribution?niche=founder%20operations&alert_threshold=2.5&high_saturation_threshold=3.0"
Invoke-RestMethod -Uri http://127.0.0.1:8000/api/admin/threshold-profiles

Small smoke ingestion:

$body = @{
  niche = "smoke-test"
  subreddits = @("Entrepreneur")
  query = "problem"
  limit_per_subreddit = 2
} | ConvertTo-Json

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/ingestion/reddit -Method Post -Body $body -ContentType "application/json"

Hacker News ingestion:

$body = @{
  niche = "developer tools"
  query = "debugging workflow"
  limit = 20
} | ConvertTo-Json

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/ingestion/hacker-news -Method Post -Body $body -ContentType "application/json"

Registry-backed Hacker News ingestion uses active hacker_news query seeds:

$body = @{
  niche = "developer tools"
  limit = 20
} | ConvertTo-Json

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/ingestion/hacker-news/from-registry -Method Post -Body $body -ContentType "application/json"

Indie Hackers ingestion uses the public Indie Hackers RSS feed bridge and accepts either a search query, a group slug, or both:

$body = @{
  niche = "founder operations"
  query = "workflow"
  group = "growth"
  limit = 20
} | ConvertTo-Json

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/ingestion/indie-hackers -Method Post -Body $body -ContentType "application/json"

Registry-backed Indie Hackers ingestion uses active indie_hackers query, tag_query, search_query, or tag seeds:

$body = @{
  niche = "founder operations"
  limit = 20
} | ConvertTo-Json

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/ingestion/indie-hackers/from-registry -Method Post -Body $body -ContentType "application/json"

Product Hunt ingestion uses the public Product Hunt Atom/RSS feed and filters feed items locally when a query is supplied:

$body = @{
  niche = "founder operations"
  query = "workflow automation"
  limit = 20
} | ConvertTo-Json

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/ingestion/product-hunt -Method Post -Body $body -ContentType "application/json"

Registry-backed Product Hunt ingestion uses active product_hunt query or search_query seeds:

$body = @{
  niche = "founder operations"
  limit = 20
} | ConvertTo-Json

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/ingestion/product-hunt/from-registry -Method Post -Body $body -ContentType "application/json"

Local Backend Helpers

If Docker is unavailable but Python dependencies are installed in .venv, use:

python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r .\backend\requirements.txt
.\scripts\run_backend_tests.ps1
.\scripts\run_backend.ps1
.\scripts\run_worker.ps1

The local scripts set PYTHONPATH to include both backend/ and the repo root so the FastAPI app and worker package resolve consistently.

The backend still expects PostgreSQL + pgvector and Redis for DB-backed APIs and Celery execution. Use Docker Compose or local services on ports 5432 and 6379. For local semantic tasks, run Ollama locally and make it available at OLLAMA_BASE_URL, then pull the configured models such as qwen3:8b and gemma3:4b.

Source Registry

Bootstrap and inspect the source registry through the backend API:

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/source-registry/bootstrap -Method Post
Invoke-RestMethod -Uri http://127.0.0.1:8000/api/source-registry/sources
Invoke-RestMethod -Uri "http://127.0.0.1:8000/api/source-registry/seeds?niche=founder%20operations"
Invoke-RestMethod -Uri "http://127.0.0.1:8000/api/source-registry/quality-summary?niche=founder%20operations"
Invoke-RestMethod -Uri "http://127.0.0.1:8000/api/source-registry/quality-trends?niche=founder%20operations"
Invoke-RestMethod -Uri "http://127.0.0.1:8000/api/source-registry/seeds/<seed-id>/score-history"
Invoke-RestMethod -Uri "http://127.0.0.1:8000/api/source-registry/seeds/<seed-id>/lifecycle-history"
Invoke-RestMethod -Uri "http://127.0.0.1:8000/api/source-registry/sources/<source-id>/lifecycle-history"
Invoke-RestMethod -Uri "http://127.0.0.1:8000/api/source-registry/sources/<source-id>/reliability-history"

The quality summary, trends, and score history are deterministic. They report latest seed score coverage, status/source counts, per-source average latest score, per-source low-quality counts, daily source score trends, low-scoring non-pinned seeds, unscored seeds, and per-seed score trends without using LLM output for scoring or ranking. Seed lifecycle edits append audit entries under source_seeds.seed_metadata.lifecycle_history and can be inspected through the lifecycle-history endpoint or Registry tab. Source lifecycle edits append audit entries under source_registry.source_metadata.lifecycle_history and can be inspected through the source lifecycle-history endpoint or Registry tab. Source reliability edits are persisted on source_registry.reliability_score, append audit entries under source_metadata.reliability_history, and can be inspected through the reliability-history endpoint or Registry tab. The frontend Registry tab exposes the manual async rescore downgrade threshold before queueing seed rescoring. The Admin tab shows runtime config, including dedupe and clustering thresholds, deterministic scoring weights, local model names, and recurring rescore settings. The Admin tab also shows deterministic opportunity-score distribution by source for the selected niche, including min, average, p50, p90, max, and alert-threshold counts. The alert and saturation thresholds can be adjusted in the Admin tab or passed as alert_threshold and high_saturation_threshold query parameters to preview threshold impact without changing persisted scores or rankings. Candidate threshold settings can be saved as inspectable threshold profiles:

$body = @{
  profile_name = "founder-ops-sensitive-review"
  status = "draft"
  niche = "founder operations"
  source = $null
  alert_threshold = 2.5
  high_saturation_threshold = 3.0
  min_frequency = 5
  min_confidence = 0.6
  profile_metadata = @{
    reason = "manual admin review"
  }
} | ConvertTo-Json -Depth 5

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/admin/threshold-profiles -Method Post -Body $body -ContentType "application/json"

Backtest a saved profile against persisted deterministic scores, latest competitor saturation snapshots, and the current default alert predicate:

Invoke-RestMethod -Uri "http://127.0.0.1:8000/api/admin/threshold-profiles/<profile-id>/backtest"

The backtest response includes source-level count deltas and a bounded changed_clusters list of clusters that would newly trigger or stop triggering under the profile.

Apply an approved threshold profile to a specific pipeline run as separate profile-scoped alert events:

$body = @{
  pipeline_run_id = "<pipeline-run-id>"
} | ConvertTo-Json

Invoke-RestMethod -Uri "http://127.0.0.1:8000/api/admin/threshold-profiles/<profile-id>/apply-alerts" -Method Post -Body $body -ContentType "application/json"

Saved profiles are tuning proposals. Approving, backtesting, or applying a profile records and previews review state but does not silently change deterministic scores, rankings, or default alert execution. Applied profile alerts use alert_type = threshold_profile and retain the profile id in threshold_snapshot.

Evaluation Fixtures

Run the deterministic fixture harness without a database:

.\scripts\run_evaluation_fixtures.ps1

Use fixture-defined quality gates for CI-style regression checks:

.\scripts\run_evaluation_fixtures.ps1 --enforce-thresholds

Compare candidate fixtures against a baseline fixture directory with matching filenames:

.\scripts\run_evaluation_fixtures.ps1 --fixtures .\shared\evals --baseline-fixtures .\shared\evals --max-regression 0.01

The command prints metric deltas and exits nonzero when any metric regresses by more than --max-regression.

To persist eval runs and metric rows, apply migrations and pass --persist:

docker compose -f infra\docker-compose.yml exec backend alembic upgrade head
.\scripts\run_evaluation_fixtures.ps1 --persist

Persisted eval runs include fixture quality-gate summaries in result_summary, and the frontend Evaluations tab shows whether gates passed, failed, or were not configured.

Persisted eval runs can also be compared through the API:

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/evaluations/runs/<candidate-run-id>/compare/<baseline-run-id>

Manual labels can be persisted and reused by later live eval runs:

$body = @{
  label_type = "ranking_relevance"
  dataset_name = "manual-ranking-review"
  pipeline_run_id = "<pipeline-run-id>"
  labels = @{
    "<cluster-id>" = 2
  }
} | ConvertTo-Json -Depth 5

$labelSet = Invoke-RestMethod -Uri http://127.0.0.1:8000/api/evaluations/label-sets -Method Post -Body $body -ContentType "application/json"
Invoke-RestMethod -Uri "http://127.0.0.1:8000/api/evaluations/label-sets?label_type=ranking_relevance&pipeline_run_id=<pipeline-run-id>"

Export a portable label-set artifact and import it for another run:

$artifact = Invoke-RestMethod -Uri http://127.0.0.1:8000/api/evaluations/label-sets/$($labelSet.id)/export
$body = @{
  artifact = $artifact
  pipeline_run_id = "<target-pipeline-run-id>"
  dataset_name = "imported-ranking-review"
  label_metadata = @{
    imported_by = "manual-review"
  }
} | ConvertTo-Json -Depth 8

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/evaluations/label-sets/import -Method Post -Body $body -ContentType "application/json"

Mark a label set as reviewed, approved, rejected, or draft and filter by review status:

$body = @{
  review_status = "approved"
  reviewed_by = "manual-review"
  review_note = "labels checked against source records"
} | ConvertTo-Json

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/evaluations/label-sets/$($labelSet.id)/review-status -Method Patch -Body $body -ContentType "application/json"
Invoke-RestMethod -Uri "http://127.0.0.1:8000/api/evaluations/label-sets?review_status=approved"

Bulk-mark label sets during review:

$body = @{
  label_set_ids = @("<left-label-set-id>", "<right-label-set-id>")
  review_status = "reviewed"
  reviewed_by = "manual-review"
  review_note = "pair checked together"
} | ConvertTo-Json

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/evaluations/label-sets/review-status -Method Post -Body $body -ContentType "application/json"

Compare two label sets before reusing or approving labels:

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/evaluations/label-sets/<left-label-set-id>/diff/<right-label-set-id>

Find conflicts across saved label sets for a pipeline run:

Invoke-RestMethod -Uri "http://127.0.0.1:8000/api/evaluations/label-sets/conflicts?label_type=clustering_expected&pipeline_run_id=<pipeline-run-id>&review_status=approved"

Resolve conflicts into a new label set by supplying explicit values for disputed label ids:

$body = @{
  label_type = "clustering_expected"
  dataset_name = "resolved-clustering-review"
  pipeline_run_id = "<pipeline-run-id>"
  source_label_set_ids = @("<left-label-set-id>", "<right-label-set-id>")
  resolutions = @{
    "<raw-post-id>" = "resolved-cluster-label"
  }
} | ConvertTo-Json -Depth 6

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/evaluations/label-sets/resolve-conflicts -Method Post -Body $body -ContentType "application/json"

In the frontend conflict-resolution panel, resolution inputs are parsed as JSON when possible, so values such as 2, true, or {"domain":"ops","pain_level":4} are preserved as typed labels.

Validate that a label set points at artifacts in a pipeline run:

Invoke-RestMethod -Uri "http://127.0.0.1:8000/api/evaluations/label-sets/<label-set-id>/validate?pipeline_run_id=<pipeline-run-id>"

Create a ranking-usefulness eval from a completed pipeline run's deterministic score order by supplying explicit relevance labels:

$body = @{
  pipeline_run_id = "<pipeline-run-id>"
  dataset_name = "manual-ranking-review"
  label_set_id = $labelSet.id
  top_n = 5
  relevance_grades = @{
    "<cluster-id>" = 2
  }
} | ConvertTo-Json -Depth 5

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/evaluations/runs/from-pipeline-ranking -Method Post -Body $body -ContentType "application/json"

Create an extraction field-accuracy eval from persisted extracted_data by supplying expected labels keyed by raw post id:

$body = @{
  pipeline_run_id = "<pipeline-run-id>"
  dataset_name = "manual-extraction-review"
  fields = @("domain", "pain_level")
  expected_by_raw_post_id = @{
    "<raw-post-id>" = @{
      domain = "finance ops"
      pain_level = 4
    }
  }
} | ConvertTo-Json -Depth 6

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/evaluations/runs/from-pipeline-extraction -Method Post -Body $body -ContentType "application/json"

Create a clustering-quality eval from persisted cluster_assignments by supplying expected cluster labels keyed by raw post id:

$body = @{
  pipeline_run_id = "<pipeline-run-id>"
  dataset_name = "manual-clustering-review"
  expected_cluster_by_raw_post_id = @{
    "<raw-post-id>" = "invoice_reconciliation"
  }
} | ConvertTo-Json -Depth 5

Invoke-RestMethod -Uri http://127.0.0.1:8000/api/evaluations/runs/from-pipeline-clustering -Method Post -Body $body -ContentType "application/json"

Services

• Backend API: http://localhost:8000
• Frontend: http://localhost:5173
• PostgreSQL + pgvector: localhost:5432
• Redis: localhost:6379

Current V1 Scope

• Reddit, Hacker News, Indie Hackers, and Product Hunt source ingestion.
• Local sentence-transformer embeddings.
• Cosine-threshold deduplication.
• HDBSCAN with deterministic fallback clustering.
• Deterministic score breakdown persistence.
• Selective local-first LLM extraction, cluster summaries, final writeups, and audited llm_outputs.
• Dashboard, cluster explorer, raw data viewer, evaluations, registry review, and admin/debug summary.

The deterministic Milestone 1 stages remain valid without LLM execution, but the current V1 pipeline also includes local-first semantic stages after scoring. Scoring and ranking remain deterministic.