Generalized Product Search

This project searches configured, permitted product sources from one natural-language request and returns normalized, comparable results. It replaces the previous single-site, hard-coded scraper.

The product scope and delivery criteria are in goal.md.

What it does

• Parses a user query into category, budget, region, and simple product constraints.
• Uses a small versioned Naive Bayes classifier with confidence-aware fallback to all general sources.
• Searches source adapters concurrently and keeps one source failure from failing the complete query.
• Normalizes JSON-LD Product data, deduplicates listings, and ranks matches.
• Uses local, environment-selected configuration; no retailer or credentials are enabled by default.
• Can append normalized query runs to a JSONL file for debugging and regression analysis.

Requirements

• Python 3.11+
• pip install -r requirements.txt to run the test suite

Configure permitted sources

Copy sources.example.json to a local sources.json, then replace the placeholder source URLs with sources whose API, feed, or website access terms allow this use. Enable the desired entries.

$env:PRODUCT_SEARCH_SOURCES_PATH = "sources.json"
python main.py "quiet 6-gallon air compressor under $250 available in Canada"

The only built-in web adapter reads application/ld+json product data from a configured search page. It does not bypass authentication, bot protection, or access restrictions. For sources without suitable JSON-LD, add a separate adapter behind SourceAdapter and cover it with saved fixtures and tests.

Restrict sources when needed:

python main.py "15 inch laptop under 800 CAD" --source permitted-retailer-one
python main.py "storage tote" --exclude-source permitted-retailer-two --record runs.jsonl

The CLI prints JSON. It reports source failures in errors while returning successful results from other sources.

Development

python -m pytest --basetemp .pytest_tmp -q
python -m product_search.evaluate
python -m compileall -q product_search main.py

The test suite uses saved HTML fixtures and never contacts live retailer pages.

Adding a source

Implement SourceAdapter.search(intent) -> list[ProductResult], configure category support, and add fixtures that cover normal results, malformed data, and source failure. The core search service should not need changes.