Skip to content

Scraper: Filter out non-coffee products (cajik, tea, equipment, gift cards from discovery) #143

Description

@peterhadac

Problem

The scraper currently discovers and extracts every link it finds on a roaster's listing page. Unfortunately, many roasters sell non-coffee items alongside their coffee:

  • cajik / čaj - herbal teas, fruit infusions, black/green tea
  • Coffee equipment - brewers, grinders, scales, towels
  • Gift cards / voucher codes
  • Chocolate / confectionery - some roasters sell chocolate bars
  • Flavorings / syrups - vanilla, caramel syrups for coffee

These clutter the table and make the data harder to navigate. 'cajik' entries appear in scrape results because the roaster's product grid mixes them with coffee - they have a valid origin and packaging but aren't coffee.

Proposed

1. Tighten product-link detection

Add a negative keyword list to the product-link discovery filter to block obviously non-coffee pages from being sent to the LLM:

  • Czech/Slovak: cajík, cajíky, čaj, doplnok, darceká karta, zdravotencky, pytel, mlynček
  • English: tea, gift card, voucher, syrup, equipment, brewer, towel, chocolate

Match against the URL slug/path segment AND the listing page link text when available.

2. LLM decline fallback (already partially working)

The extract_product tool is optional - the model can decline. But this wastes an LLM call.

3. Post-extraction sanity check

If the extracted product name strongly suggests non-coffee (e.g. 'green tea matcha', 'ceramic pour-over'), mark as not_a_product.

Implementation notes

  • Start with a simple blocklist of keywords in a new filter function near the top of the discovery loop.
  • Track how many items are filtered - should reduce unnecessary LLM calls and clean up product listings.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions