Skip to content

v1.0.1: Network binding, no telemetry, no crawler disguise

Choose a tag to compare

@grregis grregis released this 22 Oct 06:17
· 21 commits to main since this release

Added

  • APP_BIND and APP_PORT in .env choose where the admin app is
    published (default 127.0.0.1:5000, unchanged). Set APP_BIND=0.0.0.0 to
    reach it from other machines on your network, the same way MCP_BIND
    works for the MCP server; no override file needed. Suggested on Reddit by
    u/geo38.

Changed

  • Meilisearch telemetry is off (MEILI_NO_ANALYTICS). It sent anonymous
    usage data to Meilisearch by default. Spotted by u/geo38. Takes effect when
    the container is recreated (./install.sh, or
    docker compose up -d meilisearch).
  • CNN Underscored pages are refused at ingestion (/cnn-underscored/):
    they are shopping reviews with affiliate links, not news. They also answer
    403, and one of those was enough to put all of cnn.com on a six-hour
    scrape hold.

Removed

  • The scraper no longer poses as Googlebot or opens paywalled sites in a
    headless browser.
    Seven sites were fetched with Google's crawler user
    agent, and seven paywalled ones (NYT, WSJ, Washington Post, FT, Bloomberg,
    The Athletic, Wired) got a headless Chromium attempt. MuckScraper now reads
    only what an ordinary browser can, and a paywalled article stays
    headline-only. In practice little is lost: over 30 days the crawler user
    agent fetched 81 articles and the headless browser none.