v1.5.0

Highlights

• First-class MCP support for AI agents: exposes TRAWL through Streamable HTTP with tools for reading pages, scraping protected URLs, capturing screenshots, and inspecting browser activity
• Agent-ready structured web content: extracts readable text, links, metadata, screenshots, console output, network activity, redirect chains, and selected response bodies
• Expanded anti-bot coverage: adds detection and resolution for DataDome and AWS WAF challenges, including deterministic escalation to headful browser sessions
• Stronger proxy guarantees: enforces strict per-request proxy routing, preserves multiple Set-Cookie headers, supports opting out of Tier 0, and generates certificates accepted by strict OpenSSL clients
• More reliable deployments: reconnects Redis after startup failures, waits for Redis readiness in Compose, reaps orphaned browser processes with Tini, and includes FFmpeg in runtime images
• More resilient container builds: tracks rolling GeoLite releases and optionally retains OS font bundles required by spoofed browser fingerprints
• Updated runtime stack: upgrades TRAWL packages and dependencies to their latest mutually compatible versions

Breaking changes

• Environment variables have been standardized and legacy aliases removed. Existing deployments using renamed variables must update their configuration before upgrading.
• The unused CHROME_EXECUTABLE setting has been removed.
• An empty or unset REDIS_URL now explicitly disables the Redis-backed cache.

See the configuration migration guide for the complete list of renamed variables and migration instructions.

Major changes

• 4d4d2d5 expands the MCP server with agent-focused read, scrape, screenshot, and inspect tools (#116)
• 9239e3b adds the original Streamable HTTP MCP server and generic AI integration (#104, closes #67)
• c32d110, 3a01f86, and 043ce19 add bounded response-body capture, browser diagnostics, redirect tracking, and viewport screenshots (#106, #107, and #109). Thanks @emandel2630
• 0e8d93e adds first-class DataDome challenge detection and resolution (#84). Thanks @CappyT
• 0a18827 and fcdaef3 add AWS WAF challenge detection, escalation, and a dedicated waiter (#82, based on #65). Thanks @nandyalu
• 6d85015 enforces strict per-request proxy routing and prevents silently successful responses from broken proxy paths (#86, closes #73)
• 898cf58 preserves and serializes every upstream Set-Cookie header (#81, based on #64). Thanks @nandyalu
• f4330db introduces MITM_ALWAYS_SCRAPE for bypassing Tier 0 when browser processing is required (#111, closes #93)
• 7180b76 adds SKI and AKI identifiers to generated MITM certificates for strict OpenSSL compatibility (#115, closes #113). Thanks @jespo2021 for the report
• a7357ba restores Redis connectivity after startup timeouts instead of permanently disabling Tier 2 (#112, closes #92)
• 70235db adds Tini to reap orphaned Camoufox subprocesses (#80, closes #79). Thanks @alekc for the report
• 5775255 installs FFmpeg in the baseline runtime image (#95). Thanks @tomkor
• 75c8bf7 preserves spoofed OS font bundles behind an explicit build argument (#101, closes #97). Thanks @emandel2630
• 80f3e67 replaces fragile GeoLite version pins with the rolling release (#105, closes #98). Thanks @emandel2630
• 7ecfe7e standardizes runtime environment variable naming and removes ambiguous legacy aliases (#114)
• c81d628 gates all container image publishing workflows on successful CI verification
• 50a0e95 prepares the v1.5.0 release, aligns package versions, and updates the project documentation