release: Crawl4AI v0.9.1#2061
Merged
Merged
Conversation
#1894) page_timeout is in milliseconds (Playwright convention, default 60000ms), but AsyncHTTPCrawlerStrategy passed it directly to aiohttp.ClientTimeout which expects seconds. This made the default timeout 60,000 seconds (16.7 hours) instead of 60 seconds, effectively disabling timeouts for HTTP mode. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PruningContentFilter's density-based scoring strips short metadata elements (author names, timestamps, attribution) alongside actual boilerplate. Add opt-in whitelist params so users can protect specific CSS classes or HTML tags from pruning. - preserve_classes: set of CSS class names to always keep - preserve_tags: set of HTML tag names to always keep - Whitelisted nodes skip scoring entirely (score = always keep) - Default: empty sets — no behavior change for existing users Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
fix(docker): make read-only tmpfs writable
…2037) The AuthGateMiddleware blocked UI static pages with 401 because browsers cannot attach Authorization headers to top-level navigation. The UI shell serves no data, so it is safe to load without credentials. - Add public_prefixes to AuthGateMiddleware for prefix-based path bypass - Register /dashboard, /playground, /static as public prefixes - Add token input bar to both playground and dashboard UIs - Replace all bare fetch() calls with authFetch() that attaches Bearer token - Append ?token= to monitor WebSocket URL (gate already accepts it for WS) All API/data routes remain fail-closed behind the auth gate. Closes #2037
fix: close browser contexts from snapshot
adaptive_crawler copy.py was an uncommitted editor artifact that ended up tracked in the repo. It is byte-for-byte identical to adaptive_crawler.py and is not imported anywhere.
Two unreachable functions in utils.py: - The first `normalize_url` (plain urljoin wrapper) was silently shadowed by the extended `normalize_url` defined ~20 lines later. Python last-write wins, so the first definition was never callable. - `normalize_url_tmp` was a hand-rolled URL joiner (string split on "/") with no callers outside utils.py itself. `urllib.parse.urljoin` already covers this correctly.
fix: remove dead code from utils and adaptive_crawler
fix(docker): authenticate rate limit Redis storage
- supervisord: write pidfile to /tmp (tmpfs) instead of read-only /app - redis: set working dir to /var/lib/redis (tmpfs) for RDB snapshots Fixes "could not write pidfile" and "Failed opening temp RDB file" errors in read-only root filesystem deployment (docker-compose.yml security hardening). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…rvisord-redis-dirs fix(docker): use writable directories for supervisord pid and redis data
BrowserConfig.chrome_channel defaults to 'chromium'. When passed to playwright.chromium.launch() as channel='chromium', Playwright looks for a system Chrome installation instead of using the bundled ms-playwright binary. On Windows this causes Chrome to exit immediately with code 0, resulting in: TargetClosedError: BrowserType.launch: Target page, context or browser has been closed Skip passing channel when it equals the default 'chromium' value.
fix: skip channel chromium on Windows to prevent TargetClosedError
The auth gate fix (cbf5e5c) intentionally makes UI shell pages load without credentials. Update the posture-gate tests to match: move /dashboard/ and /playground/ from PROTECTED_ENDPOINTS to a new PUBLIC_UI_PATHS list with an explicit 200 assertion.
fix(docker): let dashboard/playground load when auth gate is active
fix(docker): cap FastAPI below 0.137
Relax lxml~=5.3 (>=5.3,<6) to >=5.3,<7 so crawl4ai can co-install with packages requiring lxml 6.x (e.g. scrapling). Crawl4ai only uses stable lxml APIs (etree, html, fromstring, CSSSelector) unchanged in 6.x.
fix: stabilize best-first batch ordering
…-whitelist-1900 feat: add preserve_classes/preserve_tags whitelist to PruningContentFilter (#1900)
…t-mismatch-1894 fix: convert page_timeout ms to seconds for aiohttp (#1894)
Bump version to 0.9.1 across all references. Add release notes, demo verification script, and doc updates. Key highlights: - PruningContentFilter preserve_classes/preserve_tags whitelist - 12 bug fixes across Docker, browser, core, and extraction - lxml ceiling widened to allow 6.x
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
preserve_classes/preserve_tagsparameters to protect specific elements from density-based pruning ([Bug]: md-fit strips meaningful content metadata (usernames, attribution) not just boilerplate #1900)Issues fixed
#1894, #1900, #1998, #1999, #2007, #2019, #2025, #2027, #2037, #2040, #2047, #2051
Test plan
docs/releases_review/demo_v0.9.1.py) — 6/6 testspip install -e .andcrawl4ai-doctorpass