Summary
When the body-visibility wait times out, nothing is logged. The crawl reports success=True with an empty error_message, and 30 seconds vanish with no indication of where they went.
Follow-up to #2129 / #2131. That PR added CrawlerRunConfig.body_visibility_timeout so callers can lower the ceiling — but there is currently no way for a user to discover they need it.
Why this matters
csp_compliant_wait returns False on timeout rather than raising (crawl4ai/async_crawler_strategy.py:335-342). On the default path the return value is discarded, because ignore_body_visibility defaults to True:
is_visible = await self.csp_compliant_wait(page, """...""", timeout=config.body_visibility_timeout)
if not is_visible and not config.ignore_body_visibility: # never taken by default
...So the timeout is completely silent. @mvletter had to instrument the pipeline to attribute the delay — that is a high bar for what turns out to be a one-line config change.
Measured on develop
Two identical pages differing only by ng-cloak on <body>:
visible body 0.2s success=True words=362
hidden body 30.3s success=True words=362
Instrumented:
total crawl: 30.4s
csp_compliant_wait: 30.1s (timeout=30000, returned False)
ignore_body_visibility: True -> wait result is DISCARDED
crawl reported success: True, error_message=''
Same content, 150x slower, no signal of any kind.
Suggested fix
Log a warning (or debug, if warning is judged too noisy) when the body-visibility wait times out, naming the option:
Body never became visible after 30000ms — the page may use ng-cloak/v-cloak. Lower body_visibility_timeout to speed up crawls of this site.
Scope is small: one branch at the csp_compliant_wait call site in _crawl_web (crawl4ai/async_crawler_strategy.py:815-830). No behavior change, no new config.
Note
Lowering body_visibility_timeout does not help on pages served with a CSP sandbox directive. The timeout is enforced by a JS polling loop inside the page (async_crawler_strategy.py:315-332), and sandbox disables timers, so the loop's clock never advances — same class of problem as #2139. Requires both a sandboxed page and a hidden body, so it is narrow, but a log line would surface that case too.
Summary
When the body-visibility wait times out, nothing is logged. The crawl reports
success=Truewith an emptyerror_message, and 30 seconds vanish with no indication of where they went.Follow-up to #2129 / #2131. That PR added
CrawlerRunConfig.body_visibility_timeoutso callers can lower the ceiling — but there is currently no way for a user to discover they need it.Why this matters
csp_compliant_waitreturnsFalseon timeout rather than raising (crawl4ai/async_crawler_strategy.py:335-342). On the default path the return value is discarded, becauseignore_body_visibilitydefaults toTrue:So the timeout is completely silent. @mvletter had to instrument the pipeline to attribute the delay — that is a high bar for what turns out to be a one-line config change.
Measured on
developTwo identical pages differing only by
ng-cloakon<body>:Instrumented:
Same content, 150x slower, no signal of any kind.
Suggested fix
Log a warning (or debug, if warning is judged too noisy) when the body-visibility wait times out, naming the option:
Scope is small: one branch at the
csp_compliant_waitcall site in_crawl_web(crawl4ai/async_crawler_strategy.py:815-830). No behavior change, no new config.Note
Lowering
body_visibility_timeoutdoes not help on pages served with a CSPsandboxdirective. The timeout is enforced by a JS polling loop inside the page (async_crawler_strategy.py:315-332), andsandboxdisables timers, so the loop's clock never advances — same class of problem as #2139. Requires both a sandboxed page and a hidden body, so it is narrow, but a log line would surface that case too.