fix(config): make body visibility timeout configurable

[nightcityblade]

Summary

Fixes #2129.

Add CrawlerRunConfig.body_visibility_timeout, defaulting to the existing 30-second behavior, and use it for the body visibility check. This lets callers lower the ceiling for pages whose body remains hidden without changing the default crawl behavior. The value is included in config serialization and the untrusted-config allowlist, validated as a positive number, and capped at 60 seconds for untrusted Docker input.

List of files changed and why

• crawl4ai/async_configs.py - define, document, serialize, validate, allow, and cap the new configuration field.
• crawl4ai/async_crawler_strategy.py - pass the configured timeout to the body visibility wait.
• docs/md_v2/api/parameters.md and docs/md_v2/complete-sdk-reference.md - document the new public option.
• tests/test_config_defaults.py - cover the default and round trip, invalid values, the untrusted cap, and propagation into csp_compliant_wait.

How Has This Been Tested?

• /tmp/crawl4ai-followup-venv/bin/python -m pytest tests/test_config_defaults.py -q — 40 passed, 1 existing warning.
• /tmp/crawl4ai-followup-venv/bin/python -m pytest deploy/docker/tests/test_security_trust_boundary.py -q — 38 passed, 4 existing warnings.
• /tmp/crawl4ai-followup-venv/bin/ruff check --isolated --select E9,F63,F7,F82 --ignore F821 crawl4ai/async_configs.py crawl4ai/async_crawler_strategy.py tests/test_config_defaults.py — passed. (F821 is excluded because of the existing VirtualScrollConfig forward-reference finding in async_crawler_strategy.py.)
• /tmp/crawl4ai-followup-venv/bin/python -m compileall -q crawl4ai/async_configs.py crawl4ai/async_crawler_strategy.py tests/test_config_defaults.py — passed.
• git diff --check — passed.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas — N/A; no complex logic was introduced.
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[ntohidi]

@nightcityblade Thanks — approach looks right. Default stays 30000 so nobody's existing crawls change, and I checked the plumbing: from_kwargs reads the __init__ signature so that path is covered automatically, and clone/dump/load all go through to_dict, which you updated. Allowlist entry is in the right set too.

Three things before merge:

1. Docs. docs/md_v2/api/parameters.md and docs/md_v2/complete-sdk-reference.md both document ignore_body_visibility and should get the new option too. The docstring alone isn't enough.

2. The timeout goes straight into JavaScript at async_crawler_strategy.py:324. Passing None renders > None and blows up as a RuntimeError. That's pre-existing, but this PR makes it reachable from user input, including over the Docker API now that it's allowlisted. Worth a basic check on the value.

3. The test only checks the default and a dump/load round trip. Nothing verifies the value actually reaches csp_compliant_wait — the test would still pass if the call site kept using 30000. A mock asserting the timeout that gets passed would cover the real thing.

Separate from this PR: even after it lands, the default path still spends 30 silent seconds on these pages. csp_compliant_wait returns False on timeout instead of raising, so nothing is logged and the crawl reports success. @mvletter had to profile the pipeline to find it. A debug log when that wait times out would make this option discoverable — happy to do it as a follow-up rather than expand the scope here.

[nightcityblade]

Thanks for the detailed review. Addressed in 64bf678:

• documented body_visibility_timeout in both public parameter references;
• reject non-numeric, boolean, zero, and negative values, and cap untrusted Docker input at 60 seconds;
• added a mocked crawl-path test that verifies 1234 reaches csp_compliant_wait, alongside default/round-trip/validation coverage.

Focused validation passes: 40 config tests and 38 trust-boundary tests. I kept the separate timeout-debug logging idea out of this PR as suggested.

[ntohidi]

@nightcityblade thanks for your contribution :)