Service Crawl4ai

5.2.12. Crawl4AI

1. Overview

Crawl4AI is a local, token-protected web extraction API. It renders pages in a browser so Atlas services can summarize, ingest or index them. Atlas pins unclecode/crawl4ai:0.9.0. Its API is secure by default: every endpoint except GET /health requires CRAWL4AI_API_TOKEN.

2. Access

• Direct URL: http://localhost:${CRAWL4AI_PORT}
• Kong URL: http://crawl4ai.localhost:${KONG_HTTP_PORT}
• Internal URL: http://crawl4ai:11235

The Kong route exists only when CRAWL4AI_SOURCE=container. Kong adds the dashboard-user basic-auth, acl and cors plugins, and Crawl4AI still requires its bearer token. The two credentials cannot share the Authorization header. Through Kong, send the dashboard credential as Proxy-Authorization, which Kong's basic-auth reads.

Run this Bash example from the repository root with uv and the Bootstrapper dependencies installed. Use an existing configuration for the running stack, with its credentials already generated.

The Atlas parser reads .env, or the file selected by ATLAS_ENV_FILE. Keep that selection consistent with the running stack. Relative selections resolve from the repository root. The selected file must exist and contain usable DASHBOARD_USERNAME, DASHBOARD_PASSWORD, CRAWL4AI_API_TOKEN and KONG_HTTP_PORT values.

All four values must be configured for this request. Missing, empty or whitespace-only values stop the example before curl. The loader preserves usable bytes and replaces inherited shell values. Repeat the setup after configuration changes.

if ! crawl4ai_values="$(PYTHONPATH=bootstrapper uv run --project bootstrapper python - <<'PYTHON'
import base64
import shlex
from core.config_parser import ConfigParser

parser = ConfigParser(root_dir=".")
if not parser.env_file_path.is_file():
    raise SystemExit("The selected Atlas environment file must exist.")
values = parser.parse_env_file()
for name in ("DASHBOARD_USERNAME", "DASHBOARD_PASSWORD", "CRAWL4AI_API_TOKEN", "KONG_HTTP_PORT"):
    if not values.get(name, "").strip():
        raise SystemExit(f"{name} must be set in the selected Atlas environment file.")
basic = (values["DASHBOARD_USERNAME"] + ":" + values["DASHBOARD_PASSWORD"]).encode("utf-8")
exports = {
    "CRAWL4AI_BASIC_AUTH": base64.b64encode(basic).decode("ascii"),
    "CRAWL4AI_API_TOKEN": values["CRAWL4AI_API_TOKEN"],
    "KONG_HTTP_PORT": values["KONG_HTTP_PORT"],
}
for name, value in exports.items():
    print(f"export {name}={shlex.quote(value)}")
PYTHON
)"; then
  exit 1
fi
eval "$crawl4ai_values"
curl -H "Proxy-Authorization: Basic $CRAWL4AI_BASIC_AUTH" \
     -H "Authorization: Bearer $CRAWL4AI_API_TOKEN" \
     "http://crawl4ai.localhost:${KONG_HTTP_PORT}/health"

The service permits /health without bearer authentication. This call checks the health route; it does not verify browser extraction or bearer acceptance.

3. Configuration

• CRAWL4AI_SOURCE=disabled|container controls whether the service runs. The default is disabled.
• CRAWL4AI_IMAGE=unclecode/crawl4ai:0.9.0 pins the upstream image.
• CRAWL4AI_API_TOKEN is generated by ./start.sh and should be treated as a secret.
• CRAWL4AI_TIMEOUT_SECONDS and CRAWL4AI_MAX_CHARS tune Local Deep Researcher extraction calls. The timeout must be finite and greater than zero. The character limit must be a nonnegative integer. Invalid values keep search snippets.
• CRAWL4AI_ALLOW_INTERNAL_URLS=false makes Crawl4AI reject a crawl URL with HTTP 400 URL blocked (SSRF protection). This applies when the host is localhost or host.docker.internal, or resolves to a non-public address. Atlas container names resolve to private Docker addresses, so they are blocked too. Keep it false unless you intend to crawl internal hosts.

4. Architecture & wiring

Local Deep Researcher uses Crawl4AI when LOCAL_DEEP_RESEARCHER_FULL_PAGE_MODE=crawl4ai. That mode sets FETCH_FULL_PAGE=true and patches the upstream fetch_raw_content() helper at container start.

• Each URL goes to one synchronous POST /crawl call.
• The first result's markdown, cut to CRAWL4AI_MAX_CHARS, becomes raw_content.
• A failed URL returns None, so one fetch problem does not abort the research loop.
• Startup fails if this mode is set while CRAWL4AI_SOURCE is not container.

n8n receives CRAWL4AI_ENDPOINT and CRAWL4AI_API_TOKEN for HTTP Request nodes. Atlas does not add a custom n8n community node for Crawl4AI.

5. Capabilities & limitations

Support tier: experimental — Capability contract declared; no cited cold-start, workflow, or upgrade qualification run yet (evidence at v0.1.0).

Capability	Status	Verification	Notes
Browser-backed web extraction	supported	tested	Atlas configures Crawl4AI's JavaScript-capable API and a Local Deep Researcher adapter that submits URLs and retains bounded markdown output.
Layered extraction-route authentication	supported	tested	The direct and internal APIs require the generated Crawl4AI bearer token, while the Kong alias adds dashboard Basic authentication and ACL; only health is public.
Conservative crawl isolation	supported	tested	Internal-URL access defaults off and the container is read-only with dropped capabilities, no-new-privileges, bounded processes, and ephemeral working directories.
Curated MCP package registration	not-supported	documented	The upstream server exposes MCP transports, but Atlas does not register them in its curated MCP package or provide transport-specific discovery.

6. MCP

Crawl4AI 0.9.0 exposes MCP endpoints at /mcp/sse, /mcp/ws and /mcp/schema on the same token-protected server. Atlas does not register them in the curated MCP package.

7. Troubleshooting

• 401 Unauthorized: include Authorization: Bearer ${CRAWL4AI_API_TOKEN}.
• Kong route missing: confirm CRAWL4AI_SOURCE=container and rerun ./start.sh.
• Local Deep Researcher still using snippets: set LOCAL_DEEP_RESEARCHER_FULL_PAGE_MODE=crawl4ai, ensure CRAWL4AI_SOURCE=container, and restart the stack.

8. Dependencies & Integrations

8.1. Current — Upstream (this service calls)

No upstream calls.

8.2. Current — Downstream (services that call this)

Service	Category
kong	infra
n8n	agents
local-deep-researcher	apps

8.3. Architecture diagram

Open the full-size diagram for a full-screen view.

8.4. Future — Missing pair integrations

No high-confidence opportunities identified.

8.5. Future — Candidate new services

No high-confidence opportunities identified.

8.6. Future — Unused features in this service

No high-confidence opportunities identified.