Skip to content

Python Client SDK

Piergiorgio Lucidi edited this page Nov 20, 2026 · 4 revisions

OpenCrawling Python Client SDK (oc-python-client-sdk / opencrawling-sdk) 🐍

The OpenCrawling Python Client SDK (oc-python-client-sdk, published to PyPI as opencrawling-sdk, importable as opencrawling) is a modern, strictly typed (PEP 561) Python 3.11+ library offering both synchronous (OpenCrawlingClient) and asynchronous (AsyncOpenCrawlingClient) interfaces over httpx and Pydantic v2.

It delivers 1:1 parity with oc-java-client-sdk across all 80 Platform API v1 (/api/v1) operations (including native cron job scheduling and the unified /api/v1/runs API), direct Open Ingestion Standard (OIS) HTTP ingestion, optional gRPC Push Streaming, Enterprise OAuth 2.0 / RFC 8693 Multi-Agent Identity, and Lakehouse analytics (pyarrow / pandas / polars).


🚀 Key Features

  • 100% Platform API v1 (/api/v1) Parity: Full coverage of all 80 Platform API v1 operations across jobs, schedules, job_runs, runs (job runs and dynamic jobs together), executions, connectors, settings, operations, migrations, vespa, narrativization, dynamic_jobs, discovery, and sidecars — continuously guarded in CI by tests/test_openapi_parity.py.
  • Native Sync & Async (asyncio) Clients: OpenCrawlingClient for scripts, Airflow/Dagster/Prefect DAGs, and notebooks; AsyncOpenCrawlingClient for non-blocking FastAPI services and AI agent runtimes (LangGraph, LlamaIndex, PydanticAI, CrewAI, MCP bridges).
  • Built-In REST Resilience & Concurrency Guarantees:
    • Versioned[T]: Wraps versioned resources (Job, Connector, SystemSettings, TransportSettings) with etag and location headers for optimistic concurrency (RequestOptions(if_match=...)) and conditional reads (If-None-Match → 304 Not Modified).
    • Automatic Idempotency-Key & RetryPolicy: Attaches a UUIDv4 Idempotency-Key (sdk-...) to every POST (excluding multipart uploads), retries network faults, 429, 502, 503, 504, and 409 idempotency-in-progress with jittered exponential backoff, and honors server Retry-After headers (both delta-seconds and HTTP-date).
    • RFC 9457 ApiProblemError: Parses application/problem+json error payloads into a typed exception exposing code, status, title, detail, request_id, field-level errors, current_etag, execution_id, retry_after, and extensions.
    • Cursor Pagination (iter / aiter): Lazy SyncPageIterator[T] and AsyncPageIterator[T] automatically follow nextCursor across pages on demand.
    • Lossless Settings (extra="allow"): SystemSettings and TransportSettings preserve unknown or newly introduced runtime fields losslessly across read-modify-replace cycles.
  • Enterprise OAuth 2.0 & Multi-Agent Identity (opencrawling.auth):
    • Static Bearer tokens and X-API-Key headers.
    • ClientCredentialsTokenProvider (RFC 6749 §4.4), RefreshTokenProvider (RFC 6749 §6), RFC 8693 TokenExchangeProvider (for AI agents acting on behalf of users with verified act.sub), and RFC 8628 DeviceCodeLogin.
    • Built-in token caching with 30-second pre-expiry renewal, automatic single 401 refresh-and-retry, and strict secret redaction in repr, logs, and exceptions.
  • OIS Direct Ingestion & Optional gRPC Push (opencrawling.ingest):
    • Typed OisDocument.upsert(...) and .delete(...) builders with fail-closed SecurityConfig (inheritance_enabled=False, evaluation_policy="FAIL_CLOSED") and PermissionRule helpers (allow_user, allow_group, allow_role, deny_user, deny_group, deny_role) targeting POST /api/v1/ingest/ois/{jobId}.
    • Optional GrpcPushClient and AsyncGrpcPushClient (opencrawling-sdk[grpc]) supporting Unary, Client-Streaming, Bi-Directional Streaming, and Ping RPCs against oc-grpc-push-connector, with language-neutral normalize_grpc_metadata() serialization (lowercase booleans, ISO-8601 timestamps, and compact RFC 8259 JSON scalar arrays expanded automatically by OisDocumentTranslator).
  • Governance & Enrichment Hook Services (opencrawling.hooks, opencrawling-sdk[hooks]):
    • Constant-time HMAC-SHA256 signing and verification (sign_payload, build_signed_headers, verify_signature, HookSignatureError) matching oc-runtime HttpGovernanceHook (X-OpenCrawling-Timestamp + X-OpenCrawling-Signature: sha256=<hex>, 32-character minimum secret, configurable replay window).
    • Typed GovernanceRequest, Decision (approve, sanitize, reject), EnrichmentRequest, and EnrichmentResponse enforcing GovernanceGate protected metadata keys client-side (is_protected_metadata_key, ProtectedMetadataKeyError).
    • FastAPI/Starlette @governance_endpoint, @enrichment_endpoint, create_governance_app, create_enrichment_app, plus SignedHookTestClient and AsyncSignedHookTestClient.
  • W3C Trace Context & OpenTelemetry (opencrawling-sdk[otel]):
    • Automatic W3C traceparent / tracestate propagation across sync/async HTTP and gRPC Push calls when an OpenTelemetry span is active, explicit RequestOptions(traceparent=..., tracestate=...), and optional enable_tracing=True (SpanKind.CLIENT) spans — degrading to a zero-overhead no-op when opentelemetry-api is not installed.
  • Lakehouse & Data Science Ergonomics (opencrawling-sdk[dataframe]):
    • Streams Parquet, CSV, Avro, and JSON execution audit reports directly to disk (client.executions.export_to(...)) via atomic temporary files with constant memory usage.
    • Converts DuckDB SQL QueryResult rows via .to_dicts(), .to_pyarrow(), .to_pandas(), and .to_polars().

📦 Installation

Requires Python 3.11+ (tested across Python 3.11, 3.12, 3.13, and 3.14; Production/Stable 1.0 GA).

1. Stable GA Release (PyPI)

Once a General Availability release (e.g., v1.0.5+) is published to PyPI, install it directly with pip (or uv pip install / poetry add):

pip install opencrawling-sdk

Install optional extras individually or combined as needed:

pip install "opencrawling-sdk[http2]"                          # HTTP/2 multiplexing via h2
pip install "opencrawling-sdk[grpc]"                           # High-throughput gRPC Push streaming (:9096)
pip install "opencrawling-sdk[dataframe]"                      # PyArrow, Pandas, and Polars conversions
pip install "opencrawling-sdk[hooks]"                          # FastAPI/Starlette governance & enrichment hook helpers
pip install "opencrawling-sdk[otel]"                           # OpenTelemetry W3C traceparent propagation & client spans
pip install "opencrawling-sdk[http2,grpc,dataframe,hooks,otel]" # All optional capabilities

2. Direct Wheel / Air-Gapped Installation & SHA-256 Verification (GA & Nightly RC)

Every Stable GA GitHub Release (v<x.y.z>) and Nightly Release Candidate Pre-Release (v<base>-rc.<YYYYMMDD>) also attaches pre-built pure-Python wheels (opencrawling_sdk-<pep440-version>-py3-none-any.whl), source distributions (opencrawling_sdk-<pep440-version>.tar.gz), and matching .sha256 checksum files, all indexed on the Download Center.

To verify and install a downloaded GA or Nightly RC wheel (for example in offline/air-gapped environments or before GA promotion to PyPI):

# 1. Verify cryptographic integrity (Linux: sha256sum; macOS: shasum -a 256)
sha256sum -c opencrawling_sdk-1.0.5-py3-none-any.whl.sha256

# 2. Install the verified wheel (append [http2,grpc,dataframe] for optional extras)
pip install ./opencrawling_sdk-1.0.5-py3-none-any.whl

For Nightly RC builds (where Maven 1.0.5-rc.YYYYMMDD maps to PEP 440 1.0.5rcYYYYMMDD), you can also install directly from the GitHub Release asset URL:

pip install "https://github.com/opencrawling/opencrawling/releases/download/v1.0.5-rc.<YYYYMMDD>/opencrawling_sdk-1.0.5rc<YYYYMMDD>-py3-none-any.whl"

3. Verify Installation

python -c "import opencrawling; print(opencrawling.__version__)"

💡 Code Examples

1. Synchronous Platform API v1: ETag Concurrency, Runs & Lakehouse Analytics

from opencrawling import OpenCrawlingClient, RequestOptions

with OpenCrawlingClient(
    base_url="http://localhost:8080",
    client_credentials={
        "token_endpoint": "https://idp.example.com/realms/opencrawling/protocol/openid-connect/token",
        "client_id": "opencrawling-automation",
        "client_secret": "...",
    },
) as client:
    # 1. Read a job along with its ETag
    versioned_job = client.jobs.get("1")
    print(f"Job '{versioned_job.data.name}' (ETag: {versioned_job.etag})")

    # 2. Patch conditionally with If-Match (RFC 7396 Merge Patch; raises ApiProblemError on 412)
    client.jobs.patch(
        "1",
        {"path": "/Sites/finance/documentLibrary"},
        options=RequestOptions(if_match=versioned_job.etag),
    )

    # 3. Start a job run (automatic Idempotency-Key) and poll until terminal state
    started = client.jobs.start("1")
    completed = client.job_runs.wait_for_completion(started.data.execution_id, timeout=300.0)

    # 4. Stream Parquet audit report to disk and run a DuckDB query
    client.executions.export_to(completed.execution_id, "parquet", "./audit.parquet")
    summary = client.executions.query(
        "SELECT status, COUNT(*) AS docs, SUM(rag_token_count) AS tokens "
        "FROM read_parquet('mode=RAG/**/*.parquet') GROUP BY status"
    )
    print(summary.to_dicts())

2. Asynchronous AI Agent Workflow (RFC 8693 Token Exchange)

import asyncio
from opencrawling import AsyncOpenCrawlingClient, DiscoveryDryRunRequest, DocumentIngestRequest
from opencrawling.auth import TokenExchangeProvider

async def agent_workflow(user_subject_token: str) -> None:
    delegated_auth = TokenExchangeProvider(
        token_endpoint="https://idp.example.com/realms/opencrawling/protocol/openid-connect/token",
        client_id="research-agent",
        client_secret="...",
        subject_token=user_subject_token,
    )

    async with AsyncOpenCrawlingClient("http://localhost:8080", token_provider=delegated_auth) as client:
        # 1. Preview readable documents under the user's source ACLs
        preview = await client.discovery.dry_run(
            "SharePoint_Prod",
            DiscoveryDryRunRequest(scope="/Drives/Legal", sample_size=25),
        )
        print(f"Estimated readable documents: {preview.match_count} ({preview.count_accuracy})")

        # 2. Ingest a single document on demand
        accepted = await client.dynamic_jobs.ingest_document(
            DocumentIngestRequest.upsert(
                connector="SharePoint_Prod",
                document="01BYE5RZ6QN3ZWBTUFOFD3GSPGOHDJD36K",
            )
        )
        print(f"Dynamic job triggered: {accepted.job_id} ({accepted.state})")

        # 3. Lazily iterate authorized OIS metadata sidecars
        async for sidecar in client.sidecars.iter("S3_Migration_Sink", prefix="legal/2026/"):
            print(sidecar.get("id"))

asyncio.run(agent_workflow("eyJ..."))

3. Direct OIS Document Push (POST /api/v1/ingest/ois/{jobId})

from opencrawling import OpenCrawlingClient
from opencrawling.ingest import OisContent, OisDocument, PermissionRule, SecurityConfig

with OpenCrawlingClient("http://localhost:8080", bearer_token="ois-ingest-secret-token") as client:
    doc = OisDocument.upsert(
        id="custom://crm/accounts/ACME-2026-Q4",
        content=OisContent.markdown("# ACME Corp Q4 Renewal\nAnnual contract value: $480,000."),
        metadata={"account_id": "ACME-001", "region": "EMEA"},
        security=SecurityConfig.fail_closed([
            PermissionRule.allow_group("sales-emea"),
            PermissionRule.deny_user("contractor@example.com"),
        ]),
    )
    receipt = client.ingest.push_ois("crm-rag-pipeline", doc)
    print(f"Accepted: {receipt.is_accepted} ({receipt.status}) -> {receipt.document_id}")

4. High-Throughput gRPC Push Streaming (opencrawling-sdk[grpc])

from opencrawling.ingest import (
    GrpcPushClient,
    GrpcPushDocument,
    GrpcPushRequest,
    GrpcSecurityConfig,
)

with GrpcPushClient("localhost:9096", bearer_token="grpc-push-token") as grpc_client:
    print(grpc_client.ping())

    requests = [
        GrpcPushRequest(
            job_id="1",
            document=GrpcPushDocument(
                id=f"erp://invoices/{i}",
                mime_type="text/plain",
                text_content=f"Invoice #{i} summary",
                security=GrpcSecurityConfig(allow_sids=("finance-team",)),
            ),
        )
        for i in range(100)
    ]
    summary = grpc_client.stream_documents(requests)
    print(f"Accepted {summary.total_accepted}/{summary.total_received} in {summary.duration_ms}ms")

5. Lossless Runtime Settings & RFC 9457 Error Handling

from opencrawling import ApiProblemError, OpenCrawlingClient, RequestOptions

with OpenCrawlingClient("http://localhost:8080", bearer_token="admin-jwt") as client:
    current = client.settings.system()
    try:
        # Unknown server fields are preserved automatically on round-trip
        updated = client.settings.replace_system(
            current.data.with_updates(chunk_size=1024),
            options=RequestOptions(if_match=current.etag),
        )
        print("New ETag:", updated.etag)
    except ApiProblemError as err:
        print(f"Error [{err.code}] (HTTP {err.status}, request_id={err.request_id}): {err.detail}")
        if err.current_etag:
            print("Server current ETag:", err.current_etag)

6. Native Job Scheduling (client.schedules)

from opencrawling import JobScheduleConfig, MisfirePolicy, OpenCrawlingClient, OverlapPolicy, RequestOptions

with OpenCrawlingClient("http://localhost:8080", bearer_token="admin-jwt") as client:
    # 1. Validate a cron expression and preview next fire timestamps
    preview = client.schedules.validate_cron("0 2 * * *", timezone="Europe/Rome", count=5)
    assert preview.valid

    # 2. Attach or replace a job's recurring schedule
    saved = client.schedules.set(
        "1",
        JobScheduleConfig(
            cron="0 2 * * *",
            timezone="Europe/Rome",
            enabled=True,
            overlap_policy=OverlapPolicy.SKIP,
            misfire_policy=MisfirePolicy.FIRE_ONCE,
            max_consecutive_failures=3,
        ),
    )

    # 3. Inspect scheduler cluster overview and pause with optimistic concurrency
    overview = client.schedules.overview(horizon="24h")
    print(f"Active schedules: {overview.active_schedules}, skipped (24h): {overview.skipped_last_24h}")
    client.schedules.pause("1", RequestOptions(if_match=saved.etag))

7. Building Governance & Enrichment Hook Services (opencrawling.hooks)

from opencrawling.hooks import Decision, GovernanceRequest, SignedHookTestClient, create_governance_app

SECRET = "a-shared-secret-at-least-32-characters-long"

async def redact_pii(req: GovernanceRequest) -> Decision:
    if req.text and "SSN:" in req.text:
        return Decision.sanitize(
            text=req.text.replace("SSN:", "[REDACTED]:"),
            metadata={"pii_redacted": "true"},
            audit={"rule": "ssn-mask"},
        )
    return Decision.approve()

# Serve with uvicorn/FastAPI/Starlette
app = create_governance_app(redact_pii, secret=SECRET, path="/governance")

# Unit-test with HMAC-SHA256 signed requests without starting a runtime
test_client = SignedHookTestClient(secret=SECRET, app=app)
decision = test_client.post_governance({"documentId": "doc-1", "text": "SSN: 123-45-6789"})
assert decision.decision == "SANITIZE"

8. W3C Distributed Tracing (opencrawling-sdk[otel])

from opencrawling import OpenCrawlingClient, RequestOptions

# Automatically propagates active OpenTelemetry span context (W3C traceparent/tracestate)
# and emits CLIENT spans when enable_tracing=True:
with OpenCrawlingClient("http://localhost:8080", enable_tracing=True) as client:
    print(client.version())

# Or supply explicit traceparent/tracestate headers without installing OpenTelemetry:
opts = RequestOptions(
    traceparent="00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
    tracestate="vendor=value",
)

📜 Versioning, Stability & Changelog

  • CHANGELOG.md — Keep a Changelog release history.
  • VERSIONING.md — Semantic Versioning 2.0.0 guarantees, 2-minor / 6-month deprecation policy, Python & OpenCrawling runtime compatibility matrix, and GA release checklist.

🧭 Platform API v1 Sub-Client Reference

Every sub-client exists on both OpenCrawlingClient (sync) and AsyncOpenCrawlingClient (async):

Sub-Client Methods
Root (client) index(), version(), health(), me(), with_defaults(options)
client.jobs list, iter, create, get, get_if_changed, replace, patch, delete, start, stop, pause, resume, diagnose, traces, errors
client.schedules list, iter, overview, validate_cron, get, get_if_changed, set, delete, pause, resume, dispatches, iter_dispatches
client.job_runs list, iter, get, stop, pause, resume, history, wait_for_completion
client.runs list (kind, state, job_id), iter, get, history, stop, pause, resume
client.executions list, iter, list_for_job, get, documents, iter_documents, export, export_to, query
client.connectors list, iter, get, get_if_changed, create, upsert, replace, delete, check, check_definition
client.settings system, replace_system, transport, replace_transport, test_grpc
client.operations status, throughput, logs, metrics
client.migrations plan_manifoldcf, apply_manifoldcf
client.vespa health, document_counts, query, deploy_bundled_schema, deploy_custom_schema
client.narrativization generate_template
client.dynamic_jobs start, ingest_document, list, iter, get, stop, approve, reject
client.discovery connectors, tree, dry_run
client.sidecars stores, read, iter
client.ingest push_ois, upsert, delete, with_token

🛠️ CI/CD, Quality Gates & Publishing

  • CI Matrix (.github/workflows/ci.yml): Runs ruff check, mypy --strict src/opencrawling, pytest (including tests/test_openapi_parity.py), and wheel/sdist build verification across Python 3.11, 3.12, and 3.13.
  • Version Stamping (scripts/stamp-version.sh): Automatically translates Maven versions into PEP 440 (1.0.5-SNAPSHOT → 1.0.5.dev0, 1.0.5-rc.YYYYMMDD → 1.0.5rcYYYYMMDD, 1.0.5 → 1.0.5) across pyproject.toml and src/opencrawling/_version.py.
  • Nightly RC (.github/workflows/nightly-rc.yml): Builds .whl and .tar.gz artifacts, generates SHA-256 checksums and SLSA provenance, and attaches them to the nightly GitHub Pre-Release.
  • Stable GA PyPI & GitHub Release Publishing (.github/workflows/python-sdk-publish.yml): Publishes verified stable releases (!v*-rc.* and prerelease != true) to PyPI using PyPI Trusted Publishers (OIDC) with Sigstore attestations, and attaches .whl, .tar.gz, and .sha256 assets to the GA GitHub Release (indexed automatically on the Download Center).

🗺️ Documentation


🎓 Tutorials


🧪 Testing


🤝 Community

Clone this wiki locally