Repository navigation
Python Client SDK
The OpenCrawling Python Client SDK (oc-python-client-sdk, published to PyPI as opencrawling-sdk, importable as opencrawling) is a modern, strictly typed (PEP 561) Python 3.11+ library offering both synchronous (OpenCrawlingClient) and asynchronous (AsyncOpenCrawlingClient) interfaces over httpx and Pydantic v2.
It delivers 1:1 parity with oc-java-client-sdk across all 80 Platform API v1 (/api/v1) operations (including native cron job scheduling and the unified /api/v1/runs API), direct Open Ingestion Standard (OIS) HTTP ingestion, optional gRPC Push Streaming, Enterprise OAuth 2.0 / RFC 8693 Multi-Agent Identity, and Lakehouse analytics (pyarrow / pandas / polars).
-
100% Platform API v1 (
/api/v1) Parity: Full coverage of all 80 Platform API v1 operations acrossjobs,schedules,job_runs,runs(job runs and dynamic jobs together),executions,connectors,settings,operations,migrations,vespa,narrativization,dynamic_jobs,discovery, andsidecars— continuously guarded in CI bytests/test_openapi_parity.py. -
Native Sync & Async (
asyncio) Clients:OpenCrawlingClientfor scripts, Airflow/Dagster/Prefect DAGs, and notebooks;AsyncOpenCrawlingClientfor non-blocking FastAPI services and AI agent runtimes (LangGraph, LlamaIndex, PydanticAI, CrewAI, MCP bridges). -
Built-In REST Resilience & Concurrency Guarantees:
-
Versioned[T]: Wraps versioned resources (Job,Connector,SystemSettings,TransportSettings) withetagandlocationheaders for optimistic concurrency (RequestOptions(if_match=...)) and conditional reads (If-None-Match→304 Not Modified). -
Automatic
Idempotency-Key&RetryPolicy: Attaches a UUIDv4Idempotency-Key(sdk-...) to everyPOST(excluding multipart uploads), retries network faults,429,502,503,504, and409 idempotency-in-progresswith jittered exponential backoff, and honors serverRetry-Afterheaders (both delta-seconds and HTTP-date). -
RFC 9457
ApiProblemError: Parsesapplication/problem+jsonerror payloads into a typed exception exposingcode,status,title,detail,request_id, field-levelerrors,current_etag,execution_id,retry_after, andextensions. -
Cursor Pagination (
iter/aiter): LazySyncPageIterator[T]andAsyncPageIterator[T]automatically follownextCursoracross pages on demand. -
Lossless Settings (
extra="allow"):SystemSettingsandTransportSettingspreserve unknown or newly introduced runtime fields losslessly across read-modify-replace cycles.
-
-
Enterprise OAuth 2.0 & Multi-Agent Identity (
opencrawling.auth):- Static Bearer tokens and
X-API-Keyheaders. -
ClientCredentialsTokenProvider(RFC 6749 §4.4),RefreshTokenProvider(RFC 6749 §6), RFC 8693TokenExchangeProvider(for AI agents acting on behalf of users with verifiedact.sub), and RFC 8628DeviceCodeLogin. - Built-in token caching with 30-second pre-expiry renewal, automatic single
401refresh-and-retry, and strict secret redaction inrepr, logs, and exceptions.
- Static Bearer tokens and
-
OIS Direct Ingestion & Optional gRPC Push (
opencrawling.ingest):- Typed
OisDocument.upsert(...)and.delete(...)builders with fail-closedSecurityConfig(inheritance_enabled=False,evaluation_policy="FAIL_CLOSED") andPermissionRulehelpers (allow_user,allow_group,allow_role,deny_user,deny_group,deny_role) targetingPOST /api/v1/ingest/ois/{jobId}. - Optional
GrpcPushClientandAsyncGrpcPushClient(opencrawling-sdk[grpc]) supporting Unary, Client-Streaming, Bi-Directional Streaming, and Ping RPCs againstoc-grpc-push-connector, with language-neutralnormalize_grpc_metadata()serialization (lowercase booleans, ISO-8601 timestamps, and compact RFC 8259 JSON scalar arrays expanded automatically byOisDocumentTranslator).
- Typed
-
Governance & Enrichment Hook Services (
opencrawling.hooks,opencrawling-sdk[hooks]):- Constant-time HMAC-SHA256 signing and verification (
sign_payload,build_signed_headers,verify_signature,HookSignatureError) matchingoc-runtimeHttpGovernanceHook(X-OpenCrawling-Timestamp+X-OpenCrawling-Signature: sha256=<hex>, 32-character minimum secret, configurable replay window). - Typed
GovernanceRequest,Decision(approve,sanitize,reject),EnrichmentRequest, andEnrichmentResponseenforcingGovernanceGateprotected metadata keys client-side (is_protected_metadata_key,ProtectedMetadataKeyError). - FastAPI/Starlette
@governance_endpoint,@enrichment_endpoint,create_governance_app,create_enrichment_app, plusSignedHookTestClientandAsyncSignedHookTestClient.
- Constant-time HMAC-SHA256 signing and verification (
-
W3C Trace Context & OpenTelemetry (
opencrawling-sdk[otel]):- Automatic W3C
traceparent/tracestatepropagation across sync/async HTTP and gRPC Push calls when an OpenTelemetry span is active, explicitRequestOptions(traceparent=..., tracestate=...), and optionalenable_tracing=True(SpanKind.CLIENT) spans — degrading to a zero-overhead no-op whenopentelemetry-apiis not installed.
- Automatic W3C
-
Lakehouse & Data Science Ergonomics (
opencrawling-sdk[dataframe]):- Streams Parquet, CSV, Avro, and JSON execution audit reports directly to disk (
client.executions.export_to(...)) via atomic temporary files with constant memory usage. - Converts DuckDB SQL
QueryResultrows via.to_dicts(),.to_pyarrow(),.to_pandas(), and.to_polars().
- Streams Parquet, CSV, Avro, and JSON execution audit reports directly to disk (
Requires Python 3.11+ (tested across Python 3.11, 3.12, 3.13, and 3.14; Production/Stable 1.0 GA).
Once a General Availability release (e.g., v1.0.5+) is published to PyPI, install it directly with pip (or uv pip install / poetry add):
pip install opencrawling-sdkInstall optional extras individually or combined as needed:
pip install "opencrawling-sdk[http2]" # HTTP/2 multiplexing via h2
pip install "opencrawling-sdk[grpc]" # High-throughput gRPC Push streaming (:9096)
pip install "opencrawling-sdk[dataframe]" # PyArrow, Pandas, and Polars conversions
pip install "opencrawling-sdk[hooks]" # FastAPI/Starlette governance & enrichment hook helpers
pip install "opencrawling-sdk[otel]" # OpenTelemetry W3C traceparent propagation & client spans
pip install "opencrawling-sdk[http2,grpc,dataframe,hooks,otel]" # All optional capabilitiesEvery Stable GA GitHub Release (v<x.y.z>) and Nightly Release Candidate Pre-Release (v<base>-rc.<YYYYMMDD>) also attaches pre-built pure-Python wheels (opencrawling_sdk-<pep440-version>-py3-none-any.whl), source distributions (opencrawling_sdk-<pep440-version>.tar.gz), and matching .sha256 checksum files, all indexed on the Download Center.
To verify and install a downloaded GA or Nightly RC wheel (for example in offline/air-gapped environments or before GA promotion to PyPI):
# 1. Verify cryptographic integrity (Linux: sha256sum; macOS: shasum -a 256)
sha256sum -c opencrawling_sdk-1.0.5-py3-none-any.whl.sha256
# 2. Install the verified wheel (append [http2,grpc,dataframe] for optional extras)
pip install ./opencrawling_sdk-1.0.5-py3-none-any.whlFor Nightly RC builds (where Maven 1.0.5-rc.YYYYMMDD maps to PEP 440 1.0.5rcYYYYMMDD), you can also install directly from the GitHub Release asset URL:
pip install "https://github.com/opencrawling/opencrawling/releases/download/v1.0.5-rc.<YYYYMMDD>/opencrawling_sdk-1.0.5rc<YYYYMMDD>-py3-none-any.whl"python -c "import opencrawling; print(opencrawling.__version__)"from opencrawling import OpenCrawlingClient, RequestOptions
with OpenCrawlingClient(
base_url="http://localhost:8080",
client_credentials={
"token_endpoint": "https://idp.example.com/realms/opencrawling/protocol/openid-connect/token",
"client_id": "opencrawling-automation",
"client_secret": "...",
},
) as client:
# 1. Read a job along with its ETag
versioned_job = client.jobs.get("1")
print(f"Job '{versioned_job.data.name}' (ETag: {versioned_job.etag})")
# 2. Patch conditionally with If-Match (RFC 7396 Merge Patch; raises ApiProblemError on 412)
client.jobs.patch(
"1",
{"path": "/Sites/finance/documentLibrary"},
options=RequestOptions(if_match=versioned_job.etag),
)
# 3. Start a job run (automatic Idempotency-Key) and poll until terminal state
started = client.jobs.start("1")
completed = client.job_runs.wait_for_completion(started.data.execution_id, timeout=300.0)
# 4. Stream Parquet audit report to disk and run a DuckDB query
client.executions.export_to(completed.execution_id, "parquet", "./audit.parquet")
summary = client.executions.query(
"SELECT status, COUNT(*) AS docs, SUM(rag_token_count) AS tokens "
"FROM read_parquet('mode=RAG/**/*.parquet') GROUP BY status"
)
print(summary.to_dicts())import asyncio
from opencrawling import AsyncOpenCrawlingClient, DiscoveryDryRunRequest, DocumentIngestRequest
from opencrawling.auth import TokenExchangeProvider
async def agent_workflow(user_subject_token: str) -> None:
delegated_auth = TokenExchangeProvider(
token_endpoint="https://idp.example.com/realms/opencrawling/protocol/openid-connect/token",
client_id="research-agent",
client_secret="...",
subject_token=user_subject_token,
)
async with AsyncOpenCrawlingClient("http://localhost:8080", token_provider=delegated_auth) as client:
# 1. Preview readable documents under the user's source ACLs
preview = await client.discovery.dry_run(
"SharePoint_Prod",
DiscoveryDryRunRequest(scope="/Drives/Legal", sample_size=25),
)
print(f"Estimated readable documents: {preview.match_count} ({preview.count_accuracy})")
# 2. Ingest a single document on demand
accepted = await client.dynamic_jobs.ingest_document(
DocumentIngestRequest.upsert(
connector="SharePoint_Prod",
document="01BYE5RZ6QN3ZWBTUFOFD3GSPGOHDJD36K",
)
)
print(f"Dynamic job triggered: {accepted.job_id} ({accepted.state})")
# 3. Lazily iterate authorized OIS metadata sidecars
async for sidecar in client.sidecars.iter("S3_Migration_Sink", prefix="legal/2026/"):
print(sidecar.get("id"))
asyncio.run(agent_workflow("eyJ..."))from opencrawling import OpenCrawlingClient
from opencrawling.ingest import OisContent, OisDocument, PermissionRule, SecurityConfig
with OpenCrawlingClient("http://localhost:8080", bearer_token="ois-ingest-secret-token") as client:
doc = OisDocument.upsert(
id="custom://crm/accounts/ACME-2026-Q4",
content=OisContent.markdown("# ACME Corp Q4 Renewal\nAnnual contract value: $480,000."),
metadata={"account_id": "ACME-001", "region": "EMEA"},
security=SecurityConfig.fail_closed([
PermissionRule.allow_group("sales-emea"),
PermissionRule.deny_user("contractor@example.com"),
]),
)
receipt = client.ingest.push_ois("crm-rag-pipeline", doc)
print(f"Accepted: {receipt.is_accepted} ({receipt.status}) -> {receipt.document_id}")from opencrawling.ingest import (
GrpcPushClient,
GrpcPushDocument,
GrpcPushRequest,
GrpcSecurityConfig,
)
with GrpcPushClient("localhost:9096", bearer_token="grpc-push-token") as grpc_client:
print(grpc_client.ping())
requests = [
GrpcPushRequest(
job_id="1",
document=GrpcPushDocument(
id=f"erp://invoices/{i}",
mime_type="text/plain",
text_content=f"Invoice #{i} summary",
security=GrpcSecurityConfig(allow_sids=("finance-team",)),
),
)
for i in range(100)
]
summary = grpc_client.stream_documents(requests)
print(f"Accepted {summary.total_accepted}/{summary.total_received} in {summary.duration_ms}ms")from opencrawling import ApiProblemError, OpenCrawlingClient, RequestOptions
with OpenCrawlingClient("http://localhost:8080", bearer_token="admin-jwt") as client:
current = client.settings.system()
try:
# Unknown server fields are preserved automatically on round-trip
updated = client.settings.replace_system(
current.data.with_updates(chunk_size=1024),
options=RequestOptions(if_match=current.etag),
)
print("New ETag:", updated.etag)
except ApiProblemError as err:
print(f"Error [{err.code}] (HTTP {err.status}, request_id={err.request_id}): {err.detail}")
if err.current_etag:
print("Server current ETag:", err.current_etag)from opencrawling import JobScheduleConfig, MisfirePolicy, OpenCrawlingClient, OverlapPolicy, RequestOptions
with OpenCrawlingClient("http://localhost:8080", bearer_token="admin-jwt") as client:
# 1. Validate a cron expression and preview next fire timestamps
preview = client.schedules.validate_cron("0 2 * * *", timezone="Europe/Rome", count=5)
assert preview.valid
# 2. Attach or replace a job's recurring schedule
saved = client.schedules.set(
"1",
JobScheduleConfig(
cron="0 2 * * *",
timezone="Europe/Rome",
enabled=True,
overlap_policy=OverlapPolicy.SKIP,
misfire_policy=MisfirePolicy.FIRE_ONCE,
max_consecutive_failures=3,
),
)
# 3. Inspect scheduler cluster overview and pause with optimistic concurrency
overview = client.schedules.overview(horizon="24h")
print(f"Active schedules: {overview.active_schedules}, skipped (24h): {overview.skipped_last_24h}")
client.schedules.pause("1", RequestOptions(if_match=saved.etag))from opencrawling.hooks import Decision, GovernanceRequest, SignedHookTestClient, create_governance_app
SECRET = "a-shared-secret-at-least-32-characters-long"
async def redact_pii(req: GovernanceRequest) -> Decision:
if req.text and "SSN:" in req.text:
return Decision.sanitize(
text=req.text.replace("SSN:", "[REDACTED]:"),
metadata={"pii_redacted": "true"},
audit={"rule": "ssn-mask"},
)
return Decision.approve()
# Serve with uvicorn/FastAPI/Starlette
app = create_governance_app(redact_pii, secret=SECRET, path="/governance")
# Unit-test with HMAC-SHA256 signed requests without starting a runtime
test_client = SignedHookTestClient(secret=SECRET, app=app)
decision = test_client.post_governance({"documentId": "doc-1", "text": "SSN: 123-45-6789"})
assert decision.decision == "SANITIZE"from opencrawling import OpenCrawlingClient, RequestOptions
# Automatically propagates active OpenTelemetry span context (W3C traceparent/tracestate)
# and emits CLIENT spans when enable_tracing=True:
with OpenCrawlingClient("http://localhost:8080", enable_tracing=True) as client:
print(client.version())
# Or supply explicit traceparent/tracestate headers without installing OpenTelemetry:
opts = RequestOptions(
traceparent="00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
tracestate="vendor=value",
)- CHANGELOG.md — Keep a Changelog release history.
- VERSIONING.md — Semantic Versioning 2.0.0 guarantees, 2-minor / 6-month deprecation policy, Python & OpenCrawling runtime compatibility matrix, and GA release checklist.
Every sub-client exists on both OpenCrawlingClient (sync) and AsyncOpenCrawlingClient (async):
| Sub-Client | Methods |
|---|---|
Root (client) |
index(), version(), health(), me(), with_defaults(options)
|
client.jobs |
list, iter, create, get, get_if_changed, replace, patch, delete, start, stop, pause, resume, diagnose, traces, errors
|
client.schedules |
list, iter, overview, validate_cron, get, get_if_changed, set, delete, pause, resume, dispatches, iter_dispatches
|
client.job_runs |
list, iter, get, stop, pause, resume, history, wait_for_completion
|
client.runs |
list (kind, state, job_id), iter, get, history, stop, pause, resume
|
client.executions |
list, iter, list_for_job, get, documents, iter_documents, export, export_to, query
|
client.connectors |
list, iter, get, get_if_changed, create, upsert, replace, delete, check, check_definition
|
client.settings |
system, replace_system, transport, replace_transport, test_grpc
|
client.operations |
status, throughput, logs, metrics
|
client.migrations |
plan_manifoldcf, apply_manifoldcf
|
client.vespa |
health, document_counts, query, deploy_bundled_schema, deploy_custom_schema
|
client.narrativization |
generate_template |
client.dynamic_jobs |
start, ingest_document, list, iter, get, stop, approve, reject
|
client.discovery |
connectors, tree, dry_run
|
client.sidecars |
stores, read, iter
|
client.ingest |
push_ois, upsert, delete, with_token
|
-
CI Matrix (
.github/workflows/ci.yml): Runsruff check,mypy --strict src/opencrawling,pytest(includingtests/test_openapi_parity.py), and wheel/sdist build verification across Python3.11,3.12, and3.13. -
Version Stamping (
scripts/stamp-version.sh): Automatically translates Maven versions into PEP 440 (1.0.5-SNAPSHOT→1.0.5.dev0,1.0.5-rc.YYYYMMDD→1.0.5rcYYYYMMDD,1.0.5→1.0.5) acrosspyproject.tomlandsrc/opencrawling/_version.py. -
Nightly RC (
.github/workflows/nightly-rc.yml): Builds.whland.tar.gzartifacts, generates SHA-256 checksums and SLSA provenance, and attaches them to the nightly GitHub Pre-Release. -
Stable GA PyPI & GitHub Release Publishing (
.github/workflows/python-sdk-publish.yml): Publishes verified stable releases (!v*-rc.*andprerelease != true) to PyPI using PyPI Trusted Publishers (OIDC) with Sigstore attestations, and attaches.whl,.tar.gz, and.sha256assets to the GA GitHub Release (indexed automatically on the Download Center).
- Home
- User & Installation Guide
- Quick Start
- VM & Bare-Metal Distribution Installation
- Local Development & Fast Docker Builds
- User Guide
- Architecture
- Zero-Trust Security & ACL Management
- Security and Identity Management
- Document Authorization & OpenFGA
- Enterprise Multi-Agent Integration
- Zero-Trust Data Provider Agent & Active Ingestion
- Document Lifecycle Actions & Tombstones
- Apache Tika 4 Text Extraction
- Configuration Guide
- Native Job Scheduler
- Platform API (
/api/v1) - Platform API: Traceability
- Connectors Reference
- Alfresco Content Services (ACS) Connector
- Alfresco Process Services (APS) Connector
- Flowable Repository Connector
- Camunda Repository Connector
- OASIS CMIS Repository Connector
- Doxis Repository Connector
- M-Files Repository Connector
- SharePoint Online Repository Connector
- Google Drive Repository Connector
- Atlassian Confluence Repository Connector
- IBM FileNet P8 Repository Connector
- OpenText Documentum Repository Connector
- Zammad Repository Connector
- Hyland OpenArch Integration (Repository, Output & MCP)
- Hyland Content Innovation Cloud (CIC) Connector
- Relational Database (JDBC) Connector
- Web Repository Connector
- Apache StormCrawler Repository Connector
- gRPC Push Repository Connector
- Elasticsearch Output Connector
- Apache Solr Output Connector
- Luxir Output Connector
- Apache SeaTunnel Output Connector
- Apache Ozone Output Connector (Migration Mode)
- AWS S3 Output Connector (Migration Mode)
- Alfresco Content Services Output Connector (Migration Mode)
- Apache Iceberg Output Connector
- Doxis Output Connector
- Connector Execution Modes
- Model Context Protocol
- Apache Ozone Claim Check Store
- Lakehouse Audit Reporting & Ledger
- AI-Powered Observability (AIOps)
- Auto-Narrativization Copilot
- Maven Connector Archetypes
- Java Client SDK
- Python Client SDK
- OpenCrawling CLI
- OpenAPI Specification & Swagger UI
- gRPC Internal Transport
- Migration Guide (ManifoldCF)
- Integration Test — Decoupled Pipeline
- M-Files Integration Test
- SharePoint Online Integration Test
- Google Drive Integration Test
- Confluence Integration Test
- IBM FileNet Integration Test
- OpenText Documentum Integration Test
- Zammad Integration Test
- Hyland OpenArch Integration Test
- Hyland CIC Integration Test
- Web Crawler Test
- Elasticsearch Integration Test
- Local Docker Build Test
- Root CLI Runner Scripts Test
- Secure MCP Server Integration Test
- Narrativization Integration Test
- Reporting Integration Test
- AWS S3 Migration Integration Test
- Alfresco Migration Integration Test
- OpenFGA Authorization Integration Test
- CLI Decoupled Integration Test
- Java Client SDK Decoupled Test
- Apache Iceberg Decoupled Test
- Doxis Output Integration Test
- Platform API v1 Test Suite