Repository navigation
Platform API
The Platform API is one versioned REST interface to everything OpenCrawling does, so adopters can run it entirely without the admin UI (#313, part 1). It covers what the UI can do: jobs and their runs, executions and audit reports, connectors, settings, operations, migrations and integrations. It also covers the optional features: discovery, dynamic jobs and sidecars.
-
Base URL:
http(s)://<runtime>/api/v1 -
Description: OpenAPI 3.1 at
/v3/api-docs/v1; interactive at/swagger-ui.html(pick "Platform API v1"). -
Entry point:
GET /api/v1lists every collection and the optional features enabled on the installation.
The existing endpoints (/api/jobs, /api/connectors, ...) keep working unchanged. The v1 endpoints call the same controllers, so validation, authorization and audit are the same; see Legacy endpoints for the mapping.
-
Tokens: with
opencrawling.security.identity.mode=jwtandprotect-api=true, every request needsAuthorization: Bearer <token>. The SDK and the CLI fetch tokens themselves (oc auth login, client credentials). See Security and Identity Management. -
Who may do what: with management authorization on, the rules are the legacy ones.
- Admins may do everything.
- Job creators may create jobs, list connectors (without their configuration) and use the narrativization copilot.
- Each job's
accessblock decides who may see, run and change it. A job the caller may not see answers404. - Admin-only: connector changes and checks, settings, system logs, metrics,
executions:query, migrations and integrations.
-
GET /api/v1/me: the caller as the runtime sees it, withadminandcanCreateJobs.
Every error is application/problem+json. The one exception is GET /api/v1/health, whose 503 carries the health report itself.
{
"type": "https://github.com/opencrawling/opencrawling/wiki/Platform-API#problem-validation-failed",
"title": "Validation failed",
"status": 400,
"detail": "The request has 2 invalid values.",
"instance": "/api/v1/jobs",
"requestId": "6a1f0c1e-...",
"errors": [
{"field": "name", "message": "must not be blank"},
{"field": "outputConnector", "message": "no connector named 'Nope'"}
]
}-
type: identifies the problem, and leads to its row in the table below (...Platform-API#problem-<name>). Clients should branch on the part after#problem-; the SDK gives it asApiProblem.code(). -
detail: says what went wrong, for people. -
requestId: quote it when reporting a problem; it finds the request in the runtime's log. -
Other members: some types add them, such as
errors,currentEtag,executionId,jobsormaxBytes.
The table is the complete list: a test checks that the runtime cannot answer with a type that is not in it.
Every response carries X-Request-Id. Send your own (1 to 64 letters, digits, ., _, : or -) to correlate with your logs; otherwise the runtime makes one. It is in every problem detail and in the runtime's log context (requestId).
Every Platform API answer has these headers, whatever the identity mode (OWASP REST Security Cheat Sheet):
-
Cache-Control: answers depend on the caller and may hold secrets, so nothing shared may cache them. Reads areprivate, no-cache: a client may keep them, but revalidates withIf-None-Match(RFC 9111). Everything else isno-store. -
Vary: Authorization: an answer depends on who asks. -
X-Content-Type-Options: nosniffandContent-Security-Policy: default-src 'none'; frame-ancestors 'none': a browser never treats an answer as a page.
Request size: a request body may have at most 1 MiB (opencrawling.api.max-request-bytes); a larger one answers 413 body-too-large, whether it announces its length or not. Uploads (multipart) have their own limits (spring.servlet.multipart.max-file-size), and too large an upload answers 413 too. Settings maps (crawlSpec, connector configuration) may have at most 200 entries.
{
"items": [ ... ],
"page": {"limit": 50, "nextCursor": "djEuNTAuWnE3...", "totalItems": 120}
}-
Paging:
limitfrom 1 to 500 (default 50); passpage.nextCursorascursorfor the next page. It is also inLink: <...>; rel="next".nextCursorisnullon the last page. -
Cursors are opaque and tied to the filters and sort they were made with; reusing one with others gives
400. -
Stable pages: a cursor resumes right after the last item it returned. Items added or removed before it (new executions arrive first, newest first) cause no duplicates and no gaps. Every sort ends with the item's id, so items with equal sort values keep one order. An execution's documents also tie the cursor to
limit, so keep the page size the same while following cursors. -
Filters are query parameters named after the field (
status,type,jobId, ...);qsearches text. -
Sorting:
sort=-lastRun,name, comma-separated fields with-for descending. Each collection documents its fields; an unknown one gives400.
Single resources (jobs, connectors, system and transport settings) return an ETag.
-
What a tag covers: the resource as you received it. For a job that includes its state and your permissions, so a run that changes the state changes the tag. For a connector it also includes its secrets (as a keyed digest), so changing a secret changes the tag although the masked view looks the same. With
opencrawling.control-plane.store=jdbc, the key (api-v1-etag-key.json) lives in the store too, so every runtime sharing the database gives a connector the same ETag (#315, PR#345). -
Conditional reads:
GETwithIf-None-Match: <etag>answers304 Not Modifiedonly when nothing in the answer changed. -
Conditional changes:
PUT,PATCHandDELETEwithIf-Match: <etag>answer412when the resource changed since you read it. Read it again, and redo the change on what is there now. A job whose run changed its state since you read it answers412too. -
Atomic: the check and the change happen as one step under
ControlPlaneStore.locked(aSELECT ... FOR UPDATElock onoc_lockinjdbcmode), so of several changes made from the same version—even across multiple runtimes sharing a database—exactly one succeeds and the others get412(no lost updates, RFC 9110 13.1.1). -
Without
If-Match: the change is applied unconditionally, unless the runtime setsopencrawling.api.require-if-match=true. Then every change of a job, connector or settings document through the Platform API must carryIf-Match, and one without it answers428 precondition-required(RFC 6585). The admin UI, the CLI, the SDK, and the ManifoldCF migrator (OpenCrawlingWriter,#337) read the version first, so they keep working. The deprecated legacy endpoints are not covered. -
Weak tags (a deliberate deviation from RFC 9110): a weak copy of a tag (
W/"...") is accepted inIf-Match. RFC 9110 asks for strong comparison there, but proxies that compress responses, such as nginx withgzip on, turn tags weak. The tag digests the JSON, not the bytes sent, so it still names the same version. -
Browsers: the runtime sends no CORS headers, so serve browser clients from the same origin, as the admin UI is. A proxy that adds CORS must expose the headers clients read:
Access-Control-Expose-Headers: ETag, Location, Link, X-Request-Id.
-
PUTreplaces the whole definition; optional fields left out are cleared, except a job'saccess, which is kept when left out (it is changed only by sending it). -
PATCHwithContent-Type: application/merge-patch+json(RFC 7396) changes only the members sent;nullremoves an optional member.
Operations other than create, read, update and delete are POST to {resource}:{action}, for example POST /api/v1/jobs/{id}:start or POST /api/v1/connectors:check.
Send Idempotency-Key: <key> (1 to 200 letters, digits, ., _, : or -) with any POST except uploads. The key may also be sent as a Structured Field string, Idempotency-Key: "<key>", as the IETF draft writes it.
-
Retry: the first request runs; a retry with the same key and body gets the same answer again, with
Idempotent-Replayed: true(a widely used convention; the IETF draft does not define a header for it). -
Reuse: the same key with another body gives
422; a retry while the first request still runs gives409withRetry-After. -
Size: a request with a key may have at most 1 MiB of body. An answer larger than 256 KiB (
opencrawling.api.idempotency.max-stored-bytes) is not kept: a retry gets409 idempotency-answer-unavailable, with the first status, and is not run again. -
Same request: a retry must send the same body, query string and
Content-Type; otherwise it is another request (422). -
Uploads: a multipart request with a key answers
400; send uploads without one. -
Scope and lifetime: keys are per caller, method and path, and remembered for 24 hours. Without sign-in (
identity.mode=none) every caller is the same anonymous caller, so clients must pick keys that cannot collide, such as UUIDs. -
Where they are kept: in the runtime's memory by default. With
opencrawling.api.idempotency.store=redisthey are kept in Redis, and withjdbcin the control-plane database (oc_idempotency_key,#315, PR#359); either way they are shared by every runtime instance and kept across restarts. The Docker Compose stacks use Redis. See Configuration Guide. -
Store unavailable: if Redis or the database cannot be reached, the request is refused with
503(idempotency-unavailable, withRetry-After) and not run. Retry later with the same key. - Long requests: while a request runs, its key is held (Redis or the database renews it), however long the request takes.
-
Limits of the memory store: at most 10,000 keys, and 1,000 per caller. A key whose request still runs is never dropped before the retention ends; when a new key would need dropping one, the request is refused with
503. Finished keys may be dropped before 24 hours when a caller or the runtime is at its limit. Without sign-in all clients are one caller and share its 1,000 keys; use the Redis store when many clients send keys. - The one gap: if Redis fails after the request ran and stays down while the runtime retries keeping its outcome, the outcome is lost. Once the key's in-progress marker expires (10 minutes), a retry would run the request again. The runtime logs this as an error.
- Failures: only successful answers are remembered, so a failed request can be retried with the same key.
Use one for :start, so that a network retry does not start a second run.
zsh: in a shell variable followed by
:start, zsh reads:sas a modifier, so"$API/jobs/$ID:start"loses the action. Write${ID}:start.
| Method | Path | Purpose |
|---|---|---|
| GET | /api/v1 |
Entry point: links (jobs, jobRuns, executions, connectors, systemSettings, transportSettings, systemStatus, systemThroughput, systemLogs, metrics, me, and the enabled features' collections), enabled features, documentation
|
| GET | /api/v1/version |
API and runtime versions |
| GET | /api/v1/health |
Health of the runtime and its dependencies (200 healthy, 503 otherwise) |
| GET | /api/v1/me |
The current caller |
| Method | Path | Purpose |
|---|---|---|
| GET | /api/v1/jobs |
List (filters: status, pipelineMode, repositoryConnector, outputConnector, q; sort: id, name, status, lastRun) |
| POST | /api/v1/jobs |
Create; 201 with Location and ETag
|
| GET | /api/v1/jobs/{id} |
Read: definition, state, and the caller's permissions
|
| PUT | /api/v1/jobs/{id} |
Replace the definition |
| PATCH | /api/v1/jobs/{id} |
Merge patch |
| DELETE | /api/v1/jobs/{id} |
Delete (204) |
| POST | /api/v1/jobs/{id}:start |
Start a run: 202 with the run and Location: /api/v1/job-runs/{executionId}
|
| POST | /api/v1/jobs/{id}:stop |
Stop the run (202), or mark a job with no run Finished (200) |
| POST | /api/v1/jobs/{id}:pause |
Pause the job's run: it holds before its next document. The job is marked Paused (a job with no run only that) |
| POST | /api/v1/jobs/{id}:resume |
Resume the job's paused run (409 job-not-running when there is none) |
| GET | /api/v1/jobs/{id}/executions |
The job's executions |
| GET | /api/v1/jobs/{id}/diagnosis |
AIOps root-cause analysis |
| GET | /api/v1/jobs/{id}/traces |
OpenTelemetry spans |
| GET | /api/v1/jobs/{id}/errors |
Error log entries (timeframe) |
A job: id, name, connectors (repositoryConnector, outputConnector, transformationConnector, authorityConnector), path, pipelineMode, crawlSpec, narrativization, access, optional schedule (cron, timezone, enabled, overlapPolicy, misfirePolicy, jitterSeconds, startAt, endAt, maxConsecutiveFailures), then read-only state (status, stage, documents, lastRun), scheduleState, and permissions (READ, EXECUTE, MANAGE). On input, state, scheduleState, and permissions are not accepted. Every connector named must exist and be of the right category.
| Method | Path | Purpose |
|---|---|---|
| GET | /api/v1/schedules |
List scheduled jobs (enabled, outcome, q, sort, cursor paging) |
| GET | /api/v1/schedules/overview |
Cluster scheduler configuration, counts, skippedLast24h, and upcoming fires (horizon, limit) |
| POST | /api/v1/schedules:validate-cron |
Validate cron syntax/cadence guardrail and preview nextFireTimes
|
| GET | /api/v1/jobs/{id}/schedule |
Read the job's schedule definition, live state, and next fire times (with ETag) |
| PUT | /api/v1/jobs/{id}/schedule |
Create or update the job's cron schedule (If-Match supported) |
| DELETE | /api/v1/jobs/{id}/schedule |
Remove the job's schedule (204) |
| POST | /api/v1/jobs/{id}/schedule:pause |
Pause the job's schedule (enabled = false) |
| POST | /api/v1/jobs/{id}/schedule:resume |
Resume the job's schedule and recompute nextFireAt
|
| GET | /api/v1/jobs/{id}/schedule/dispatches |
Cursor-paged scheduler dispatch audit history (outcome filter) |
| Method | Path | Purpose |
|---|---|---|
| GET | /api/v1/job-runs |
Runs in progress (jobId) |
| GET | /api/v1/job-runs/{executionId} |
One run; when it has ended, see its execution |
| POST | /api/v1/job-runs/{executionId}:stop |
Stop exactly this run: 404 if it has ended, and a newer run of the same job is never stopped |
| POST | /api/v1/job-runs/{executionId}:pause |
Pause exactly this run: it holds before its next document until :resume or :stop
|
| POST | /api/v1/job-runs/{executionId}:resume |
Resume this paused run |
| GET | /api/v1/job-runs/{executionId}/history |
The run, in progress or ended, with every change of its state; kept for the run retention |
A run has a state: RUNNING or PAUSED while in progress. With the jdbc control-plane store the runs are shared by every runtime on the database: a run of another runtime is listed, paused, resumed and stopped like a local one (its runtime notices within opencrawling.control-plane.run-heartbeat-interval), and runtimeId names the runtime that runs it. A run's history (STARTED, PAUSED, RESUMED, STOP_REQUESTED, then COMPLETED, STOPPED, FAILED or INTERRUPTED) outlives a restart.
| Method | Path | Purpose |
|---|---|---|
| GET | /api/v1/runs |
Job runs and dynamic jobs the caller may see, active and ended while they are kept, newest first (kind, state, jobId) |
| GET | /api/v1/runs/{runId} |
One run of any kind, by its run id (its execution id) |
| GET | /api/v1/runs/{runId}/history |
The run with every change of its state, from the moment it was requested |
| POST | /api/v1/runs/{runId}:stop |
Stop the run; a dynamic job waiting for approval is cancelled |
| POST | /api/v1/runs/{runId}:pause |
Pause the run: it holds before its next document until :resume or :stop
|
| POST | /api/v1/runs/{runId}:resume |
Resume the paused run |
One API for every run, whatever started it (#316, PRs #427 and #432):
-
Kinds:
JOB(a run of a pipeline job),DYNAMIC_FOLDERandDYNAMIC_DOCUMENT(a dynamic job, when dynamic jobs are on). -
One run record: a dynamic job is one run from its request to its end, identified by the run id assigned when it is filed. Its history begins with
QUEUEDorAPPROVAL_REQUESTED, then recordsAPPROVED(including the approver),STARTEDand subsequent events as applicable. -
States: one state machine applies to all run kinds. Waiting states are
QUEUED(waiting for an execution slot) andPENDING_APPROVAL; active states areRUNNINGandPAUSED; ended states areCOMPLETED,FAILED,STOPPED,INTERRUPTED,REJECTEDandEXPIRED. Only active runs hold a slot and count toward concurrency caps. A request exceeding a cap still receives429and is not left queued. Pipeline job runs are recorded when they start, so they do not useQUEUED. The dynamic-job API retains its own states and represents a queued job asRUNNING. -
Filtering:
/api/v1/runsaccepts any state in this state machine as astatefilter. -
Visibility: each run as its own API shows it. A job run is visible to those who may read its job; a dynamic job to its owner and the dynamic jobs'
admin-groups, and only to a caller signed in with the dynamic jobs' scopes. -
Links:
linkspoints at the run's job (job) or dynamic job (dynamicJob), its history and, once it ran, its execution. -
Starting and approving stay with each kind's own API:
/jobs/{id}:start,/dynamic-jobs,/dynamic-jobs/{jobId}:approve. -
Errors:
:stop,:pauseand:resumeon a run that has ended answer409 job-not-running.
| Method | Path | Purpose |
|---|---|---|
| GET | /api/v1/executions |
List, newest first (filters: jobId, status, pipelineMode; sort: startTime, endTime, durationMs, totalDocuments, jobName) |
| GET | /api/v1/executions/{executionId} |
The report: totals, timings, cost, report files, provenance |
| GET | /api/v1/executions/{executionId}/documents |
The document ledger (status, q, paged) |
| GET | /api/v1/executions/{executionId}/export?format= |
The report file: json, csv, parquet, avro (avro only when spring.opencrawling.reporting.formats.avro is on; otherwise 404) |
| POST | /api/v1/executions:query |
DuckDB SQL over the audit lake (admins) |
| Method | Path | Purpose |
|---|---|---|
| GET | /api/v1/connectors |
List (type, q); job creators get no configuration |
| POST | /api/v1/connectors |
Create (409 if the name is taken) |
| GET | /api/v1/connectors/{name} |
Read |
| PUT | /api/v1/connectors/{name} |
Replace |
| DELETE | /api/v1/connectors/{name} |
Delete; 409 while jobs use it, unless force=true
|
| POST | /api/v1/connectors/{name}:check |
Live connectivity check of a saved connector |
| POST | /api/v1/connectors:check |
The same for a definition not saved yet |
No API returns secrets. The Platform API masks them as described below, and so do the deprecated GET /api/connectors and GET /api/connectors/{type}. The deprecated POST /api/connectors and POST /api/connectors/check accept a value sent back as it was read, under the same rules; when one is refused, they answer 400, and the save's X-OpenCrawling-Error header (or the check result's message) names the setting.
-
Which keys are secret: any key that contains
secret(clientSecret,s3SecretAccessKey), keys that end inpassword,token(includingapiToken,personalAccessToken,refreshToken,accessToken),apiKey,credential,privateKey,passphrase,ticketorconnectionString(but notchangeLogToken, a position in a change log),serviceAccountJson/service_account_json(Google Drive Service Account private key JSON,#339, PR#436), and header settings (customHeaders,authorization). -
How they look: they are returned as
********. A password inside a URL (scheme://user:password@host) is returned asscheme://user:********@host, and apassword=parameter of a URL or connection string aspassword=********. -
Edited values: a value that still holds
********but is not exactly the value that was read is refused, so the mask itself is never stored as a password. -
Sending them back: a value sent back as it was read keeps the stored secret, so a client can
GET, change something andPUTwithout knowing the secrets. -
Changing the address: if the change also changes where or how the connector connects, secrets sent back as the mask are refused (
400, naming them) and must be entered again. That covers settings named like a URL, host, server or port, endpoint, bootstrap, TLS or SSL settings, and a change ofclassName. Otherwise a stored password could be sent to a server of the caller's choice, or in clear.
| Method | Path | Purpose |
|---|---|---|
| GET, PUT | /api/v1/settings/system |
Crawling and extraction settings (ETag, If-Match) |
| GET, PUT | /api/v1/settings/transport |
Internal transport (gRPC or REST) |
| POST | /api/v1/settings/transport:test-grpc |
Test the gRPC transport |
| Method | Path | Purpose |
|---|---|---|
| GET | /api/v1/system/status |
Each dependency: UP or DOWN |
| GET | /api/v1/system/throughput |
Recent throughput |
| GET | /api/v1/system/logs |
Recent log lines (admins) |
| GET | /api/v1/observability/metrics |
Throughput metrics per connector (admins) |
| Method | Path | Purpose |
|---|---|---|
| POST | /api/v1/migrations/manifoldcf:plan |
Plan a ManifoldCF migration (admins) |
| POST | /api/v1/migrations/manifoldcf:apply |
Apply it (admins) |
| GET |
/api/v1/integrations/vespa/health, /document-counts
|
Vespa (admins) |
| POST |
/api/v1/integrations/vespa:query, :deploy-bundled, :deploy-custom
|
Vespa (admins) |
| POST | /api/v1/narrativization:generate-template |
Suggest a narrativization template (admins, job creators) |
Only present when the feature is enabled; GET /api/v1 says which are.
| Feature | Endpoints |
|---|---|
Discovery (opencrawling.discovery.enabled) |
GET /api/v1/discovery/connectors, GET .../{connector}/tree, POST .../{connector}:dry-run
|
Dynamic jobs (opencrawling.jobs.dynamic.enabled) |
POST /api/v1/dynamic-jobs, POST /api/v1/dynamic-jobs:ingest-document, GET /api/v1/dynamic-jobs, GET .../{jobId}, POST .../{jobId}:stop, :approve, :reject; pause and resume through /api/v1/runs
|
Sidecars (opencrawling.sidecars.enabled) |
GET /api/v1/sidecar-stores, GET /api/v1/sidecar-stores/{connector}/sidecars
|
API=http://localhost:8080/api/v1
AUTH="Authorization: Bearer $TOKEN" # leave out when the API is open
# Create a job (safe to retry)
curl -s -X POST "$API/jobs" -H "$AUTH" -H 'Content-Type: application/json' -H 'Idempotency-Key: hr-contracts-1' \
-d '{"name":"HR contracts","repositoryConnector":"FileSystem_Local","outputConnector":"PGVector_Output",
"path":"/data/hr","pipelineMode":"rag"}'
# 201 Location: /api/v1/jobs/1763038591234
# Start it, follow it
curl -s -X POST "$API/jobs/1763038591234:start" -H "$AUTH" -H 'Idempotency-Key: hr-contracts-run-1'
# 202 {"executionId":"exec-1a2b3c4d","jobId":"1763038591234",...}
curl -s "$API/jobs/1763038591234" -H "$AUTH" | jq .state
# The audit trail of the run
curl -s "$API/executions/exec-1a2b3c4d" -H "$AUTH"
curl -s "$API/executions/exec-1a2b3c4d/documents?status=FAILED&limit=100" -H "$AUTH"
curl -s -o run.parquet "$API/executions/exec-1a2b3c4d/export?format=parquet" -H "$AUTH"
# Rename it without overwriting someone else's change
ETAG=$(curl -s -D - -o /dev/null "$API/jobs/1763038591234" -H "$AUTH" | awk -F': ' 'tolower($1)=="etag"{print $2}' | tr -d '\r')
curl -s -X PATCH "$API/jobs/1763038591234" -H "$AUTH" -H 'Content-Type: application/merge-patch+json' -H "If-Match: $ETAG" \
-d '{"name":"HR contracts (2026)"}'| Legacy | Platform API |
|---|---|
GET/POST /api/jobs, GET/DELETE /api/jobs/{id}
|
/api/v1/jobs, /api/v1/jobs/{id} (plus PUT, PATCH) |
POST /api/jobs/{id}/start, /stop, /pause
|
POST /api/v1/jobs/{id}:start, :stop, :pause
|
GET /api/jobs/permissions |
permissions on each job |
GET /api/jobs/runs, GET /api/jobs/{id}/run
|
GET /api/v1/job-runs, GET /api/v1/job-runs?jobId=
|
GET /api/reports, /api/reports/{jobId}/{executionId}
|
GET /api/v1/executions, /api/v1/executions/{executionId}
|
GET /api/reports/{jobId}/{executionId}/documents, /export
|
.../executions/{executionId}/documents, /export
|
POST /api/reports/query |
POST /api/v1/executions:query |
/api/connectors..., POST /api/connectors/check
|
/api/v1/connectors..., POST /api/v1/connectors:check
|
/api/system/settings, /api/system/settings/transport
|
/api/v1/settings/system, /api/v1/settings/transport
|
/api/system/status, /throughput, /logs
|
/api/v1/system/status, /throughput, /logs
|
/api/observability/diagnose/{jobId}, /traces/{jobId}, /errors/{jobId}, /metrics
|
/api/v1/jobs/{id}/diagnosis, /traces, /errors, /api/v1/observability/metrics
|
/api/mcf-migration/plan, /apply
|
/api/v1/migrations/manifoldcf:plan, :apply
|
/api/vespa/... |
/api/v1/integrations/vespa/... |
/api/transformation/copilot/generate |
/api/v1/narrativization:generate-template |
/api/discovery/..., /api/jobs/dynamic/..., /api/sidecars/...
|
/api/v1/discovery/..., /api/v1/dynamic-jobs/..., /api/v1/sidecar-stores/...
|
/api/auth/me |
/api/v1/me |
Not duplicated: OIS ingest (/api/v1/ingest/ois/{jobId}) and BPM events (/api/events/bpm/...) are machine-to-machine endpoints with their own authentication; /api/auth/config stays public for the admin UI.
Deprecation: the admin UI, the CLI and the Java SDK's platform() client use /api/v1, so the legacy management endpoints in the table are deprecated. Each answer of one also links to its exact successor (/api/jobs/42/start to /api/v1/jobs/42:start), and to this section with Link: <...>; rel="deprecation" (RFC 9745). They still work exactly as before, for existing integrations. What marks them:
-
Response headers: each answer carries
Deprecation: @1794700800(RFC 9745: deprecated since 2026-11-15) andLink: </api/v1/...>; rel="successor-version"naming the replacement. - Swagger UI: in "All endpoints" they are marked deprecated, with their successor.
-
SDK: its earlier clients log a warning, once per client, pointing at
platform().
Removal will be announced separately; no Sunset date is set. The endpoints listed under "Not duplicated" are not deprecated.
Every call the admin UI makes goes to /api/v1, except the public GET /api/auth/config, which it reads before sign-in. A test (PlatformApiV1UiParityTest) reads the UI's sources and checks two things: each call is a documented v1 operation, and everything the UI did before still uses one.
- Editing a job or a connector, and saving settings, use the version read with the form (
If-Match). A change made meanwhile answers412, and the page asks to reopen or reload. Deletes read the current version first. - Lists follow the cursors.
- Errors show the problem's
detailand field errors.
client.platform() is the Platform API client (package org.opencrawling.sdk.v1):
PlatformClient api = OpenCrawlingClient.builder().baseUrl("http://localhost:8080").tokenProvider(tokens).build().platform();
Versioned<Job> job = api.jobs().create(JobInput.of("HR contracts", "FileSystem_Local", "PGVector_Output", "/data/hr"),
RequestOptions.idempotent("hr-contracts-1")); // a retry returns the same job
Versioned<JobRun> run = api.jobs().start(job.value().id(), RequestOptions.idempotent("hr-run-1"));
Versioned<Job> read = api.jobs().get(job.value().id());
api.jobs().patch(read.value().id(), Map.of("path", "/data/hr/2026"), RequestOptions.ifMatch(read.etag()));
JobRunHistory finished = api.jobRuns().waitForCompletion(run.value().executionId()); // polls until terminal state
api.executions().stream(ExecutionFilter.all().withJobId(job.value().id())) // follows the cursors
.forEach(execution -> System.out.println(execution.status()));
api.sidecars().stream("Filesystem Output Connector", "contracts/") // lazy cursor stream of OIS sidecars
.forEach(sidecar -> System.out.println(sidecar.get("id")));
try {
api.jobs().start(job.value().id(), RequestOptions.none());
} catch (ApiProblemException e) {
e.code(); // "job-already-running"
e.problem().getRequestId(); // quote it when reporting a problem
}-
Coverage: one method per operation. A test (
PlatformApiV1SdkTest) runs the SDK against the runtime and checks that it reaches every operation in the API description. -
Retries: requests that are safe to repeat are repeated after network errors,
429,502,503,504and409 idempotency-in-progress, with backoff and the runtime'sRetry-After(RetryPolicy). The safe ones areGET,PUT,DELETE, aPATCHwithIf-Match, and everyPOST, because the SDK gives eachPOSTanIdempotency-Keyunless you do. Uploads are sent once.withDefaults(...)applies correlation and request ids to every call; keys andIf-Matchbelong to single calls. -
Settings & Sidecars: both
SystemSettings(settings().system()) andTransportSettings(settings().transportSettings()) keep every field the runtime sends (ObjectNode-backed), so a read, change and replace never resets the settings you did not touch.sidecars().listStores(),readPage(...), andstream(connector, prefix)provide typed sidecar store inspection and lazy cursor iteration (#362, PR#363). -
OIS Direct Ingestion (
client.ingest()):client.ingest().withToken(oisToken)pushes typedOisDocumentUPSERTandDELETEpayloads toPOST /api/v1/ingest/ois/{jobId}with automaticIdempotency-KeyandRetryPolicybackoff (#362, PR#363). -
Earlier clients: the SDK's other clients (
jobs(),connectors(),system(), ...) still use the legacy endpoints, which are deprecated; they log a warning once. Theirsystem().updateSettingschanges only the fields its model has and keeps the runtime's other settings.
opencrawling-sdk (oc-python-client-sdk, installed via pip install opencrawling-sdk on GA or from GitHub Release wheels and importable as opencrawling) provides both synchronous (OpenCrawlingClient) and asynchronous (AsyncOpenCrawlingClient) Python 3.11+ clients over httpx and Pydantic v2 covering all 74 Platform API v1 operations (including client.schedules), Versioned[T] ETags, automatic Idempotency-Key and RetryPolicy, cursor-based lazy iterators (iter / aiter), RFC 9457 ApiProblemError, lossless SystemSettings / TransportSettings, OAuth 2.0 / RFC 8693 TokenExchangeProvider, OIS direct ingestion (client.ingest.push_ois), optional gRPC Push streaming (opencrawling-sdk[grpc]), and Lakehouse DataFrame conversions (opencrawling-sdk[dataframe]). See Python Client SDK.
Every oc command uses /api/v1, with these exceptions and changes:
-
Discovery with an asserted identity:
oc discovery ... --user-principal/--user-roleskeeps the legacy endpoint. Asserted identities are a dev/test feature that the Platform API never accepts; it takes the caller from the token. -
oc connector create: still creates or updates. It creates the connector, and replaces it when the name exists. -
oc connector delete: refuses a connector that jobs use, and names them;--forcedeletes it anyway. -
oc job start: creates and starts the job with idempotency keys.--outputpicks the output connector, and--waitfollows the run until it ends. -
oc job schedule: manages recurring job cron schedules (set,get,list,pause,resume,remove,dispatches,validate). -
oc report ledger --page N: follows the cursors to page N. -
oc system settings: shows every setting.
Open the admin UI and choose API Console. It is generated from /v3/api-docs/v1, so it always matches the runtime it talks to.
- Explorer: every operation, grouped by area, searchable.
- Documentation: each operation's description, its responses, and a curl sample.
-
Request builder:
- path, query and header parameters, each with its description, and the required ones marked;
- free headers, with helpers for
Idempotency-KeyandX-Request-Id; - a body pre-filled with the documented example, checked as JSON as you type.
- Sending: Send or Ctrl+Enter. Requests carry your sign-in token; untick "Send my token" to see how the API answers an anonymous caller.
-
Response inspector:
- Body: status, time and size; the body pretty-printed; problem details shown as a readable card with the field errors; files offered as a download.
- Headers: every response header, and what the important ones mean.
-
REST: the mechanics of the answer, with one-click follow-ups:
- ETag to
If-MatchorIf-None-Match; - open the
Location; - load the next page;
- idempotent replay,
Retry-Afterand the request id.
- ETag to
-
curl: the exact request as a command, with the token as
$OC_TOKEN. - History: the last 25 requests, restorable in one click. The token is never stored.
/swagger-ui.html, "Platform API v1": the same description, with "Try it out", and "Authorize" for a token.
Import http(s)://<runtime>/v3/api-docs/v1 as an OpenAPI collection. Every operation comes with its parameters, descriptions and request examples. Set the bearer token at collection level when the API requires one.
Every operation's documentation ends with a ready curl command (x-codeSamples). It uses $OC_URL for the runtime and $OC_TOKEN for the token.
scripts/test-platform-api-v1.sh --url http://localhost:8080 [--token $TOKEN] [--crawl-path /dir] checks a running runtime:
- the index and the API description;
- connectors and secret masking;
- a job's whole lifecycle, with ETags, merge patch and idempotency;
- with
--crawl-path, a real run and its execution report; - the error format.
Without --url it boots the decoupled stack, as CI does ("Platform API v1 Test Suite").
Platform API: traceability and test evidence maps every scope item and every operation to the tests that prove it. The tests also guard the contract:
- the documented operations are exactly the implemented ones;
- every operation is fully documented;
- every documented example works;
- every admin UI call is a documented v1 operation, and nothing the UI did was lost;
- the Java SDK and Python SDK both reach every operation, and each CLI command sends the requests it should;
- every admin-only route refuses other callers;
- every deprecated legacy endpoint names its successor, and the others are not marked.
- Home
- User & Installation Guide
- Quick Start
- VM & Bare-Metal Distribution Installation
- Local Development & Fast Docker Builds
- User Guide
- Architecture
- Zero-Trust Security & ACL Management
- Security and Identity Management
- Document Authorization & OpenFGA
- Enterprise Multi-Agent Integration
- Zero-Trust Data Provider Agent & Active Ingestion
- Document Lifecycle Actions & Tombstones
- Apache Tika 4 Text Extraction
- Configuration Guide
- Native Job Scheduler
- Platform API (
/api/v1) - Platform API: Traceability
- Connectors Reference
- Alfresco Content Services (ACS) Connector
- Alfresco Process Services (APS) Connector
- Flowable Repository Connector
- Camunda Repository Connector
- OASIS CMIS Repository Connector
- Doxis Repository Connector
- M-Files Repository Connector
- SharePoint Online Repository Connector
- Google Drive Repository Connector
- Atlassian Confluence Repository Connector
- IBM FileNet P8 Repository Connector
- OpenText Documentum Repository Connector
- Zammad Repository Connector
- Hyland OpenArch Integration (Repository, Output & MCP)
- Hyland Content Innovation Cloud (CIC) Connector
- Relational Database (JDBC) Connector
- Web Repository Connector
- Apache StormCrawler Repository Connector
- gRPC Push Repository Connector
- Elasticsearch Output Connector
- Apache Solr Output Connector
- Luxir Output Connector
- Apache SeaTunnel Output Connector
- Apache Ozone Output Connector (Migration Mode)
- AWS S3 Output Connector (Migration Mode)
- Alfresco Content Services Output Connector (Migration Mode)
- Apache Iceberg Output Connector
- Doxis Output Connector
- Connector Execution Modes
- Model Context Protocol
- Apache Ozone Claim Check Store
- Lakehouse Audit Reporting & Ledger
- AI-Powered Observability (AIOps)
- Auto-Narrativization Copilot
- Maven Connector Archetypes
- Java Client SDK
- Python Client SDK
- OpenCrawling CLI
- OpenAPI Specification & Swagger UI
- gRPC Internal Transport
- Migration Guide (ManifoldCF)
- Integration Test — Decoupled Pipeline
- M-Files Integration Test
- SharePoint Online Integration Test
- Google Drive Integration Test
- Confluence Integration Test
- IBM FileNet Integration Test
- OpenText Documentum Integration Test
- Zammad Integration Test
- Hyland OpenArch Integration Test
- Hyland CIC Integration Test
- Web Crawler Test
- Elasticsearch Integration Test
- Local Docker Build Test
- Root CLI Runner Scripts Test
- Secure MCP Server Integration Test
- Narrativization Integration Test
- Reporting Integration Test
- AWS S3 Migration Integration Test
- Alfresco Migration Integration Test
- OpenFGA Authorization Integration Test
- CLI Decoupled Integration Test
- Java Client SDK Decoupled Test
- Apache Iceberg Decoupled Test
- Doxis Output Integration Test
- Platform API v1 Test Suite