Repository navigation
Security and Identity Management
OpenCrawling does not store users or passwords. It trusts an OpenID Connect identity provider (Keycloak, Microsoft Entra ID, Okta, or any other) and checks the bearer token on every request:
- The REST API and the MCP endpoints validate the token's signature, expiry, issuer and audience (OAuth2 resource server).
-
The admin UI signs users in with the authorization code flow with PKCE (
S256viaoidc-client-ts), as a public client without a secret, and sends their access token with every API call (including report exports). - Management authorization decides on the verified caller who may see, run and change which jobs, and who may use the admin-only pages. See Configuration Guide, Management Authorization.
- Document authorization filters search results by the same caller. See Document Authorization.
This guide sets up the identity provider. The full list of settings is in Configuration Guide, Caller Identity.
opencrawling:
security:
identity:
mode: jwt # validate bearer tokens
protect-api: true # on /api/** too, not only the MCP endpoints
admin-ui:
client-id: opencrawling-admin-ui # the UI's public client; blank: the UI does not sign in
scope: openid profile
management:
enabled: true
admin-groups: opencrawling-admin
job-creator-groups: hr-team,finance-team
spring:
security:
oauth2:
resourceserver:
jwt:
issuer-uri: https://idp.example.com/realms/opencrawling
audiences: opencrawling-api-
Audience: always set
audiences. Without it, any valid token from the issuer is accepted, including tokens issued for other applications. -
Issuer as the browser sees it: tokens carry the issuer URL the user signed in at. When the runtime reaches the identity provider at another address (inside a Docker network, for example), keep
issuer-uriat the browser's URL and pointjwk-set-uriat the internal one. The runtime then fetches the signing keys there and still checks the issuer. If the admin UI must sign in at yet another URL, setadmin-ui.issuer. -
Groups: the caller's groups come from the claims in
opencrawling.security.identity.jwt.group-claims(defaultrealm_access.roles,groups). Group names inadmin-groups,job-creator-groupsand jobaccessblocks must match those values. -
Who the user is: the first non-blank claim of
principal-claims(defaultpreferred_username,sub).
The admin UI asks the runtime for its sign-in settings (GET /api/auth/config, public), signs the user in, then reads GET /api/auth/me (the caller, admin, canCreateJobs) and GET /api/jobs/permissions (what the caller may do with each job). It hides pages and buttons the user may not use (and forwards /auth/callback to the SPA when served from oc-runtime). The runtime enforces the same rules either way.
The repository has a ready Keycloak 26 for local evaluation: docker/docker-compose-with-keycloak.yml with the realm docker/keycloak/opencrawling-realm.json. It runs in development mode over plain HTTP. Do not use it in production.
-
Client
opencrawling-cli: public, device authorization grant only, foroc auth login. -
Client
opencrawling-automation: confidential, client credentials grant only, for the SDK and CI. Its service account has the realm rolehr-team. The realm ships without its secret: Keycloak generates one on import, which you read in the admin console (Clients,opencrawling-automation, Credentials). -
Client
opencrawling-admin-ui: public, standard flow only, PKCES256required, redirect URIshttp://localhost:3000/auth/callback(the UI container),http://localhost:8080/auth/callback(the UI served by the runtime) andhttp://localhost:5173/auth/callback(npm run dev). An audience mapper addsopencrawling-apito its access tokens. -
Realm roles:
opencrawling-admin,hr-team,finance-team. Keycloak puts them inrealm_access.roles, which the runtime reads by default. -
Demo users without passwords:
hr_admin(hr-team),finance_admin(finance-team),global_admin(opencrawling-admin).
Start the infrastructure first (docker compose up -d), then:
export KC_BOOTSTRAP_ADMIN_USERNAME=admin
export KC_BOOTSTRAP_ADMIN_PASSWORD='<choose a password>'
docker compose -f docker-compose-apps.yml -f docker/docker-compose-with-keycloak.yml up -dThe overlay adds Keycloak on http://localhost:8180 and configures the runtime: jwt mode with protect-api, the UI client, management authorization with opencrawling-admin as admins and hr-team/finance-team as job creators, the issuer http://localhost:8180/realms/opencrawling, keys fetched from keycloak:8080 inside the network, and the audience opencrawling-api.
The realm ships without passwords. Set one per user; kcadm.sh prompts for each password, so none ends up in your shell history:
docker exec -it opencrawling-keycloak /opt/keycloak/bin/kcadm.sh config credentials \
--server http://localhost:8080 --realm master --user "$KC_BOOTSTRAP_ADMIN_USERNAME"
for user in hr_admin finance_admin global_admin; do
docker exec -it opencrawling-keycloak /opt/keycloak/bin/kcadm.sh set-password -r opencrawling --username "$user"
doneYou can also set them in the admin console (http://localhost:8180, realm opencrawling, Users).
Open http://localhost:3000 and sign in:
| User | Sees | May do |
|---|---|---|
global_admin |
every job and every page | everything |
hr_admin |
the jobs they own or are listed on; Connectors (without configuration) and Narrativization | create jobs, run and edit their own |
finance_admin |
the same for their own jobs | the same; never sees HR's jobs |
Jobs that existed before authorization was turned on have no owner. Only global_admin sees them, until it assigns them by editing their access block.
oc auth login # the runtime from `oc config set`, or --server-url
oc auth status # who you are to the runtime: user, groups, admin, may create jobs
oc auth logout # revokes the session at the identity provider and forgets itoc auth login uses the OAuth 2.0 device authorization grant: it shows a URL and a code, you confirm them in any browser, and the CLI receives a token. The CLI never sees your password. It finds the identity provider through the runtime (GET /api/auth/config); --issuer names another, and --client-id (default opencrawling-cli) the public client, which must allow the device grant.
The sign-in is kept in ~/.oc/credentials.json (or $OC_CONFIG_DIR), readable only by you, and renewed with the refresh token as needed. It is sent only to the server it was made for. Commands authenticate with, first match wins: --token, client credentials from the environment, the stored sign-in, the bearer token from oc config set.
In CI or scripts, give the CLI a confidential client through the environment, never on the command line:
export OPENCRAWLING_CLIENT_ID=opencrawling-automation
export OPENCRAWLING_CLIENT_SECRET=... # from your CI's secret store
export OPENCRAWLING_TOKEN_URL=... # optional: found through the runtime's issuer otherwise
export OPENCRAWLING_CLIENT_SCOPE=... # optional
oc job listThe Java SDK does the same with OpenCrawlingClient.builder().clientCredentials(tokenUrl, clientId, clientSecret, scope), or with the Spring Boot properties opencrawling.client.oauth2.token-uri (or token-endpoint), client-id, client-secret, scope, and audience (plus opencrawling.client.retry.* for RetryPolicy). The token is cached and renewed before it expires. A service account acts as itself: give it the groups it needs (job-creator-groups, or a job's allowedGroups), like any user.
When an AI agent orchestrator, MCP bridge, or intermediary microservice calls OpenCrawling on behalf of an end user (#362, PR #363), use TokenExchangeProvider (urn:ietf:params:oauth:grant-type:token-exchange) via OpenCrawlingClient.builder().tokenExchange(tokenUrl, clientId, clientSecret, subjectTokenSupplier, audience):
- Exchanges the caller's incoming subject JWT for an OpenCrawling-scoped access token (
audience: opencrawling-api), so management authorization and document-level ACLs evaluate against the end user's principal and groups rather than a broad service account. - Caches exchanged tokens per
(SHA-256(subjectToken), SHA-256(actorToken))digest pair, renews 30 seconds before expiry, and automatically re-exchanges when the caller's subject token changes. In Spring Boot applications, registering aTokenProvider@Bean(such asTokenExchangeProvider) automatically wires it into the auto-configuredOpenCrawlingClient.
Register two applications, or one that plays both roles:
-
The API: under Expose an API, set the Application ID URI (for example
api://<api-client-id>) and add a scope such asaccess_as_user. In its manifest, set"accessTokenAcceptedVersion": 2so access tokens carry the v2 issuer. For groups, either add thegroupsclaim under Token configuration (values are group object IDs) or define App roles and assign them (they arrive inroles). -
The admin UI: add a Single-page application platform with the redirect URI
https://<ui-host>/auth/callback. Single-page clients use PKCE and have no secret. Grant it the API'saccess_as_userpermission.
Then configure the runtime:
opencrawling.security.identity:
admin-ui:
client-id: <ui-client-id>
scope: openid profile api://<api-client-id>/access_as_user # so the access token is for your API
jwt:
group-claims: roles,groups
spring.security.oauth2.resourceserver.jwt:
issuer-uri: https://login.microsoftonline.com/<tenant-id>/v2.0
audiences: <api-client-id>The UI must ask for your API's scope: without it Entra ID issues an access token for Microsoft Graph, which the runtime rejects. With group object IDs in groups, use those IDs in admin-groups, job-creator-groups and job access blocks, or use app roles with readable names.
- Use a custom authorization server (for example
default, issuerhttps://<your-org>.okta.com/oauth2/default). Tokens from the org authorization server are meant for Okta itself and cannot be validated by other applications. - Create an OIDC single-page application with grant type Authorization Code (PKCE is required for single-page apps), sign-in redirect URI
https://<ui-host>/auth/callback, and sign-out redirect URIhttps://<ui-host>/. - On the authorization server, add a
groupsclaim to access tokens (for example, filter "Matches regex.*", or a narrower one).
opencrawling.security.identity:
admin-ui:
client-id: <spa-client-id>
spring.security.oauth2.resourceserver.jwt:
issuer-uri: https://<your-org>.okta.com/oauth2/default
audiences: api://default # the authorization server's audienceOkta access tokens name the user in sub (the login), which the default principal-claims fall back to.
- HTTPS everywhere: the identity provider, the admin UI and the runtime. Redirect URIs must use the UI's real HTTPS address.
-
Audience set:
spring.security.oauth2.resourceserver.jwt.audiences. - Short-lived access tokens: the UI renews them with the refresh token before they expire; a failed renewal sends the user back to sign in.
-
Group names agreed:
admin-groupsandjob-creator-groupsexist in the provider, and jobaccessblocks use the same names. - Automation secrets: client secrets come from a secret store or the environment, never from committed files or command lines.
- Home
- User & Installation Guide
- Quick Start
- VM & Bare-Metal Distribution Installation
- Local Development & Fast Docker Builds
- User Guide
- Architecture
- Zero-Trust Security & ACL Management
- Security and Identity Management
- Document Authorization & OpenFGA
- Enterprise Multi-Agent Integration
- Zero-Trust Data Provider Agent & Active Ingestion
- Document Lifecycle Actions & Tombstones
- Apache Tika 4 Text Extraction
- Configuration Guide
- Native Job Scheduler
- Platform API (
/api/v1) - Platform API: Traceability
- Connectors Reference
- Alfresco Content Services (ACS) Connector
- Alfresco Process Services (APS) Connector
- Flowable Repository Connector
- Camunda Repository Connector
- OASIS CMIS Repository Connector
- Doxis Repository Connector
- M-Files Repository Connector
- SharePoint Online Repository Connector
- Google Drive Repository Connector
- Atlassian Confluence Repository Connector
- IBM FileNet P8 Repository Connector
- OpenText Documentum Repository Connector
- Zammad Repository Connector
- Hyland OpenArch Integration (Repository, Output & MCP)
- Hyland Content Innovation Cloud (CIC) Connector
- Relational Database (JDBC) Connector
- Web Repository Connector
- Apache StormCrawler Repository Connector
- gRPC Push Repository Connector
- Elasticsearch Output Connector
- Apache Solr Output Connector
- Luxir Output Connector
- Apache SeaTunnel Output Connector
- Apache Ozone Output Connector (Migration Mode)
- AWS S3 Output Connector (Migration Mode)
- Alfresco Content Services Output Connector (Migration Mode)
- Apache Iceberg Output Connector
- Doxis Output Connector
- Connector Execution Modes
- Model Context Protocol
- Apache Ozone Claim Check Store
- Lakehouse Audit Reporting & Ledger
- AI-Powered Observability (AIOps)
- Auto-Narrativization Copilot
- Maven Connector Archetypes
- Java Client SDK
- Python Client SDK
- OpenCrawling CLI
- OpenAPI Specification & Swagger UI
- gRPC Internal Transport
- Migration Guide (ManifoldCF)
- Integration Test — Decoupled Pipeline
- M-Files Integration Test
- SharePoint Online Integration Test
- Google Drive Integration Test
- Confluence Integration Test
- IBM FileNet Integration Test
- OpenText Documentum Integration Test
- Zammad Integration Test
- Hyland OpenArch Integration Test
- Hyland CIC Integration Test
- Web Crawler Test
- Elasticsearch Integration Test
- Local Docker Build Test
- Root CLI Runner Scripts Test
- Secure MCP Server Integration Test
- Narrativization Integration Test
- Reporting Integration Test
- AWS S3 Migration Integration Test
- Alfresco Migration Integration Test
- OpenFGA Authorization Integration Test
- CLI Decoupled Integration Test
- Java Client SDK Decoupled Test
- Apache Iceberg Decoupled Test
- Doxis Output Integration Test
- Platform API v1 Test Suite