Repository navigation
Native Job Scheduler
OpenCrawling embeds a cluster-safe Native Job Scheduling Engine (JobSchedulerEngine) that executes recurring ingestion and migration jobs directly inside oc-runtime with no external cron daemon, Airflow bridge, or Kubernetes CronJob glue required.
It spans every layer of the platform: Flyway V5 JDBC and file persistence, 9 Platform API v1 (/api/v1) endpoints, the Java Client SDK, the Python Client SDK, the OpenCrawling CLI (oc job schedule), the ManifoldCF Migrator, and a dedicated Scheduler view in oc-admin-ui.
flowchart LR
UI["Admin UI / CLI / SDKs"] -->|"/api/v1/schedules & /api/v1/jobs/{id}/schedule"| API["SchedulesV1Controller"]
API --> JC["JobController + ControlPlaneStore"]
JC --> SS["ScheduleStore (file or jdbc)"]
R1["oc-runtime Replica A (JobSchedulerEngine)"] -->|"ControlPlaneStore.locked (oc_lock)"| DB[("PostgreSQL / H2 (oc_schedule_state, oc_schedule_dispatch)")]
R2["oc-runtime Replica B (JobSchedulerEngine)"] -->|"ControlPlaneStore.locked (oc_lock)"| DB
R1 -->|"startScheduled (SOURCE_SCHEDULER)"| RUN["JobRunRegistry -> JobOrchestrator"]
RUN -->|"trigger_source = scheduler"| LEDGER["Audit Ledger & Reports"]
-
Zero duplicate fires across replicas (
jdbcmode): Every poll cycle acquires the shared database lock viaControlPlaneStore.locked(SELECT ... FOR UPDATEonoc_lock), claims up tomax-batch-per-tickdue schedules, advancesnextFireAtatomically, records the dispatch outcome inoc_schedule_dispatch, and starts the job run throughJobController/JobRunRegistry. -
Persistence (
filevsjdbc):-
file(default): stores runtime state and bounded dispatch audit records indata/schedule-state.jsonanddata/schedule-dispatches.json(FileScheduleStore). -
jdbc: managed by Flyway migrationV5__control_plane_schedules.sql(oc_schedule_stateandoc_schedule_dispatch, portable between PostgreSQL and H2,JdbcScheduleStore).
-
-
Retention:
RunRetentionautomatically sweeps expiredoc_schedule_dispatchentries older thanopencrawling.scheduler.dispatch-retention(default90d). -
Startup-crawl bypass: Runtimes launched with
--crawl-on-startupbypass background scheduler polling (isActiveOnThisRuntime() == false).
CronScheduleParser supports:
-
5-field POSIX/UNIX cron (e.g.,
*/5 * * * *,0 2 * * *,0 8 * * MON-FRI) -
6-field Spring cron (with seconds, e.g.,
0 0 2 * * *) -
Standard macros:
@hourly,@daily,@midnight,@weekly,@monthly,@yearly -
Explicit IANA timezones: e.g.,
UTC,Europe/Rome,America/New_York(falls back toopencrawling.scheduler.default-timezone, defaultUTC). -
Minimum cadence guardrail: Rejects expressions whose consecutive interval is shorter than
opencrawling.scheduler.min-interval(defaultPT1M) with HTTP400.
| Field | Values / Type | Purpose |
|---|---|---|
cron |
string (required) |
5/6-field cron expression or macro |
timezone |
IANA TZ (default UTC) |
Timezone used to evaluate fire times and DST transitions |
enabled |
boolean (default true) |
Whether automatic ticks are active |
overlapPolicy |
SKIP (default), COALESCE, STOP_PREVIOUS
|
Behavior when a tick fires while the job already has an active run (COALESCE runs one catch-up immediately after finish; STOP_PREVIOUS requests clean stop and waits up to stop-previous-wait) |
misfirePolicy |
SKIP (default), FIRE_ONCE
|
Behavior for ticks missed past misfire-grace-period (default PT15M) |
jitterSeconds |
int (0..3600, optional) |
Deterministic/bounded delay window to spread bursts across jobs |
startAt / endAt
|
ISO-8601 instant (optional) | Active scheduling window bounds |
maxConsecutiveFailures |
int (optional) |
Auto-pauses the schedule (AUTO_PAUSED) after N consecutive failed scheduled runs; resuming resets the failure counter |
Every scheduler evaluation appends an immutable ScheduleDispatchRecord with one of:
-
STARTED— run started (executionIdpopulated) -
SKIPPED_OVERLAP— skipped because the job was already running (overlapPolicy=SKIPor coalesced) -
SKIPPED_CAPACITY— skipped becausemax-concurrent-runscapacity was exhausted -
MISSED_SKIPPED— missed during downtime pastmisfire-grace-period(misfirePolicy=SKIP) -
REJECTED— run start rejected by runtime validation/governance -
AUTO_PAUSED— circuit breaker tripped after reachingmaxConsecutiveFailures
| Environment Variable | Property | Default | Description |
|---|---|---|---|
OPENCRAWLING_SCHEDULER_ENABLED |
opencrawling.scheduler.enabled |
true |
Enable or disable background scheduler ticks on this runtime |
OPENCRAWLING_SCHEDULER_POLL_INTERVAL |
opencrawling.scheduler.poll-interval |
PT5S |
Poll interval for claiming due schedules |
OPENCRAWLING_SCHEDULER_MIN_INTERVAL |
opencrawling.scheduler.min-interval |
PT1M |
Shortest allowed interval between consecutive cron ticks |
OPENCRAWLING_SCHEDULER_MISFIRE_GRACE_PERIOD |
opencrawling.scheduler.misfire-grace-period |
PT15M |
Grace window before applying misfirePolicy
|
OPENCRAWLING_SCHEDULER_DEFAULT_TIMEZONE |
opencrawling.scheduler.default-timezone |
UTC |
Fallback IANA timezone when omitted on a schedule |
OPENCRAWLING_SCHEDULER_MAX_BATCH_PER_TICK |
opencrawling.scheduler.max-batch-per-tick |
25 |
Max due schedules claimed per poll cycle (>= 1) |
OPENCRAWLING_SCHEDULER_DISPATCH_RETENTION |
opencrawling.scheduler.dispatch-retention |
90d |
Retention period for oc_schedule_dispatch audit rows |
OPENCRAWLING_SCHEDULER_STOP_PREVIOUS_WAIT |
opencrawling.scheduler.stop-previous-wait |
PT2S |
Wait window for STOP_PREVIOUS overlap handling before coalescing |
-
Execution Provenance: Scheduled runs stamp
ExecutionProvenance.SOURCE_SCHEDULER(trigger_source = "scheduler") on every execution report, OIS metadata envelope, and Parquet/Avro audit ledger row, rendered with aScheduledbadge in the Admin UI Reports dashboard. -
Micrometer Metrics:
-
opencrawling.scheduler.jobs.active(Gauge) — enabled schedules -
opencrawling.scheduler.jobs.paused(Gauge) — paused schedules -
opencrawling.scheduler.dispatches.total(Counter, tags:outcome,job_id) -
opencrawling.scheduler.lag.seconds(Timer, tag:outcome) — delay betweenscheduledForanddispatchedAt
-
-
OpenTelemetry Spans: Scheduler ticks annotate spans with
opencrawling.schedule.outcomeandopencrawling.execution.id.
-
Scheduler panel: Dedicated sidebar view showing Active / Paused / Auto-Paused KPIs,
skippedLast24h, a 24-hour UTC fire density strip, searchable/filterable schedule table, one-click pause/resume/edit/remove actions, and a slide-out Dispatches audit drawer. -
Schedule Modal: Accessible from either Scheduler → Add Schedule or the CalendarClock action icon on any job row in Jobs. Includes quick presets (
Every 5 min,Hourly,Every 6 hours,Nightly 02:00,Weekdays 08:00,Monthly 1st 03:00), live cron validation & next-5-fire preview (POST /api/v1/schedules:validate-cron), browser/IANA timezone selector, overlap/misfire radio cards, jitter, window bounds, and circuit-breaker limit.
All 9 endpoints enforce RBAC (READ / MANAGE job permissions), ETag / If-Match concurrency, and cursor pagination:
GET /api/v1/schedulesGET /api/v1/schedules/overviewPOST /api/v1/schedules:validate-cronGET / PUT / DELETE /api/v1/jobs/{id}/schedule-
POST /api/v1/jobs/{id}/schedule:pauseand:resume GET /api/v1/jobs/{id}/schedule/dispatches
Jobs created/updated via POST/PUT/PATCH /api/v1/jobs also accept an optional schedule object and return read-only scheduleState. See Platform API → Schedules.
# Preview and validate a cron expression in Europe/Rome
oc job schedule validate --cron "0 2 * * *" --timezone Europe/Rome --count 5
# Attach a schedule with COALESCE overlap policy and 3-failure auto-pause
oc job schedule set --id 1 --cron "0 2 * * *" --timezone Europe/Rome \
--overlap COALESCE --misfire FIRE_ONCE --jitter 30 --max-failures 3
# Inspect, list, pause, resume, view dispatches, or remove
oc job schedule get --id 1
oc job schedule list --enabled true
oc job schedule dispatches --id 1 --outcome STARTED
oc job schedule pause --id 1
oc job schedule resume --id 1
oc job schedule remove --id 1-
Java (
client.platform().schedules()):list,stream,overview,validateCron,get,getIfChanged,set,delete,pause,resume,dispatches,streamDispatches(see Java Client SDK). -
Python (
client.schedules, sync & async):list,iter,overview,validate_cron,get,get_if_changed,set,delete,pause,resume,dispatches,iter_dispatches(see Python Client SDK).
When migrating Apache ManifoldCF jobs, CronTranslator automatically converts supported <schedule> windows into native JobScheduleConfig entries attached to migrated jobs with enabled = false (paused by default) so imported pipelines never trigger unexpectedly before administrative review.
- Home
- User & Installation Guide
- Quick Start
- VM & Bare-Metal Distribution Installation
- Local Development & Fast Docker Builds
- User Guide
- Architecture
- Zero-Trust Security & ACL Management
- Security and Identity Management
- Document Authorization & OpenFGA
- Enterprise Multi-Agent Integration
- Zero-Trust Data Provider Agent & Active Ingestion
- Document Lifecycle Actions & Tombstones
- Apache Tika 4 Text Extraction
- Configuration Guide
- Native Job Scheduler
- Platform API (
/api/v1) - Platform API: Traceability
- Connectors Reference
- Alfresco Content Services (ACS) Connector
- Alfresco Process Services (APS) Connector
- Flowable Repository Connector
- Camunda Repository Connector
- OASIS CMIS Repository Connector
- Doxis Repository Connector
- M-Files Repository Connector
- SharePoint Online Repository Connector
- Google Drive Repository Connector
- Atlassian Confluence Repository Connector
- IBM FileNet P8 Repository Connector
- OpenText Documentum Repository Connector
- Zammad Repository Connector
- Hyland OpenArch Integration (Repository, Output & MCP)
- Hyland Content Innovation Cloud (CIC) Connector
- Relational Database (JDBC) Connector
- Web Repository Connector
- Apache StormCrawler Repository Connector
- gRPC Push Repository Connector
- Elasticsearch Output Connector
- Apache Solr Output Connector
- Luxir Output Connector
- Apache SeaTunnel Output Connector
- Apache Ozone Output Connector (Migration Mode)
- AWS S3 Output Connector (Migration Mode)
- Alfresco Content Services Output Connector (Migration Mode)
- Apache Iceberg Output Connector
- Doxis Output Connector
- Connector Execution Modes
- Model Context Protocol
- Apache Ozone Claim Check Store
- Lakehouse Audit Reporting & Ledger
- AI-Powered Observability (AIOps)
- Auto-Narrativization Copilot
- Maven Connector Archetypes
- Java Client SDK
- Python Client SDK
- OpenCrawling CLI
- OpenAPI Specification & Swagger UI
- gRPC Internal Transport
- Migration Guide (ManifoldCF)
- Integration Test — Decoupled Pipeline
- M-Files Integration Test
- SharePoint Online Integration Test
- Google Drive Integration Test
- Confluence Integration Test
- IBM FileNet Integration Test
- OpenText Documentum Integration Test
- Zammad Integration Test
- Hyland OpenArch Integration Test
- Hyland CIC Integration Test
- Web Crawler Test
- Elasticsearch Integration Test
- Local Docker Build Test
- Root CLI Runner Scripts Test
- Secure MCP Server Integration Test
- Narrativization Integration Test
- Reporting Integration Test
- AWS S3 Migration Integration Test
- Alfresco Migration Integration Test
- OpenFGA Authorization Integration Test
- CLI Decoupled Integration Test
- Java Client SDK Decoupled Test
- Apache Iceberg Decoupled Test
- Doxis Output Integration Test
- Platform API v1 Test Suite