Skip to content

Native Job Scheduler

Piergiorgio Lucidi edited this page Nov 19, 2026 · 1 revision

Native Job Scheduler (#197, PR #379) ⏰

OpenCrawling embeds a cluster-safe Native Job Scheduling Engine (JobSchedulerEngine) that executes recurring ingestion and migration jobs directly inside oc-runtime with no external cron daemon, Airflow bridge, or Kubernetes CronJob glue required.

It spans every layer of the platform: Flyway V5 JDBC and file persistence, 9 Platform API v1 (/api/v1) endpoints, the Java Client SDK, the Python Client SDK, the OpenCrawling CLI (oc job schedule), the ManifoldCF Migrator, and a dedicated Scheduler view in oc-admin-ui.


🏗️ Architecture & HA Cluster Safety

flowchart LR
    UI["Admin UI / CLI / SDKs"] -->|"/api/v1/schedules & /api/v1/jobs/{id}/schedule"| API["SchedulesV1Controller"]
    API --> JC["JobController + ControlPlaneStore"]
    JC --> SS["ScheduleStore (file or jdbc)"]

    R1["oc-runtime Replica A (JobSchedulerEngine)"] -->|"ControlPlaneStore.locked (oc_lock)"| DB[("PostgreSQL / H2 (oc_schedule_state, oc_schedule_dispatch)")]
    R2["oc-runtime Replica B (JobSchedulerEngine)"] -->|"ControlPlaneStore.locked (oc_lock)"| DB
    R1 -->|"startScheduled (SOURCE_SCHEDULER)"| RUN["JobRunRegistry -> JobOrchestrator"]
    RUN -->|"trigger_source = scheduler"| LEDGER["Audit Ledger & Reports"]
Loading
  • Zero duplicate fires across replicas (jdbc mode): Every poll cycle acquires the shared database lock via ControlPlaneStore.locked (SELECT ... FOR UPDATE on oc_lock), claims up to max-batch-per-tick due schedules, advances nextFireAt atomically, records the dispatch outcome in oc_schedule_dispatch, and starts the job run through JobController / JobRunRegistry.
  • Persistence (file vs jdbc):
    • file (default): stores runtime state and bounded dispatch audit records in data/schedule-state.json and data/schedule-dispatches.json (FileScheduleStore).
    • jdbc: managed by Flyway migration V5__control_plane_schedules.sql (oc_schedule_state and oc_schedule_dispatch, portable between PostgreSQL and H2, JdbcScheduleStore).
  • Retention: RunRetention automatically sweeps expired oc_schedule_dispatch entries older than opencrawling.scheduler.dispatch-retention (default 90d).
  • Startup-crawl bypass: Runtimes launched with --crawl-on-startup bypass background scheduler polling (isActiveOnThisRuntime() == false).

🕒 Cron Expressions, Timezones & Guardrails

CronScheduleParser supports:

  • 5-field POSIX/UNIX cron (e.g., */5 * * * *, 0 2 * * *, 0 8 * * MON-FRI)
  • 6-field Spring cron (with seconds, e.g., 0 0 2 * * *)
  • Standard macros: @hourly, @daily, @midnight, @weekly, @monthly, @yearly
  • Explicit IANA timezones: e.g., UTC, Europe/Rome, America/New_York (falls back to opencrawling.scheduler.default-timezone, default UTC).
  • Minimum cadence guardrail: Rejects expressions whose consecutive interval is shorter than opencrawling.scheduler.min-interval (default PT1M) with HTTP 400.

Policies & Circuit Breaker (JobScheduleConfig)

Field Values / Type Purpose
cron string (required) 5/6-field cron expression or macro
timezone IANA TZ (default UTC) Timezone used to evaluate fire times and DST transitions
enabled boolean (default true) Whether automatic ticks are active
overlapPolicy SKIP (default), COALESCE, STOP_PREVIOUS Behavior when a tick fires while the job already has an active run (COALESCE runs one catch-up immediately after finish; STOP_PREVIOUS requests clean stop and waits up to stop-previous-wait)
misfirePolicy SKIP (default), FIRE_ONCE Behavior for ticks missed past misfire-grace-period (default PT15M)
jitterSeconds int (0..3600, optional) Deterministic/bounded delay window to spread bursts across jobs
startAt / endAt ISO-8601 instant (optional) Active scheduling window bounds
maxConsecutiveFailures int (optional) Auto-pauses the schedule (AUTO_PAUSED) after N consecutive failed scheduled runs; resuming resets the failure counter

Dispatch Audit Outcomes (DispatchOutcome)

Every scheduler evaluation appends an immutable ScheduleDispatchRecord with one of:

  • STARTED — run started (executionId populated)
  • SKIPPED_OVERLAP — skipped because the job was already running (overlapPolicy=SKIP or coalesced)
  • SKIPPED_CAPACITY — skipped because max-concurrent-runs capacity was exhausted
  • MISSED_SKIPPED — missed during downtime past misfire-grace-period (misfirePolicy=SKIP)
  • REJECTED — run start rejected by runtime validation/governance
  • AUTO_PAUSED — circuit breaker tripped after reaching maxConsecutiveFailures

⚙️ Configuration (opencrawling.scheduler.*)

Environment Variable Property Default Description
OPENCRAWLING_SCHEDULER_ENABLED opencrawling.scheduler.enabled true Enable or disable background scheduler ticks on this runtime
OPENCRAWLING_SCHEDULER_POLL_INTERVAL opencrawling.scheduler.poll-interval PT5S Poll interval for claiming due schedules
OPENCRAWLING_SCHEDULER_MIN_INTERVAL opencrawling.scheduler.min-interval PT1M Shortest allowed interval between consecutive cron ticks
OPENCRAWLING_SCHEDULER_MISFIRE_GRACE_PERIOD opencrawling.scheduler.misfire-grace-period PT15M Grace window before applying misfirePolicy
OPENCRAWLING_SCHEDULER_DEFAULT_TIMEZONE opencrawling.scheduler.default-timezone UTC Fallback IANA timezone when omitted on a schedule
OPENCRAWLING_SCHEDULER_MAX_BATCH_PER_TICK opencrawling.scheduler.max-batch-per-tick 25 Max due schedules claimed per poll cycle (>= 1)
OPENCRAWLING_SCHEDULER_DISPATCH_RETENTION opencrawling.scheduler.dispatch-retention 90d Retention period for oc_schedule_dispatch audit rows
OPENCRAWLING_SCHEDULER_STOP_PREVIOUS_WAIT opencrawling.scheduler.stop-previous-wait PT2S Wait window for STOP_PREVIOUS overlap handling before coalescing

📊 Observability & Run Provenance

  • Execution Provenance: Scheduled runs stamp ExecutionProvenance.SOURCE_SCHEDULER (trigger_source = "scheduler") on every execution report, OIS metadata envelope, and Parquet/Avro audit ledger row, rendered with a Scheduled badge in the Admin UI Reports dashboard.
  • Micrometer Metrics:
    • opencrawling.scheduler.jobs.active (Gauge) — enabled schedules
    • opencrawling.scheduler.jobs.paused (Gauge) — paused schedules
    • opencrawling.scheduler.dispatches.total (Counter, tags: outcome, job_id)
    • opencrawling.scheduler.lag.seconds (Timer, tag: outcome) — delay between scheduledFor and dispatchedAt
  • OpenTelemetry Spans: Scheduler ticks annotate spans with opencrawling.schedule.outcome and opencrawling.execution.id.

🔌 Using the Scheduler Across Every Surface

1. Admin UI (oc-admin-ui)

  • Scheduler panel: Dedicated sidebar view showing Active / Paused / Auto-Paused KPIs, skippedLast24h, a 24-hour UTC fire density strip, searchable/filterable schedule table, one-click pause/resume/edit/remove actions, and a slide-out Dispatches audit drawer.
  • Schedule Modal: Accessible from either Scheduler → Add Schedule or the CalendarClock action icon on any job row in Jobs. Includes quick presets (Every 5 min, Hourly, Every 6 hours, Nightly 02:00, Weekdays 08:00, Monthly 1st 03:00), live cron validation & next-5-fire preview (POST /api/v1/schedules:validate-cron), browser/IANA timezone selector, overlap/misfire radio cards, jitter, window bounds, and circuit-breaker limit.

2. Platform API v1 (/api/v1)

All 9 endpoints enforce RBAC (READ / MANAGE job permissions), ETag / If-Match concurrency, and cursor pagination:

  • GET /api/v1/schedules
  • GET /api/v1/schedules/overview
  • POST /api/v1/schedules:validate-cron
  • GET / PUT / DELETE /api/v1/jobs/{id}/schedule
  • POST /api/v1/jobs/{id}/schedule:pause and :resume
  • GET /api/v1/jobs/{id}/schedule/dispatches

Jobs created/updated via POST/PUT/PATCH /api/v1/jobs also accept an optional schedule object and return read-only scheduleState. See Platform API → Schedules.

3. OpenCrawling CLI (oc job schedule)

# Preview and validate a cron expression in Europe/Rome
oc job schedule validate --cron "0 2 * * *" --timezone Europe/Rome --count 5

# Attach a schedule with COALESCE overlap policy and 3-failure auto-pause
oc job schedule set --id 1 --cron "0 2 * * *" --timezone Europe/Rome \
  --overlap COALESCE --misfire FIRE_ONCE --jitter 30 --max-failures 3

# Inspect, list, pause, resume, view dispatches, or remove
oc job schedule get --id 1
oc job schedule list --enabled true
oc job schedule dispatches --id 1 --outcome STARTED
oc job schedule pause --id 1
oc job schedule resume --id 1
oc job schedule remove --id 1

4. Java & Python Client SDKs

  • Java (client.platform().schedules()): list, stream, overview, validateCron, get, getIfChanged, set, delete, pause, resume, dispatches, streamDispatches (see Java Client SDK).
  • Python (client.schedules, sync & async): list, iter, overview, validate_cron, get, get_if_changed, set, delete, pause, resume, dispatches, iter_dispatches (see Python Client SDK).

5. ManifoldCF Migrator (oc-mcf-migrator)

When migrating Apache ManifoldCF jobs, CronTranslator automatically converts supported <schedule> windows into native JobScheduleConfig entries attached to migrated jobs with enabled = false (paused by default) so imported pipelines never trigger unexpectedly before administrative review.

🗺️ Documentation


🎓 Tutorials


🧪 Testing


🤝 Community

Clone this wiki locally