feat(scheduler): Add multithreaded scheduler

[rawo]

Summary

Implements KOJAK-77: adds a concurrency knob to OutboxSchedulerConfig, letting a single scheduler tick fan out to N parallel workers instead of processing one batch at a time. Each worker claims its own disjoint batch via the existing FOR UPDATE SKIP LOCKED, so no app-level coordination is needed — cross-worker and cross-instance safety is free.

• OutboxSchedulerConfig: new concurrency: Int (1-256, validated) and workerExecutorFactory: (Int) -> ExecutorService, with defaultPlatformPool (named outbox-worker-N daemon threads) and virtualThreadPool companion factory shortcuts.
• OutboxScheduler: fans out each tick to concurrency workers via the configured executor, blocking until all complete before the next interval (so ticks never overlap). concurrency=1 (default) skips the executor entirely — zero overhead, byte-for-byte the original single-worker behavior.
• okapi-spring-boot: okapi.processor.concurrency property wired through with the same validation/startup-failure behavior as the existing processor properties.

Benchmark & key finding

New OutboxSchedulerConcurrencyBenchmark (JMH) measures executorType ∈ {platform, virtual} × concurrency ∈ {1, 4, 16, 64} against Kafka's deliverBatch transport (HTTP deliverBatch isn't implemented yet — KOJAK-74 — so this ticket, formally blocked on it, benchmarks against Kafka instead).

concurrency	speedup vs. concurrency=1
4	3.6×
16	6.2×
64	6.6× (diminishing)

The ticket's virtual-thread hypothesis did not hold, and is reported as such rather than adjusted to fit: virtual threads showed no advantage over platform threads at any tested concurrency, even on JDK 25 (JEP 491, so the usual "pre-JDK-24 pinning" explanation doesn't apply) — platform was tied-or-ahead throughout. Full writeup, including why (workload never oversubscribes the platform pool at these sizes), the round-count artifact at concurrency=64, and the tuning recommendation: benchmarks/results-postopt-KOJAK-77.md.

Test plan

• Unit: OutboxSchedulerConfigTest (concurrency validation 1-256, defaultPlatformPool/virtualThreadPool factory behavior), OutboxSchedulerTest (fan-out dispatches exactly concurrency workers per tick, concurrency=1 never touches workerExecutorFactory, one worker's exception doesn't block the others)
• Integration: ConcurrentClaimTests extended with a real OutboxScheduler(concurrency=4) against both Postgres and MySQL — disjoint claims across workers and ticks, zero delivery amplification
• okapi-spring-boot: property binding + invalid-value startup-failure tests
• ./gradlew ktlintCheck clean, full ./gradlew build green

Not done / deliberately out of scope

• HTTP-transport benchmark numbers (blocked on KOJAK-74)
• Splitting processNext()'s single transaction to shorten lock hold time during delivery I/O — tracked separately as KOJAK-79 (decision-pending; needs actual lock-contention metrics from a high-concurrency run before deciding implement-vs-reject, which this PR's benchmark didn't specifically capture)

Refs: KOJAK-77 (blocked-by KOJAK-73 ✅ done, KOJAK-74 ⏳ To Do; blocks KOJAK-79)

[Copilot]

Pull request overview

Adds configurable parallelism to the outbox scheduler so a single tick can execute multiple independent processNext() workers concurrently (disjoint row claims via FOR UPDATE SKIP LOCKED), and wires the knob through Spring Boot with tests/benchmarks/documentation.

Changes:

• Introduces OutboxSchedulerConfig.concurrency (1–256) and workerExecutorFactory with platform/virtual thread pool helpers.
• Updates OutboxScheduler to fan out work per tick and wait for all workers before the next interval.
• Adds JMH benchmark + results writeup, updates docs, and extends integration/unit tests for concurrency behavior.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
README.md	Documents scheduler concurrency benchmark results and tuning guidance.
okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorAutoConfigurationTest.kt	Adds Spring property binding coverage for okapi.processor.concurrency and invalid-value startup failure.
okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt	Adds concurrency property with validation.
okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt	Wires Spring properties into OutboxSchedulerConfig(concurrency=...).
okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/concurrency/ConcurrentClaimTests.kt	Adds integration test running real OutboxScheduler(concurrency=4) ensuring no amplification and disjoint claims.
okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerTest.kt	Adds unit tests for fan-out, zero-overhead concurrency=1, and worker exception isolation.
okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfigTest.kt	Adds validation tests and executor factory behavior tests.
okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfig.kt	Adds concurrency + executor factory to scheduler config and companion factories.
okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt	Implements per-tick worker fan-out and worker pool shutdown.
okapi-benchmarks/src/jmh/kotlin/com/softwaremill/okapi/benchmarks/OutboxSchedulerConcurrencyBenchmark.kt	Adds JMH benchmark for scheduler fan-out using platform vs virtual executors.
benchmarks/scheduler-concurrency.json	Stores raw JMH results for the new benchmark.
benchmarks/results-postopt-KOJAK-77.md	Adds benchmark analysis/writeup for KOJAK-77.
benchmarks/README.md	Documents the new benchmark and interpretation caveats.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt:101

• The warning/error text in shutdownAndAwait() is misleading:

• The timeout warning says "worker threads" even though this helper is used for both the scheduler executor and the worker executor.
• The InterruptedException branch logs "Executor terminated" but the executor wasn't terminated; the shutdown wait was interrupted.

Consider updating these messages to be accurate and include the executor type for troubleshooting.

                logger.warn("Executor did not terminate after shutdownNow(); some worker threads may still be running")
            }
        } catch (e: InterruptedException) {
            Thread.currentThread().interrupt()
            logger.warn("Interrupted while awaiting executor termination; forcing shutdownNow()", e)

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt:165

• awaitWorker swallows interrupts during normal operation to avoid overlapping ticks, but it currently clears the thread's interrupt status permanently. This can hide cancellation signals and makes the scheduler thread violate the common contract of preserving interrupts. Record the interrupt and restore it after the worker completes (while still continuing to wait).

            } catch (e: InterruptedException) {
                if (!running.get()) {
                    Thread.currentThread().interrupt()
                    logger.warn("Interrupted while awaiting outbox worker during shutdown", e)
                    return

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.