Repository navigation
oncebox 2.0.0
The library is now named by its two mechanisms, outbox and inbox, every time is taken from the database
clock instead of the JVM, and the inbox gets a second, at-most-once consumer. Configuration keys, tables, metrics,
bean names and message headers change, so this is a major version.
Highlights
- Two idempotent consumers.
TransactionalIdempotentConsumer(effectively-once: the event id and the operation
share one transaction) andNonTransactionalIdempotentConsumer(at-most-once: the id is committed before the
operation runs, for side effects that must not repeat, such as sending an email). A listener chooses one with
@TransactionalConsumeror@NonTransactionalConsumer. - Time zone independent. Stuck-batch recovery, retry scheduling and every cleanup compare against the database
clock, so instances in different time zones agree on what is stuck or expired. - Fencing of captured batches. Each capture writes a lock token, and a poller that stalled past stuck recovery
can no longer overwrite the outcome of the poller that captured the batch after it. - DLQ REST API errors as RFC 9457
ProblemDetail, with a type URI per error and validation errors listed by field.
Breaking changes
Naming
- Properties:
oncebox.publisher.*tooncebox.outbox.*,oncebox.consumer.*tooncebox.inbox.*. - Annotation:
@EnableOutboxto@EnableOncebox. - Classes:
OutboxIdempotentConsumertoIdempotentConsumer,OutboxEventIdExtractortoEventIdExtractor,
OutboxHeaderstoOnceboxHeaders,OutboxHeadersUtilstoOnceboxHeadersUtils. - Packages:
*.publisherto*.outbox,*.consumerto*.inbox;OutboxEvent,EventStatusand
SenderResultmove fromcore.outbox.domaintocore.outbox. - Artifacts:
oncebox-consumer-cachetooncebox-inbox-cache;oncebox-messagingis merged intooncebox-core. - Tables:
outbox_consumed_eventstoinbox_events,outbox_jobstooncebox_jobs. - Metrics:
consumed_outbox_*toinbox_*;outbox_*_tasks_total,outbox_task_processing_durationand
outbox_polling_delay_millisecondstooncebox_*; thetask_typevalueoutbox-consumed-cleanupis now
inbox-cleanup. The bundled Grafana dashboards are renamed accordingly. - Message headers:
outbox_event_{id,type,payload_type}tooncebox_event_*. - Bean names: the listener container factories and message converters are
inbox*, shared beans such as
onceboxJdbcTemplateandonceboxScheduledExecutorServiceareoncebox*.
Inbox
IdempotentConsumerhas two implementations, registered asinboxTransactionalIdempotentConsumerand
inboxNonTransactionalIdempotentConsumer. Neither is@Primary: injectingIdempotentConsumerwithout
@TransactionalConsumer,@NonTransactionalConsumeror@Qualifierfails the context.- An implementation is replaced through the beans
inboxUndecoratedTransactionalIdempotentConsumerand
inboxUndecoratedNonTransactionalIdempotentConsumer. - Within one service an event is handled by one consumer: the inbox records an event by its id alone, whichever
consumer recorded it.
Database
- Every time is set by the database. The time columns are PostgreSQL
TIMESTAMPTZ, MySQLDATETIME(6)holding UTC
and OracleTIMESTAMP WITH TIME ZONE, with defaults from the database clock; inserts do not write them. - MySQL 8.0.13 or newer is required, for the expression defaults.
outbox_eventshas a nullablelock_tokencolumn.- The starter no longer needs a
Clockbean. Only the DLQ API error handler reads one: the applicationClockwhen
there is exactly one, the UTC system clock otherwise.
API for custom implementations
- The repositories, managers,
DefaultOutboxDlqEventMapper,JacksonOutboxSerializer,DefaultOutboxProcessor,
DefaultOnceboxJobCreateCommandand the repository factories no longer take aClock. - Repository methods take a
Duration ageinstead of anInstant threshold:deleteBatchOlderThan,
deleteBatchByStatusOlderThan,updateBatchStatusByStatusOlderThan. - The outcome of a batch is written by
updateBatchStatusByLockTokenandpartiallyUpdateBatchByLockToken;
OutboxManager.loadBatchandfinalizeBatchtake the lock token. OutboxManager.finalizeBatchtakes aFunction<Integer, Duration>giving the delay before the next retry;
OutboxEventcarries it asretryDelay.- A new event is made by
new OutboxEvent(id, eventType, payloadType, payload); its times arenulluntil it is read
from the database. - New methods:
OutboxRepository.saveBatchWithCreatedAtandOutboxManager.saveBatchWithCreatedAt; in
AbstractOutboxRepositoryupdateBatchStatusByLockTokenSql,partiallyUpdateBatchByLockTokenSqland
setCreatedAt; inAbstractOutboxDlqApiRepositorycurrentTimestampSql. SqlIdHelperisSqlUuidHelper. A MySQL repository built by hand takesMySqlResultSetMapper.OutboxManager.loadBatch(EventStatus, int)andOutboxDlqManager.loadAndLockBatchrequire a transaction of the
caller (MANDATORY);finalizeBatchrejects a negativemaxRetryCount.moved_atis set by the database:OutboxDlqRepository.saveBatchdoes not write it, and events made by
OutboxDlqEventMapper.toDlqEventcarry anullmovedAt.InboxRepository.saveIfAbsent(Set)andOutboxDlqRepository.saveBatchanddeleteBatchthrow
NullPointerExceptiononnull.
DLQ REST API
- Errors are
application/problem+json(RFC 9457) with atypeunderhttps://oncebox.io/errors/, the request path
asinstanceand a timestamp. Validation errors come as a list of field and message pairs. - An event in
IN_PROCESSis answered with 409 Conflict instead of 400, andIN_PROCESSis rejected as a target
status.
Startup checks
- The outbox refuses to start when
sender.emergency-timeoutis not below
stuck-recovery.max-batch-processing-time: a batch still being sent would otherwise be recovered as stuck and
published twice.
New
@TransactionalConsumerand@NonTransactionalConsumerqualifier annotations for a constructor or method
parameter, a field or a setter.NonTransactionalIdempotentConsumerwith an at-most-once guarantee; the ids a failed call recorded are logged at
error.- Events brought back from the DLQ keep their original
created_at, start their retries over and are due at once.
Fixed
- Recovery, retry scheduling and the cleanup of processed events, the DLQ and the inbox no longer depend on the time
zone of the JVM: an instance in another zone no longer takes a fresh capture for stuck, and no longer deletes
records too early or too late. - A stale poller can no longer overwrite the outcome of a newer capture, which could leave a delivered event
FAILED
and in the DLQ, or a failed onePROCESSED. - Durations below a second were truncated to whole seconds: a
backoff.delayof500msretried at once, an
emergency-timeoutof500msfailed every batch, a sub-second metrics cache ttl expired every value. A fractional
backoff.multiplierlost its fraction (1.5 with 10s gave 10s, 20s, 30s instead of 15s, 22.5s, 33.75s). - A schedule strategy listener that threw stopped the polling of its event type until restart, and a failing
onExecutionSucceededwas counted as a failed run. - The inbox cache never hit on in-memory caches (
ConcurrentMapCacheManager, Caffeine): ids were read as strings and
written as UUIDs. - The MySQL DLQ schema shipped by the starter lacked the index on
dlq_statusandmoved_atused by the DLQ API. - The Kafka inbox warned about a non-manual
ack-mode, which is not needed; it now warns when
enable.auto.commit=true, which does lose events. - DLQ REST API: standard Spring MVC failures (unreadable body, missing parameter, unsupported method) keep their own
status instead of 500, the 500 response no longer exposes the exception message, and a catch-all
@ControllerAdviceof the application no longer answers DLQ API exceptions in its own format.
Improved
- Logs carry their context: task, worker, event type, lock token and counts; an empty poll is logged at debug.
- Configuration error messages name the property as the YAML key, for example
'sender.emergency-timeout'. - README: the two delivery guarantees, broker acknowledgement settings, cache name uniqueness across services sharing
a cache store, and corrected property names and defaults. - The example stack provisions Grafana with the datasource and both dashboards.