Skip to content

[Bug] KafkaActor resolves its eachMessage promise before the handler runs while autoCommit is on, so the documented at-least-once default is at-most-once, and a partial withConsumer erases a HOCON manual commit mode #975

Description

@pathosDev

Problem

KafkaActor's class doc promises at-least-once for the default commit mode. The code delivers at-most-once: eachMessage calls target.tell(...) — an enqueue into the target actor's mailbox — and then returns. kafkajs commits the offset when that promise resolves, which is before the handler has been scheduled, let alone finished. A pod that dies while the record is still in the mailbox has already told the broker it is done with it.

Two independent things then make the escape hatch unusable:

  1. commitMode: 'manual' is the documented fix, but mergeOptions is a shallow spread, so .withConsumer({ groupId: 'orders' }) replaces the whole consumer object and erases a HOCON commitMode: 'manual' underneath it. The one field the operator set in config is silently dropped by an application that only meant to override the group id.
  2. Nothing surfaces the resulting mode. There is no log line, no BrokerConnected field, no way to observe from outside that the actor is running auto-commit.

Evidence

The doc, src/io/broker/KafkaActor.ts:114-119:

src/io/broker/KafkaActor.ts:114-119
 * **Offset-commit semantics.**
 *
 *   - `commitMode: 'auto'` (default)  kafkajs commits after each
 *     handler returns successfully  **at-least-once**.  Cheap; OK
 *     for idempotent handlers.
 *   - `commitMode: 'manual'`  pump pauses on each message and waits

The code, src/io/broker/KafkaActor.ts:243-264target.tell is a mailbox enqueue, and if (!manualCommit) return; resolves the pump promise immediately after it:

src/io/broker/KafkaActor.ts:243-264
      const target = this.options.target;
      const manualCommit = this.options.consumer.commitMode === 'manual';
      const commitTimeoutMs = this.options.consumer.commitTimeoutMs ?? 30_000;

      // We deliberately don't await `run` — it's a long-running pump.
      void this.consumer.run({
        // kafkajs v2: `autoCommit: false` disables the auto-commit
        // path so eachMessage's resolution doesn't trigger a commit.
        // We drive commits from the actor via `commitOffsets` instead.
        autoCommit: !manualCommit,
        eachMessage: async ({ topic, partition, message, heartbeat }: KafkaConsumedMessage): Promise<void> => {
          if (!target) return;
          target.tell({
            topic, partition,
            offset: message.offset,
            key: message.key,
            value: message.value,
            timestamp: message.timestamp,
            headers: message.headers ?? {},
          });

          if (!manualCommit) return;

The merge that erases the HOCON setting, src/util/OptionsMerge.ts:25-29:

src/util/OptionsMerge.ts:25-29
  return {
    ...builtInDefaultOptions,
    ...stripUndefined(fromConfig),
    ...stripUndefined(fromExplicit),
  } as S;

stripUndefined only walks the top level, so consumer is replaced wholesale rather than merged leaf by leaf.

Proposal

  • Make the default honest. Either await the handler (an ask with a timeout, or the manual-commit pump made the default), or change the doc and KafkaCommitMode's own comment to say 'auto' is at-most-once and name the mailbox as the loss window. The current text is the opposite of what the code does, and it is the text an adopter reads.
  • Merge nested option blocks per-leaf, or split consumer into flat keys (consumerGroupId, consumerCommitMode, …) so withConsumer cannot erase a config file. This applies to every nested block in every broker options type, not just Kafka.
  • Log the resolved commit mode once at connect, at info.

Acceptance sketch

  • A record delivered in the default mode is not committed until the target actor has finished processing it — or the documentation states plainly that it is not.
  • withConsumer({ groupId }) layered over a HOCON consumer.commitMode = manual resolves to commitMode: 'manual'.
  • A test asserts the resolved consumer block for the builder-over-HOCON case field by field.
  • The connect log line names the effective commit mode.

Verification status

Found in the ten-lens production-readiness review of 2026-08-05 (v0.13.0) and re-verified before filing: reproduced by execution. A KafkaActor subclass overrode the createKafkaInstance test seam with a fake kafkajs that captures the run arguments and drives one record through eachMessage; the system config carried consumer: { groupId: 'from-hocon', commitMode: 'manual' }, and the constructor passed KafkaOptions.create().withConsumer({ groupId: 'orders' }):

resolved consumer options : {"groupId":"orders"}
autoCommit passed to kafkajs: true
  0. eachMessage ENTER
  1. eachMessage RESOLVED  <-- kafkajs commits offset 42 here
  2. handler START offset=41
  3. handler DONE  offset=41

The commit point is step 1; the handler has not started until step 2. The HOCON commitMode is gone from the resolved options entirely.

Adjacent issues: #2 is the manual-commit feature this doc refers to and is about the opt-in path, not the default. #652 (reconnect jitter) and #659 (broker observability) touch the same class but neither covers the commit point or the nested-merge erasure.

Part of the production-readiness review batch — tracked in #913.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority: highTop priority — high impact, plan nextproduction-goalBlocks or defines the path to production readiness

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions