Intermittent consumption “gaps” with SmallRye Kafka connector, no broker lag

[joystuffs]

Hi, I’m seeing occasional, unexpected “delays” in message consumption with a SmallRye Reactive Messaging Kafka consumer.

The use case is straightforward:

Consume records from a topic

Do lightweight mapping (non-blocking)

Perform a blocking HTTP request

Main requirement: near-real-time processing (minimize time between record production and handling)

Even under low/steady load, sometimes the received record appears “older than expected” (e.g., > 2 seconds), based on:

long messagePollingDuration = System.currentTimeMillis() - message.timestamp();

when this happens, Kafka broker-side metrics look fine (no visible lag / backlog). This makes me suspect buffering/queuing on the client/connector side, but I’m not sure.

Configuration

mp.messaging.incoming.consumer.connector=smallrye-kafka
mp.messaging.incoming.consumer.client.id=foo_client_id

mp.messaging.incoming.consumer.group.id=foo_consumer_group_id
mp.messaging.incoming.consumer.group.instance.id=${hostname}_foo

mp.messaging.incoming.consumer.topic=foo
mp.messaging.incoming.consumer.enable.auto.commit=true
mp.messaging.incoming.consumer.auto.offset.reset=latest
mp.messaging.incoming.consumer.failure-strategy=ignore
mp.messaging.incoming.consumer.partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
mp.messaging.incoming.consumer.graceful-shutdown=true

Consumer code

@Incoming("consumer")
@Acknowledgment(Acknowledgment.Strategy.NONE)
@Blocking(ordered = false)
@RunOnVirtualThread
public void consume(ConsumerRecord<String, String> message) {

    long messagePollingDuration = System.currentTimeMillis() - message.timestamp();

    if (messagePollingDuration > thresholdMs) {
        log.warn("Observed delay between record timestamp and consumption: {} ms", messagePollingDuration);
    }

    try {
        // Blocking operation
        restClient.post(request);
    } catch (Exception e) {
        // swallow error for simplicity in this example
        log.warn("HTTP call failed", e);
    }
}

Observed behavior

From time to time, messagePollingDuration spikes above the threshold (e.g., 2000 ms)

This happens even when load is not high

Broker metrics do not show lag/backlog at those moments (at least from what I can see)

What I investigated

While debugging, I noticed records being queued in:

RecordQueue queue inside KafkaRecordStreamSubscription

I couldn’t find exposed logs/metrics about that internal queue (depth, enqueue/dequeue rate, time-in-queue). If records are spending time there, it could explain the “age” spikes I’m observing, but I don’t know if this is the right direction.

I also found a possibly related issue: issue:1507 , but I’m not sure if it applies to my setup.

Questions

Is my acknowledgment/commit configuration correct for the goal “keep polling fast and don’t let processing slow down consumption”?

I enabled enable.auto.commit=true and also use @acknowledgment(Acknowledgment.Strategy.NONE).

The connector docs warn against enabling auto-commit because it ignores processing outcome. In my case that’s intentional (I want offsets committed regardless of processing result), but I want to confirm this combination is supported and won’t introduce unexpected behavior.

Are there known causes of intermittent consumption gaps without visible broker lag?

For example:

internal buffering/queueing in the connector

consumer poll loop starvation

backpressure interaction with @Blocking(ordered=false) and/or virtual threads

Is there any existing metric/log/tracing hook to observe the internal RecordQueue behavior?

If not, would you accept a feature request to expose:

queue depth

time spent in queue

records dropped/paused/backpressured

poll-to-delivery latency

Sanity check: since I’m computing System.currentTimeMillis() - record.timestamp(), is there a recommended approach to measure “time from poll to consumer method” within the connector?

(My current measurement relies on the record timestamp, which is produced-side time.)

Quarkus: 3.27.1

smallrye-reactive-messaging-kafka:4.28.0

[ozangunalp]

Very probably, the bottleneck in your processing is the call to the rest client.

Have you tried limiting your VT concurrency ? It is explained in here : https://quarkus.io/guides/messaging#execution_model

[joystuffs]

Hi @ozangunalp thank you for the feedback.

tbh, not yet. I will take a look to the VT concurrency.

so far, I tried naively to offload on the eventbus, but I have the feeling that's overhead and won't change much given that I still face the same issue with the eventbus.

another question, does it make sense how the consumer is configure?