Performance issues at dnstap listener - buffer is full

[Kisuke-CZE]

Hi,

run dnscollector to push replylogs from two unbound resolvers to Loki. The traffic is ~380 qps (for both resolvers combined). But some messages are being dropped at dnstap collector side.

I just get ocasional messages like this at dnscollector log:
Oct 21 09:38:33 dnscollector[951424]: WARNING: 2026/10/21 09:38:33.475662 worker - [dnstap-listener] dnstap - worker[dnstap-processor] buffer is full, 25 dnsmessage(s) dropped

I am now running version 3.2.0 now. I kept raising sock-read-buffer-size: and buffer-size and batch-size quite randomly (just doubled value every time]. Now "buffer is full" messages are quite rare ( <20 per day) and dropped messages are between 1-50 in those reports. When I was at defaults, messages were very often and number of dropped messages were thousands.

My full config right now is:

global:
  server-identity: "dns-collector-prod"
  pid-file: "/var/run/dnscollector/dnscollector.pid"
  trace:
    verbose: true
    log-malformed: true
  telemetry:
    enabled: false
  worker:
    buffer-size: 16384
    batch-size: 512

pipelines:
  - name: dnstap-listener
    dnstap:
      num-workers: 12
      sock-read-buffer-size: 524288
      listen-ip: 0.0.0.0
      listen-port: 6000
      tls-support: true
      cert-file: /etc/pki/dnscollector/dnscollector.cer
      key-file: /etc/pki/dnscollector/dnscollector.key
    transforms:
      normalize:
        qname-lowercase: true
        qname-replace-nonprintable: true
        rr-lowercase: true
    routing-policy:
      forward: [ loki ]
      
  - name: loki
    lokiclient:
      server-url: "http://localhost:3100/loki/api/v1/push"
      job-name: "dnscollector"
      batch-size: 6291456
      mode: "json"
      relabel-configs:
        - source_labels: ["__dns_rcode"]
          target_label: rcode

I am not sure now how to continue. I think I am far behind recommended values in docs. So not sure if increasing limits is even useful at this point.

Anyone who is using dnscollector in this central mode? What is your setup or recomendation?

[morvayo]

that warning identifies the internal dnstap processor queue rather than the socket receive buffer in v3.2.0 a full processor channel drops frames immediately the raw batch queue is sized as max(100, buffer-size / batch-size) so raising batch size does not independently give it more queue slots allocation enable telemetry and compare a short controlled run with a discard sink instead of Loki to separate decoding pressure from sink pressure then tune one value at a time average qps alone can hide short bursts and increasing the socket buffer will not directly fix this internal queue

[dmachard]

The root cause is SendEntries being synchronous inside StartLogging. When Loki is slow, it blocks the entire chain
The real saturation is on the Loki worker's input channel. Your large batch-size: 6291456 on the Loki logger makes it worse

Workaround: lower the Loki batch-size significantly.
Fix needed: SendEntries should run async (own goroutine + internal queue) so StartLogging is never blocked by HTTP latency => could you open an issue to track it ?

[dmachard]

A fix will be included in the upcoming v3.2.3 release. Could you try reverting your buffer sizes closer to standard values and let me know if the drops completely disappear on your 380 QPS setup? Your feedback will be very valuable!

[Kisuke-CZE]

Hi, thanks for such a swift responses.

I've just updated to v3.2.3 and went back to mode default values. More or less defaults, num-workers is set to number of CPUs. Whole config here:

global:
  server-identity: "dns-collector-prod"
  pid-file: "/var/run/dnscollector/dnscollector.pid"
  trace:
    verbose: true
    log-malformed: true
  telemetry:
    enabled: false
  worker:
    buffer-size: 2048
    batch-size: 64

pipelines:
  - name: dnstap-listener
    dnstap:
      num-workers: 12
      sock-read-buffer-size: 65536
      listen-ip: 0.0.0.0
      listen-port: 6000
      # tls-support: false
      tls-support: true
      cert-file: /etc/pki/dnscollector/dnscollector.cer
      key-file: /etc/pki/dnscollector/dnscollector.key
    transforms:
      normalize:
        qname-lowercase: true
        qname-replace-nonprintable: true
        rr-lowercase: true
    routing-policy:
      forward: [ loki ]

  - name: loki
    lokiclient:
      server-url: "http://localhost:3100/loki/api/v1/push"
      job-name: "dnscollector"
      batch-size: 1048576
      mode: "json"
      relabel-configs:
        - source_labels: ["__dns_rcode"]
          target_label: rcode

So far I did get one warning:

Oct 24 11:06:25 dnscollector[1556065]: WARNING: 2026/10/24 11:06:25.513894 worker - [loki] loki - send buffer is full, batch dropped

But it is weekend now. Real traffic hits on monday.

Sorry I did not created issue. Is it still relevant?
I planned to dive in that more deeply as suggested. Originally I believed my setup was bad - maybe bad design choice when choosing that central mode.

[Kisuke-CZE]

I've increased batch-channel-size to 64 on lokiclient, it seem this can prevent dropping some more messages when loki gets slow.

Now I believe it is time to look into loki to fine tune it.

[dmachard]

Hi @Kisuke-CZE,
Thanks for the feedback!

Batch size vs flush interval: A 1MB batch (batch-size: 1048576) is good for compression, but if query rate is variable, you can also lower flush-interval (e.g. 1s or 2s) so smaller batches get pushed more steadily instead of big spikes.

No need to open a separate issue now that the discussion is tracked and solved in v3.2.3. Feel free to report back on Monday after the peak traffic!