buffer: stage_size drifts upward when a partially-filled staged chunk is unstaged

[aeurielesn]

Describe the bug

@stage_size drifts upward slowly under load over sustained period of time each time a partially-filled staged chunk is not subtracted when it's unstaged because a record is too big to be appended.

Eventually leading to a refusal of new writes once @stage_size + @queue_size reaches total_limit_size. Only a pod restart resets @staged.

To Reproduce

Using a tcp input and a discard output with a buffer configuration as shown below.

A constant stream of records of a size of 1.5KB causes the @stage_size to drift upwards.

Expected behavior

@staged_size does not drifts and writes are not refused, and pod restarts are not required.

Your Environment

- Fluentd version: 1.19.1
- Operating system: Debian GNU/Linux 13 (trixie)
- Kernel version: 6.12.68+
- Environment: Kubernetes

Your Configuration

fluentd.conf

<source>
  @type tcp
  @log_level info
  port 20001
  bind 0.0.0.0
  tag input.json
  <parse>
    @type json
  </parse>
</source>

<match input.json>
  @type discard
  @log_level info
  <buffer>
    @type memory
    chunk_limit_size 8k
    chunk_limit_records 250
    total_limit_size 4m
    overflow_action drop_oldest_chunk
  </buffer>
</match>

out_discard.rb

require "json"
require "fluent/plugin/output"

module Fluent
  module Plugin
    class DiscardOutput < Output
      Fluent::Plugin.register_output("discard", self)

      def format(tag, time, record)
        "#{JSON.generate(record)}\n"
      end

      def write(chunk)
        # discard on purpose
      end
    end
  end
end

Your Error Log

2026-07-16 23:56:33 2026-07-16 21:56:33 +0000 [warn]: #0 failed to write data into buffer by buffer overflow action=:drop_oldest_chunk
2026-07-16 23:56:33 2026-07-16 21:56:33 +0000 [error]: #0 no queued chunks to be dropped for drop_oldest_chunk
2026-07-16 23:56:33 2026-07-16 21:56:33 +0000 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin/buffer.rb:332:in 'Fluent::Plugin::Buffer#write'" tag="input.json"
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin/buffer.rb:332:in 'Fluent::Plugin::Buffer#write'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin/output.rb:1080:in 'block in Fluent::Plugin::Output#handle_stream_simple'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin/output.rb:957:in 'Fluent::Plugin::Output#write_guard'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin/output.rb:1079:in 'Fluent::Plugin::Output#handle_stream_simple'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin/output.rb:947:in 'Fluent::Plugin::Output#execute_chunking'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin/output.rb:877:in 'emit_buffered'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/event_router.rb:115:in 'Fluent::EventRouter#emit_stream'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin/in_tcp.rb:200:in 'block in Fluent::Plugin::TcpInput#start'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin_helper/server.rb:661:in 'on_read_with_connection'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [warn]: #0 /usr/local/bundle/gems/cool.io-1.9.0/lib/cool.io/io.rb:123:in 'Coolio::IO#on_readable'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [warn]: #0 /usr/local/bundle/gems/cool.io-1.9.0/lib/cool.io/io.rb:186:in 'Coolio::IO::Watcher#on_readable'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [warn]: #0 /usr/local/bundle/gems/cool.io-1.9.0/lib/cool.io/loop.rb:88:in 'Coolio::Loop#run_once'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [warn]: #0 /usr/local/bundle/gems/cool.io-1.9.0/lib/cool.io/loop.rb:88:in 'Coolio::Loop#run'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin_helper/event_loop.rb:93:in 'block in Fluent::PluginHelper::EventLoop#start'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin_helper/thread.rb:78:in 'block in Fluent::PluginHelper::Thread#thread_create'
2026-07-16 23:56:33 2026-07-16 21:56:33 +0000 [error]: #0 unexpected error on reading data host="a.b.c.d" port=37448 error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data"
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin/buffer.rb:332:in 'Fluent::Plugin::Buffer#write'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin/output.rb:1080:in 'block in Fluent::Plugin::Output#handle_stream_simple'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin/output.rb:957:in 'Fluent::Plugin::Output#write_guard'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin/output.rb:1079:in 'Fluent::Plugin::Output#handle_stream_simple'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin/output.rb:947:in 'Fluent::Plugin::Output#execute_chunking'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin/output.rb:877:in 'emit_buffered'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/event_router.rb:115:in 'Fluent::EventRouter#emit_stream'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin/in_tcp.rb:200:in 'block in Fluent::Plugin::TcpInput#start'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin_helper/server.rb:661:in 'on_read_with_connection'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [error]: #0 /usr/local/bundle/gems/cool.io-1.9.0/lib/cool.io/io.rb:123:in 'Coolio::IO#on_readable'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [error]: #0 /usr/local/bundle/gems/cool.io-1.9.0/lib/cool.io/io.rb:186:in 'Coolio::IO::Watcher#on_readable'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [error]: #0 /usr/local/bundle/gems/cool.io-1.9.0/lib/cool.io/loop.rb:88:in 'Coolio::Loop#run_once'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [error]: #0 /usr/local/bundle/gems/cool.io-1.9.0/lib/cool.io/loop.rb:88:in 'Coolio::Loop#run'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin_helper/event_loop.rb:93:in 'block in Fluent::PluginHelper::EventLoop#start'
2026-07-16 23:56:33   2026-07-16 21:56:33 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.19.0/lib/fluent/plugin_helper/thread.rb:78:in 'block in Fluent::PluginHelper::Thread#thread_create'

Additional context

No response

[kenhys]

What plugin are you using for "@type discard" to reproduce issue? (I mean that fluent-plugin-[SOMETHING])

[aeurielesn]

For the reproduction, I used this out_discard.rb file:

require "json"
require "fluent/plugin/output"

module Fluent
  module Plugin
    class DiscardOutput < Output
      Fluent::Plugin.register_output("discard", self)

      def format(tag, time, record)
        "#{JSON.generate(record)}\n"
      end

      def write(chunk)
        # discard on purpose
      end
    end
  end
end

[Watson1978]

Thanks for the report, and for the PR!

I reproduced this on master with a minimal setup. Posting the observation as a data point.

Setup

fluent.conf (@type discard is the plugin from the issue description, loaded via -p):

<source>
  @type monitor_agent
  bind 0.0.0.0
  port 24220
</source>

<label @FLUENT_LOG>
  <match fluent.**>
    @type null
  </match>
</label>

<source>
  @type tcp
  port 9999
  bind 0.0.0.0
  tag input.json
  <parse>
    @type none
  </parse>
</source>

<match input.json>
  @type discard
  <buffer>
    @type memory
    chunk_limit_size 8k
    chunk_limit_records 250
    total_limit_size 4m
    overflow_action drop_oldest_chunk
  </buffer>
</match>

Client:

require "socket"

s = TCPSocket.open("localhost", 9999)
data = "a" * 1024

5000.times do |i|
  s.puts data
  sleep 0.01
end
s.close

Monitor:

$ watch -n1 'curl -s http://localhost:24220/api/plugins.json | \
    jq ".plugins[] | select(.type==\"discard\") |
        {buffer_stage_length, buffer_stage_byte_size,
         buffer_queue_length, buffer_queue_byte_size, buffer_total_queued_size}"'

Result

While the client is still sending, monitor result:

{
  "buffer_stage_length": 1,
  "buffer_stage_byte_size": 1970983,
  "buffer_queue_length": 0,
  "buffer_queue_byte_size": 0,
  "buffer_total_queued_size": 1970983
}

Observation

A staged chunk can never exceed chunk_limit_size: Buffer#chunk_size_over? is evaluated right after every append, and chunk.rollback runs whenever it returns true. So this should always hold:

buffer_stage_byte_size <= buffer_stage_length * chunk_limit_size

Here the upper bound is 1 * 8k = 8192, but the observed value is 1970983 — roughly 240x over.

buffer_queue_length is 0, so buffer_total_queued_size is entirely @stage_size. The buffer is occupying 1.9MB (47% of total_limit_size 4m) that has no data behind it, and it never comes back down. This matches the no queued chunks to be dropped for drop_oldest_chunk line in the original report: the buffer reports itself as full while the queue is actually empty.

Conditions: memory buffer, fresh start. (Buffer#resume legitimately adds up existing chunk sizes, so a file buffer whose chunk_limit_size was shrunk afterwards could exceed the bound for a valid reason — not the case here.)