warn message should include right status

[paihu]

Check CONTRIBUTING guideline first and here is the list to help us investigate the problem.

Describe the bug

I received a log message like below

2020-07-14 01:24:15 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted
 event stream: 574180bytes

But I set to chunk_limit_size is 8M and chunk_full_threshold is 0.8.
8M*0.8 = 6.4M
chunk has free space more than 1.6M but log message say 574K.

I checked the chunk and realized that there are many records.

The message received was a bit unfriendly.

To Reproduce

1. set large chunk_limit_size
2. set small chunk_limit_records
3. start fluentd
4. push large number of minimum records
5. show warn message

Expected behavior

2020-07-14 01:24:15 +0000 [warn]: #0 [rawlog2elasticsearch] chunk size limit exceeds for an emitted
 event stream: 1400records

Your Environment

• Fluentd or td-agent version: fluentd --version or td-agent --version

$ td-agent --version
td-agent 1.11.1

• Operating system: cat /etc/os-release

$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

• Kernel version: uname -r

$ uname -r
4.14.177-139.254.amzn2.x86_64

If you hit the problem with older fluentd version, try latest version first.

Your Configuration

  <source>
    @type forward
    @id input_forward
    port 24224
    bind "0.0.0.0"
  </source>
  <filter **.es>
    @type prometheus
    <metric>
      name fluentd_input_status_num_records_total
      type counter
      desc The total number of incoming records
      <labels>
        tag ${tag}
        hostname ${hostname}
      </labels>
    </metric>
  </filter>
  <match *.*.es>
    @type elasticsearch
    @id rawlog2elasticsearch
    hosts "xxxx"
    log_es_400_reason true
    reconnect_on_error true
    reload_on_failure true
    reload_connections false
    logstash_format true
    logstash_prefix "raw-${tag[1]}-${tag[0]}"
    <buffer tag>
      @type "file"
      path "/data/buffers/raw"
      chunk_limit_size 8M
      total_limit_size 1G
      chunk_limit_records 1000
      chunk_full_threshold 0.8
      flush_mode interval
      flush_interval 60s
      flush_thread_count 1
      retry_forever true
      retry_type periodic
      retry_wait 30
    </buffer>
  </match>

Your Error Log

2020-07-13 04:59:59 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 474709bytes
2020-07-13 05:00:24 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 553495bytes
2020-07-13 05:04:19 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 603165bytes
2020-07-13 05:04:24 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 551115bytes
2020-07-13 05:06:24 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 581636bytes
2020-07-13 05:07:19 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 562004bytes
2020-07-13 05:07:24 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 572290bytes
2020-07-13 05:11:10 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 627635bytes
2020-07-13 05:12:15 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 460620bytes
2020-07-13 05:13:20 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 485934bytes
2020-07-13 05:14:15 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 464083bytes
2020-07-13 05:16:20 +0000 [warn]: #0 [rawlog2elasticsearch] chunk bytes limit exceeds for an emitted event stream: 466951bytes

Additional context

[repeatedly]

Thanks for the report. Will check soon.

[repeatedly]

Patch: #3079