Too many decompressed-data20220209-1054-****** files are created by td-agent (4.3.0).

[vikranth06]

Describe the bug

We are seeing too many decompressed-data20220209-1054-****** files created by td-agent (4.3.0) in /tmp folder of our Ubuntu 20.04 machine.
This started after it was upgraded to 4.3.0 from 4.0.1.
We have downgraded it back to 4.0.1 but the files are still getting created.
These files consists of the data that is to be pushed to ElasticSearch.

To Reproduce

Td-agent was upgraded while applying the latest updates using sudo apt-update command.
No other actions were performed.

Expected behavior

dont see a need of creating so many data-decompressed files.

Your Environment

- Fluentd version:1.11.2
- TD Agent version:4.0.1
- Operating system: Ubuntu 20.04

Your Configuration

<system>
  log_level error
  workers 5
</system>

<worker 0>
  <source>
    @type tail
    @label IISLogs
    path C:/inetpub/logs/LogFiles/*****/W3SVC/*
    pos_file C:/opt/td-agent/IIS.log.pos
    tag IISLogs
	
    <parse>
	@type multiline
	format_firstline /\d{4}-\d{1,2}-\d{1,2} /
	format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) (?<sitename>[^ ]+) (?<computername>[^ ]+) (?<ip>[^ ]+) (?<method>[^ ]+) (?<apipath>[^ ]+) (?<apiquery>[^ ]+) (?<port>[^ ]+) (?<username>[^ ]+) (?<ipaddress>[^ ]+) (?<httpversion>[^ ]+) (?<useragent>[^ ]+) (?<referrer>[^ ]+) (?<host>[^ ]+) (?<status>[^ ]+) (?<sub-status>[^ ]+) (?<Win32status>[^ ]+) (?<outgoingbytes>[^ ]+) (?<incomingbytes>[^ ]+) (?<responsetime>[0-9]+)/
    </parse>
  </source>

  <label IISLogs>
    <match IISLogs>
	@type copy
	<store ignore_error>
	  @type aws-elasticsearch-service
	  index_name *****-iis-logs
	  logstash_format true
	  logstash_prefix *****-iis-logs
	  flush_interval 1s
	  <endpoint>
		url ***********************************************
		region us-east-1
	  </endpoint>
	</store>
    </match>
  </label>
</worker>

<source>
  @type forward
  @id input_forward
  skip_invalid_event false
</source>

<label @FLUENT_LOG>
  # If you want to capture only error events, use 'fluent.error' instead.
  <filter fluent.**>
    @type record_modifier
    <record>
      host "#{Socket.gethostname}"
    </record>
</filter>

  <match fluent.**>
    @type copy
    <store ignore_error>
      @type aws-elasticsearch-service
      index_name *****-logs
      logstash_format true
      logstash_prefix *****-logs
	  bulk_message_request_threshold 8M
	  
      flush_interval 10s
      <endpoint>
        url *********************************
        region us-east-1
      </endpoint>
    </store>
  </match>
</label>

<filter *****.Transactions>
  @type record_modifier
  <record>
    log_json ${record["message"]}
  </record>
</filter>

<filter *****.Transactions>
  @type parser
  key_name log_json
  reserve_data false
  remove_key_name_field true
  emit_invalid_record_to_error true
  <parse>
    @type json
  </parse>
</filter>

<filter *****.Transactions>
  @type record_modifier
  <record>
    ***** ${record.dig("******","ID")}
    ***** ${record.dig("******","ID")}
    ***** ${record.dig("******","ID")}
    ***** ${record.dig("******Type","ID")}
    #******************* ${record.dig("******","Type","ID")}
  </record>
</filter>

<filter *****.Transactions>
  @type parser
  key_name ******_ID
  key_name ******_ID
  key_name ******_ID
  key_name ******Type_ID
  #key_name ******_Type_ID
  reserve_data true
  remove_key_name_field false
  emit_invalid_record_to_error true
  hash_value_field parsed
  <parse>
    @type json
  </parse>
</filter>

<match ****.Elasticsearch.LOGS>
  @type forward
  send_timeout 60s
  recover_wait 10s
  hard_timeout 60s
  time_as_integer true
  <buffer>
	@type file
	path "C:\\opt\\td-agent\\*****\\BufferLogs"
	chunk_limit_size 10m
	total_limit_size 10g
	flush_mode interval
	flush_interval 30s
	flush_thread_count 5
	overflow_action drop_oldest_chunk
	retry_wait 5s
  </buffer>
  <secondary>
	@type "secondary_file"
	directory "C:\\opt\\td-agent\\****\\SecondaryLogs"
	basename "****.logs"
  </secondary>
  <server>
    name myserver1
    host **************************
    port 24224
  </server>
</match>

<match *****.Maintenance.LOGS>
  @type forward
  send_timeout 60s
  recover_wait 10s
  hard_timeout 60s
  time_as_integer true
  <server>
    name myserver1
    host *************
    port 24224
  </server>
</match>

<match *****.Transactions>
  @type forward
  send_timeout 60s
  recover_wait 10s
  hard_timeout 60s
  time_as_integer true
  <server>
    name myserver1
    host xxxxxxxxxxxxxxxxxxxx
    port 24224
  </server>
  <buffer>
   flush_interval 1s
  </buffer>
  <secondary>
    @type secondary_file
    directory C:\opt\td-agent
    basename my.logs
  </secondary>
</match>

Your Error Log

total 66652452
srwx------ 1 td-agent td-agent        0 Feb 11 10:53 SERVERENGINE_SOCKETMANAGER_2022-02-11T10:53:33Z_181622
-rw------- 1 td-agent td-agent  1344556 Feb  9 20:03 decompressed-data20220209-1054-10361z8
-rw------- 1 td-agent td-agent  2148800 Feb  9 20:28 decompressed-data20220209-1054-108kp3k
-rw------- 1 td-agent td-agent  1027255 Feb  9 22:59 decompressed-data20220209-1054-10c3k5q
-rw------- 1 td-agent td-agent  2276967 Feb  9 20:36 decompressed-data20220209-1054-110fs50
-rw------- 1 td-agent td-agent  1870888 Feb  9 19:16 decompressed-data20220209-1054-11sad6r
-rw------- 1 td-agent td-agent  2582597 Feb  9 22:53 decompressed-data20220209-1054-11sqxvr
-rw------- 1 td-agent td-agent  1304223 Feb  9 23:09 decompressed-data20220209-1054-11vot6n
-rw------- 1 td-agent td-agent  1581582 Feb  9 18:39 decompressed-data20220209-1054-11xar74
-rw------- 1 td-agent td-agent  1201187 Feb  9 18:08 decompressed-data20220209-1054-11y6o0l
-rw------- 1 td-agent td-agent  1201187 Feb  9 18:10 decompressed-data20220209-1054-121jmo4
-rw------- 1 td-agent td-agent  1201187 Feb  9 18:08 decompressed-data20220209-1054-12oh1ed
-rw------- 1 td-agent td-agent  1327891 Feb  9 19:56 decompressed-data20220209-1054-12qb8ee
-rw------- 1 td-agent td-agent  1847018 Feb  9 18:27 decompressed-data20220209-1054-12z0x8h
-rw------- 1 td-agent td-agent  1581612 Feb  9 17:36 decompressed-data20220209-1054-135ry71
-rw------- 1 td-agent td-agent  1427644 Feb  9 23:06 decompressed-data20220209-1054-139q2ro
-rw------- 1 td-agent td-agent  2582597 Feb  9 22:55 decompressed-data20220209-1054-13c1351

Additional context

No response

[vikranth06]

@repeatedly - Can you please help us with this issue?

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions]

This issue was automatically closed because of stale in 30 days

[jeremych1000]

Can this be reopened? We are seeing the same thing, around 2.5GB of these decompressed data files being created per minute, leading to us having a full volume in 100 minutes. Fluentd then blocks because we've configured it to throw exceptions if the buffer is full.

My config is something like

<match *>
  @type copy
  <source>
    <buffer>
      file
    </buffer>
    elasticsearch 1 details
  </source>
  <source>
    <buffer>
      file
    </buffer>
    elasticsearch 2
  </source>
</match<>

[jeremych1000]

Can this be reopened please? @vikranth06 did you end up fixing this?

[jeremych1000]

Does

fluentd/lib/fluent/plugin/buffer/chunk.rb

Line 209 in 5844f72

Tempfile.new('decompressed-data')

ever get cleaned up?

[hawk-bernhard-altendorfer]

We see the same issue.

This is our config:

<buffer>
        @type file
        path output/opensearch/buffer
        flush_mode interval
        flush_interval 15s
        flush_thread_count 2
        flush_at_shutdown true
        retry_type exponential_backoff
        retry_forever false
        retry_max_times 10
        retry_max_interval 30
        total_limit_size 10GB
        # https://docs.aws.amazon.com/opensearch-service/latest/developerguide/limits.html#network-limits
        # AWS sets a hard limit on the maximum size of HTTP request payloads dependent on the instance type used.
        # Currently, this is either 10 MiB or 100 MiB.
        chunk_limit_size 64MB
        chunk_full_threshold 0.9
        queued_chunks_limit_size 10
        disable_chunk_backup true
        compress gzip
        overflow_action block
</buffer>

As we use gzip compression I wonder why we see those tmp files. According to the code we should not go this path when the data is compressed?

def open(**kwargs, &block)
  if kwargs[:compressed] == :gzip
    super
  else
    super(**kwargs) do |chunk_io|
      output_io = if chunk_io.is_a?(StringIO)
                    StringIO.new
                  else
                    Tempfile.new('decompressed-data')
                  end
      output_io.binmode if output_io.is_a?(Tempfile)
      decompress(input_io: chunk_io, output_io: output_io)
      output_io.seek(0, IO::SEEK_SET)
      yield output_io
    end
  end
end