on XFS might not be able to read local storage pos file after crash

[jim-minter]

td-agent 2.3.6
rhel 7.6, XFS filesystem

    <source>
      @type systemd
      <storage>
        @type local
        path /var/log/journald.pos
      </storage>
      tag journald
    </source>

After a kernel panic and reboot, td-agent refused to start with:

2019-04-07 15:55:50 +0000 [error]: fluent/log.rb:362:call: config error file="/td-agent/config/fluent.conf" error_class=Fluent::ConfigError error="Unexpected error: failed to read data from plugin storage file: '/var/log/journald.pos/worker0/storage.json'"

/var/log/journald.pos/worker0/storage.json existed but was 0 bytes long.

It is good practice to use fdatasync() or fsync(); on XFS it is essential:

• https://stackoverflow.com/questions/47077828/xfs-rhel7-3-cold-reboot-file-truncate
• https://bugzilla.redhat.com/show_bug.cgi?id=845233

I recommend an fdatasync or fsync call is added around here:

fluentd/lib/fluent/plugin/storage_local.rb

Line 131 in 0eec554

open(tmp_path, 'w:utf-8', @mode){ |io| io.write json_string }

It may also be worth considering logging a warning in the event of a 0-length file at

fluentd/lib/fluent/plugin/storage_local.rb

Line 94 in 0eec554

raise Fluent::ConfigError, "Unexpected error: failed to read data from plugin storage file: '#{@path}'"

but continuing startup as if the file did not exist.

[repeatedly]

Thanks for the report.

Adding io.fsync in block is ok for me.

In addition, we need to add option like flush_mode.

• shutdown: sync data at shutdown
• each_update: sync data at each put/update operation
• interval or number: sync data when time/number of operation passed.

Without sync, storage content can't be stored by machine crush.

How about this?

[jim-minter]

makes sense to me :)