File input appears to stop reading from NFS when rsyncing files

[jordansissel]

(This issue was originally filed by @ScottChapman at elastic/logstash#2803)

---

So, I have a file input clause, and logstash is actively reading files from an NFS share.

Then rsync comes along and updates files, and my logstash just stops.

I've watched the files that are getting updated, and no new files, and no inode changes. Just a log file increases in size.

I can do a tail -f on the file and that runs just fine which rsync-ing.

Any ideas?

[jordansissel]

@ScottChapman I usually see this behavior when stat() calls against files over NFS are cached too eagerly on the client side, and it results in files incorrectly observed (the OS says the file size isn't changing). Can you confirm ls -ld /some/path/to/your/file while it is being written to? Specifically, across multiple writes, then output of that ls command would help a bit.

[jordansissel]

Waiting on response to my above question. No rush :)

[raphaele81]

I have a similar issue with logstash-2.0.0 running on windows server 2012 :

I have log files being written asynchronously by a java program on one server. Logstash reads those files via a shared folder.
On start up, everything works fine, but then, logstash stops reading the files even though they are being updated. When i restart logstash it reads the latest entries and then stops again.

I first thought it was a caching problem, however, when I put logstash in debug mode, after the last line being read, I have logstash log entries showing that it sees the files has changed:

{:timestamp=>"2015-11-06T11:12:24.478000+0100", :message=>"//path/to/my/log/trace_20151106.log: file grew, old size 352281, new size 352427", :level=>:debug, :file=>"/elk/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.5/lib/filewatch/watch.rb", :line=>"96", :method=>"each"}

I also realized that when the problem occurs, the last line of the logs file is full af ascii 'NUL' characters. When i stop logstash the 'NUL' line disappear. So I might be coming across the problem discribed here : https://discuss.elastic.co/t/logstash-vs-nfs-null-characters/25918

We can't launch our ELK stack in production if this problem isn't fixed.

[guyboertje]

I think this is fixed now with the "never read beyond the declared file size (stat)" strategy.