Filebeat - Support using O_DIRECT when reading files

[andrewkroh]

When reading files over NFS it would be useful to be able to use the O_DIRECT flag when opening the file for reading due to file cache coherency issues that can occur. Ideally this would be a config option that can be specified for a specific prospector/harvester. The config option would be supported only on platforms where we have the ability to use the O_DIRECT flag.

From: http://nfs.sourceforge.net/#section_a

If you need absolute cache coherency among clients, applications can use file locking, where a client purges file data when a file is locked, and flushes changes back to the server before unlocking a file; or applications can open their files with the O_DIRECT flag to disable data caching entirely.

From: https://access.redhat.com/solutions/22717

. . . when the size of the file increases on the NFS server after the NFS client has sent an asynchronous read request that reaches the EOF we see this issue.

Once the file size increases, the NFS client assumes that the previous read reply contains the data from the new region between the previous file size and the new file size.

Inside the NFS client, this region contains NULLs because the NFS client has never asked for nor received data for this region from the NFS server.

Other References

• https://stackoverflow.com/questions/5055859/how-are-the-o-sync-and-o-direct-flags-in-open2-different-alike

[ruflin]

+1 on this and using a config option for it. Someone even create a "library" around this: https://github.com/ncw/directio But we should definitively have our own implementation.

[shelomentsevd]

Hi!
I want to try to add this option to filebeats, but there is a problem:
In https://github.com/ncw/directio implemented only Unix, Darwin and Windows OS's.
Not sure if it right to implement direct_o only for them, I suppose beats works on many systems not only darwin,windows and unix-like.
I'll be glad if somebody tells me which systems I should take in account.

[andrewkroh]

I'd say that library would be fine to use. It says

This is library for the Go language to enable use of Direct IO under all supported OSes of Go (except openbsd and plan9).

Filebeat is used by people on openbsd so I'd take care to make the O_DIRECT config option not available on that system.

[shelomentsevd]

@andrewkroh I tried to exclude DIRECT_O on OpenBSD and Plan9 but I am not sure that did it right.
Can you please check my PR?

Thank you for your participation!

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This issue doesn't have a Team:<team> label.