Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(file source): sleep to avoid split reads #1236

Merged
merged 1 commit into from
Nov 25, 2019
Merged

Conversation

lukesteensen
Copy link
Member

Closes #1125

The file watcher works by repeatedly reading from the file being tailed until it sees a newline or EOF (i.e. the read syscall returns 0). Whatever data it has accumulated to that point is returned as a line.

When reads are closely interleaved with writes, however, it's possible for read to return part of a single write, and then EOF, and then the rest of that same write sometime later. This causes the file watcher to split what should be a single line into two. We've seen this happen ~3 times over 30M+ reads in our test environment.

This change causes the file watcher to wait 1 ms for any in-flight writes to become fully visible before returning a line that was never actually terminated with a newline.

Signed-off-by: Luke Steensen <luke.steensen@gmail.com>
@lukesteensen lukesteensen added type: bug A code related bug. source: file Anything `file` source related labels Nov 23, 2019
@binarylogic
Copy link
Contributor

@Jeffail do you mind reviewing this? Thank you sir.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
source: file Anything `file` source related type: bug A code related bug.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Split reads
3 participants