Set offset of files under ignore_older to file.size() #2907
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ruflin
force-pushed
the
ignore_older-state
branch
from
d08d68e
to
88528a2
Compare
Until now if a file was falling under ignore_older, offset 0 was set for the file. In case the file was updated again, all content of the file would be read. This change will set the offset for files falling under ignore_older to the size of the file. This applies on start / restart for files which were not seen before. The assumption behind this change is, that files falling under ignore_older are normally not updated, and if these files are updated only the newly added lines are expected to be read no the complete file. The setting of the offset only happens once when it falls for the first time under ignore_older and no state exists. For files which were harvested and then fall under ignore_older, the offset is already file.size(). As this change only applies to files without a state, this should also not have any side affects on Windows where it can happen that a file was updated but the timestamp wasn't and it falls under ignore_older. The reason it doesn't have an affect is that ignore_older only applies an offset if there is no previous state.
ruflin
force-pushed
the
ignore_older-state
branch
from
88528a2
to
a516a23
Compare
urso
reviewed
Nov 3, 2016
| @@ -298,6 +298,8 @@ func (p *ProspectorLog) handleIgnoreOlder(lastState, newState file.State) error | |||
| return nil | |||
| } | |||
|
|
|||
| newState.Offset = newState.Fileinfo.Size() | |||
There was a problem hiding this comment.
Add a comment explaining why we set offset to filesize + why is ignored file added to registry. See PR description.
There was a problem hiding this comment.
Documentation note added. For all details I referenced to this PR.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Until now if a file was falling under ignore_older, offset 0 was set for the file. In case the file was updated again, all content of the file would be read. This change will set the offset for files falling under ignore_older to the size of the file. This applies on start / restart for files which were not seen before.
The assumption behind this change is, that files falling under ignore_older are normally not updated, and if these files are updated only the newly added lines are expected to be read no the complete file.
The setting of the offset only happens once when it falls for the first time under ignore_older and no state exists. For files which were harvested and then fall under ignore_older, the offset is already file.size().
As this change only applies to files without a state, this should also not have any side affects on Windows where it can happen that a file was updated but the timestamp wasn't and it falls under ignore_older. The reason it doesn't have an affect is that ignore_older only applies an offset if there is no previous state.