New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify clean_removed handling #3827

Merged
merged 2 commits into from Apr 18, 2017

Conversation

Projects
None yet
2 participants
@ruflin
Collaborator

ruflin commented Mar 28, 2017

Previously if a file could be found under the same name as a state, the state was not removed. But this could have been also an other file. With this change also the file itself is compared and if it is not the same file, the state will be removed. This has the affect that in case a file is renamed after monitoring the file finished, the state will be removed. In most cases this should not have any side affect.

The positive effect of this change is that there will be less left over states in the registry.

Closes #3789

@ruflin

This comment has been minimized.

Collaborator

ruflin commented Apr 6, 2017

@elastic/beats Would be good to hear your thoughts on this change.

@@ -94,24 +94,35 @@ func (l *Log) Run() {
if l.config.CleanRemoved {
for _, state := range l.Prospector.states.GetStates() {
// os.Stat will return an error in case the file does not exist
_, err := os.Stat(state.Source)
stat, err := os.Stat(state.Source)
if err != nil {

This comment has been minimized.

@7AC

7AC Apr 6, 2017

Contributor

Should we be more strict here and check vs os.PathError?

This comment has been minimized.

@ruflin

ruflin Apr 10, 2017

Collaborator

Good idea. What should we do in all other cases? Log an error and not remove the state?

This comment has been minimized.

@7AC

7AC Apr 10, 2017

Contributor

Yea that's probably the best we can do. os.PathError is all that can return right now but you never now..

This comment has been minimized.

@ruflin

ruflin Apr 13, 2017

Collaborator

changed

// Only clean up files where state is Finished
if state.Finished {
state.TTL = 0
err := l.Prospector.updateState(input.NewEvent(state))

This comment has been minimized.

@7AC

7AC Apr 6, 2017

Contributor

We could return here and check the error in the caller, and also remove the else

This comment has been minimized.

@ruflin

ruflin Apr 13, 2017

Collaborator

I changed it to return directly if state is not finished but I kept the error handling local.

ruflin added some commits Mar 28, 2017

Modify clean_removed handling
Previously if a file could be found under the same name as a state, the state was not removed. But this could have been also an other file. With this change also the file itself is compared and if it is not the same file, the state will be removed. This has the affect that in case a file is renamed after monitoring the file finished, the state will be removed. In most cases this should not have any side affect.

The positive effect of this change is that there will be less left over states in the registry.

Closes #3789
@ruflin

review changes implemented and rebase on master

// Only clean up files where state is Finished
if state.Finished {
state.TTL = 0
err := l.Prospector.updateState(input.NewEvent(state))

This comment has been minimized.

@ruflin

ruflin Apr 13, 2017

Collaborator

I changed it to return directly if state is not finished but I kept the error handling local.

@@ -94,24 +94,35 @@ func (l *Log) Run() {
if l.config.CleanRemoved {
for _, state := range l.Prospector.states.GetStates() {
// os.Stat will return an error in case the file does not exist
_, err := os.Stat(state.Source)
stat, err := os.Stat(state.Source)
if err != nil {

This comment has been minimized.

@ruflin

ruflin Apr 13, 2017

Collaborator

changed

@7AC

7AC approved these changes Apr 18, 2017

@7AC 7AC merged commit caf398f into elastic:master Apr 18, 2017

6 checks passed

CLA Commit author has signed the CLA
Details
codecov/patch 86.95% of diff hit (target 64.34%)
Details
codecov/project Absolute coverage decreased by -11.39% but relative coverage increased by +22.61% compared to 81c5957
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
default Build finished.
Details

@ruflin ruflin deleted the ruflin:improved-clean-removed branch Apr 18, 2017

@tsg tsg referenced this pull request Jul 24, 2017

Closed

Document breaking changes in 6.0 #4737

28 of 28 tasks complete

athom added a commit to athom/beats that referenced this pull request Jan 25, 2018

Modify clean_removed handling (elastic#3827)
* Modify clean_removed handling

Previously if a file could be found under the same name as a state, the state was not removed. But this could have been also an other file. With this change also the file itself is compared and if it is not the same file, the state will be removed. This has the affect that in case a file is renamed after monitoring the file finished, the state will be removed. In most cases this should not have any side affect.

The positive effect of this change is that there will be less left over states in the registry.

Closes elastic#3789

* implement review changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment