Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow filebeat to only run once

Merged
merged 2 commits into from Sep 16, 2016
Merged

Allow filebeat to only run once #2456

merged 2 commits into from Sep 16, 2016

Conversation

ruflin
Copy link
Collaborator

@ruflin ruflin commented Sep 5, 2016

When the -once flag is used, filebeat starts all configured harvesters and prospectors and runs each prospector until the harvesters are closed. It is recommended to use the flag in combination with close_eof so harvester directly close when the end of the file is reached. By default harvesters are closed after close_inactive.

Closes #880

@ruflin ruflin added in progress review Filebeat labels Sep 5, 2016
if *once {
logp.Debug("filebeat", "Running filebeat once. Waiting for completion ...")
spooler.Flush()
publisher.Publish()
Copy link
Collaborator Author

@ruflin ruflin Sep 5, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@urso What I'm trying to do here is to make sure that all events which are in the queue are sent. Is there a better way to wait / check the publisher if all events were sent?

ruflin added a commit to ruflin/beats that referenced this issue Sep 5, 2016
* Separate async and sync publisher into own files
* Add Publish interface method which can be used to manually trigger publish (see elastic#2456)
* Add getDataEvents function
ruflin added a commit to ruflin/beats that referenced this issue Sep 5, 2016
* Separate async and sync publisher into own files
* Add Publish interface method which can be used to manually trigger publish (see elastic#2456)
* Add getDataEvents function
* Rename publish to publisher package
ruflin added a commit to ruflin/beats that referenced this issue Sep 5, 2016
This metric can be useful in combination with elastic#2456 to track how long the harvesting of a file took.
urso pushed a commit that referenced this issue Sep 5, 2016
* Separate async and sync publisher into own files
* Add Publish interface method which can be used to manually trigger publish (see #2456)
* Add getDataEvents function
* Rename publish to publisher package
ruflin added a commit to ruflin/beats that referenced this issue Sep 5, 2016
This metric can be useful in combination with elastic#2456 to track how long the harvesting of a file took. Uptime is reported before shutdown.
tsg pushed a commit that referenced this issue Sep 6, 2016
This metric can be useful in combination with #2456 to track how long the harvesting of a file took. Uptime is reported before shutdown.
@ruflin ruflin force-pushed the run-once branch 2 times, most recently from 6830490 to dfd46c9 Compare Sep 9, 2016
@ruflin ruflin removed the in progress label Sep 9, 2016
var once *bool

func init() {
once = flag.Bool("once", false, "Run filebeat only once until all harvesters reach EOF")
Copy link

@urso urso Sep 9, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can be initialized directly at line 19.

Copy link
Collaborator Author

@ruflin ruflin Sep 9, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed

@ruflin ruflin force-pushed the run-once branch 2 times, most recently from 690a9a9 to 41e9c9b Compare Sep 12, 2016
wgEvents = &sync.WaitGroup{}
finishedLogger = newFinishedLogger(wgEvents)
}
//if fb.config.ShutdownTimeout > 0 || *once {
Copy link
Collaborator Author

@ruflin ruflin Sep 13, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@urso do you see an issue with this? As long as we don't use wgEvents nothing will happen.

Copy link

@urso urso Sep 13, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's correct. as long as wgEvent.Wait is not installed in sigWait there is no difference in initializing these always. I just didn't want to have all the wgEvent update by default, but overhead should be ok.

Copy link
Collaborator Author

@ruflin ruflin Sep 13, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will leave it in as default for the moment and we can still make it optional in case it becomes a performance issue (I would not assume so)

}

fb.sigWait.AddChan(fb.done)
fb.sigWait.Wait()
Copy link

@urso urso Sep 13, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's use a separate signalWait waiting for runOnce or fb.done to finished. This way the waiter waiting for shutdown and the one waiting for events published will not get mixed up. The fb.done channel can be installed multiple times. With channel being closed, it will send a signal immediately.

@ruflin ruflin force-pushed the run-once branch 3 times, most recently from 1593f58 to a8a7100 Compare Sep 15, 2016
When the `-once` flag is used, filebeat starts all configured harvesters and prospectors and runs each prospector until the harvesters are closed. It is recommended to use the flag in combination with `close_eof` so harvester directly close when the end of the file is reached. By default harvesters are closed after `close_inactive`.
logp.Info("Shutdown output timer started. Waiting for max %v.", timeout)
waitEvents.Add(withLog(waitDuration(timeout),
"Continue shutdown: Time out waiting for events being published."))
}
Copy link

@urso urso Sep 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add:

} else {
    waitEvents.AddChan(fb.done)
}

This will ensure CTRL-C will be handled immediately/properly if run-once without shutdown_timeout is configured. If fb.done has been closed already, waitEvents.Wait will return immediately. If fb.done is closed while waiting for wgEvents.Wait, shutdown will continue.

Copy link
Collaborator Author

@ruflin ruflin Sep 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Added.

@urso urso merged commit 6b7df0c into elastic:master Sep 16, 2016
4 checks passed
suraj-soni pushed a commit to suraj-soni/beats that referenced this issue Sep 20, 2016
* Separate async and sync publisher into own files
* Add Publish interface method which can be used to manually trigger publish (see elastic#2456)
* Add getDataEvents function
* Rename publish to publisher package
suraj-soni pushed a commit to suraj-soni/beats that referenced this issue Sep 20, 2016
This metric can be useful in combination with elastic#2456 to track how long the harvesting of a file took. Uptime is reported before shutdown.
suraj-soni pushed a commit to suraj-soni/beats that referenced this issue Sep 20, 2016
* Allow filebeat to only run once

When the `-once` flag is used, filebeat starts all configured harvesters and prospectors and runs each prospector until the harvesters are closed. It is recommended to use the flag in combination with `close_eof` so harvester directly close when the end of the file is reached. By default harvesters are closed after `close_inactive`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants