Upon restart Filebeat won't wait for old instance to shutdown leading to registry update problems

[shoggeh]

Discussed in:

https://discuss.elastic.co/t/filebeat-rc1-resends-full-data-set-data-upon-random-restart-registry-file-not-updated-properly/63580/17

• Version: Filebeat 5.0 GA and other 5.0s
• Steps to Reproduce:

As discussed in forum thread, upon restart Filebeat won't wait for old filebeat instance to shutdown before spawning new instance. This apparently leads to a situation where new instance stops updating registry file after a moment. As a final result every sub-sequential restart will trigger entire dataset collected from that point to be reread/resend.

As per @ruflin's notes on the forum it looks like shutdown_timeout only waits for filebeat-god to shutdown, but not for filebeat instance itself.

Silly workaround: Adding sleep loop into restart() section of init script seems to mitigate problem.

Quotes from forum thread:

• shutdown_timeout: 60s in the config

# pgrep -l filebeat
29213 filebeat-god
29214 filebeat

# time service filebeat restart
Config OK
Stopping filebeat:                    [  OK  ]
Starting filebeat: Config OK          [  OK  ]

real	0m3.309s
user	0m0.067s
sys	0m0.054s

# pgrep -l filebeat
29214 filebeat
29563 filebeat-god
29564 filebeat

Current time: 11:47 UTC,
node1: process restart at 05:54, last on-disc registry file update: 06:00
node2: process restart at 08:30, last on-disc registry file update: 08:35
node3: process restart at 05:05, last on-disc registry file update: 05:06
node4: process restart at 07:32, last on-disc registry file update: 07:32
node5: process restart at 11:20, last on-disc registry file update: 11:21
node6: process restart at 05:40, last on-disc registry file update: 05:50
node7: process restart at 04:43, last on-disc registry file update: 04:47

[ruflin]

@shoggeh Thanks for opening the issue. I already opened one here: #2951 Could you paste the content from here to a comment in #2951 ?

[shoggeh]

Copied & closing as a dup of #2951