Serf reconnect

[rmoriz]

When using goiardi with serf for shovey/shob, goiardi does not reconnect to serf when the connection gets dropped (e.g. serf instance restarts).
Log:

2017/07/28 16:38:57 [ERR] agent.client: Failed to decode response header: EOF

[ctdk]

Thanks for the heads up. Also-also, I must say I'm a bit surprised (pleased, but surprised) that someone's actually doing something with shovey -- after I went through the effort to get it all up and going I found to my disappointment and mild annoyance that no one (including myself) had much of a use case for it. Incidentally, I've been told that state of events also applies to Chef Push.

[rmoriz]

Well… I'm a bit in a hate-love relationship with Chef since 2010 so I could tell a loooong story ;-)

But to cut it short:

The only push-job use case is IMHO to force a chef-client run (which then deploys a new app release or whatever).

Some people use consul but I though why not give shovey a try. And it works nice and Serf seems to be a lot less heavy in operations as Consul.

[ctdk]

Take a look at https://github.com/ctdk/goiardi/tree/serf-reconnecting and let me know whether or not that clears up the serf reconnecting issue.

[rmoriz]

Looks good, thank you!

Here is the log when using the branch. I killed and restarted serf:

2017/08/11 17:23:40 [DEBUG] [main] (goiardi.go:runEventMonitor:715) Got an event: map[LTime:655150 Name:node_status Payload:[...] Event:query ID:39] nil? false
2017/08/11 17:23:40 [ERR] agent.client: Failed to decode response header: EOF
2017/08/11 17:23:40 [ERR] agent.client: Failed to decode response header: EOF
2017/08/11 17:23:40 [DEBUG] [main] (goiardi.go:runEventMonitor:715) Got an event: map[] nil? true
2017/08/11 17:23:40 [DEBUG] [main] (goiardi.go:runEventMonitor:718) Serf client has been closed, returning from runEventMonitor in hopes of being able to reconnect
2017/08/11 17:23:40 [ERROR] [main] Error from event monitor: serf client closed
2017/08/11 17:23:40 [ERROR] [main] Error recreating serf client, waiting 5 seconds before recreating: dial tcp 178.63.77.21:7373: getsockopt: connection refused
2017/08/11 17:23:46 [DEBUG] [main] (goiardi.go:runEventMonitor:715) Got an event: map[Event:query ID:1 LTime:655154 Name:node_status Payload:[...]] nil? false

A nitpick: Maybe you can print a message once the reconnect worked. I assume it is before the "Got an event" message.

[ctdk]

This has been merged into master and released.