Add "timeout" to CheckConfig

[grepory]

CheckConfigs should have a Timeout field that specifies the maximum duration that a check process can take to return--after which we should terminate the external process and publish a check result with a status of 2 and an output of "Execution timed out".

[nikkictl]

golang/go#22485, golang/go#18874, and golang/go#23019 document a similar issue I discovered in the command.go function ExecuteCommand. exec.CommandContextonly kills the immediate child process and not any additional spawned processes, causing cmd.Wait() to hang indefinitely. This does not impact invocations of single commands for ex. sleep 10, but will hang if execution.Command spawns multiple processes, ex. sleep 10; echo hanging. Some solutions include killing the process group which has OS specific implications, forcefully escaping cmd.Wait() which potentially leaves orphaned processes running, or not invoking the shell.

[nikkictl]

I also find this 1.x bug particularly interesting... @cwjohnston @agoddard I'd love your input if you have some context on the 1.x issue, as it seems like it could be hit from a similar use case here. From golang/go#18874, "The difference in shell behavior here is that bash, zsh, etc will exec a single command in place, where as dash always executes in a sub process."

[grepory]

Similar behavior but different cause.

[nikkictl]

@grepory ah.. I found it curious since we were still invoking a shell https://github.com/sensu/sensu-spawn/blob/f7149f9660f530ce777ae8ed4fb29b28eb2593a1/lib/sensu/spawn.rb#L83-L97.. trying to dig into how we kill child processes in 1.x...

How we appear to kill child processes on Windows and Unix in Sensu 1.x.

[palourde]

Here's a common solution we found after our quick pairing this morning: https://github.com/hashicorp/consul/tree/master/agent/exec