Extend default signal grace period to 9 seconds

[triarius]

Description

When a job is cancelled, the agent sends a SIGTERM to the bootstrap process of the job. This immediately sends a SIGTERM¹ to child processes of the bootstrap process. After the signal-grace-period, a SIGKILL is sent to these child processes and the bootstrap process exits. Currently, the signal-grace-period, though configurable, defaulted to 0s. This was done to simulate the previous behaviour where a SIGKILL was sent immediately. This PR changes the default to 9s. This value was chosen because it is the largest integer less than 10, which is the default value of the cancel-grace-period. That is the time the bootstrap process has to exit before it is itself sent a SIGKILL by the agent worker process.

Context

#2250
#2654

Changes

• signal-grace-period-seconds is now 9 by default.
• a more helpful error message when the assertion signal-grace-period-seconds < cancel-grace-period fails.

Testing

• Tests have run locally (with go test ./...). Buildkite employees may check this if the pipeline has run automatically.
• Code is formatted (with go fmt ./...)

Footnotes

1. It's SIGTERM by default but that's configurable. ↩