Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better signal handling (for Job cancellations) #392

Merged
merged 19 commits into from
Jun 17, 2023
Merged

Conversation

toote
Copy link
Contributor

@toote toote commented May 23, 2023

This adds signal handling on INT, TERM and QUIT on the command hook script so that the main container is stopped gracefully. It also gets logs for the container if not removed (though technically only strictly necessary for compose v1 as v2 still shows the output until the container is killed).

I have tested this with the following ruby code:

begin
  (1..20).each do |x|
    puts "#{x}\n"
    STDOUT.flush
    sleep 10
  end
rescue SignalException => e
  puts  "\nSignal: #{e.signo} / #{e}"
  sleep 2
  exit e.signo
end

Save that file and use docker compose to run the file in a vanilla ruby container. Once it is running, cancel the job.

Without this code, the step cancels immediately, but the main container is still running and (by default) killed during the pre-exit hook. Even when using graceful-termination so that the container is stopped instead, you are missing the actual container output since the job was cancelled and then removed (which is now guaranteed by #386).

With this code, instead, the container gets sent its corresponding stop signal (as defined in its dockerfile)

Closes #389

@toote toote requested a review from pzeballos May 23, 2023 20:21
commands/run.sh Show resolved Hide resolved
@pzeballos pzeballos merged commit 73c55e7 into master Jun 17, 2023
1 check passed
@pzeballos pzeballos deleted the toote_signal_handling branch June 17, 2023 00:00
zzak added a commit to zzak/buildkite-config that referenced this pull request Feb 17, 2024
atosbucket added a commit to atosbucket/rails-buildkit that referenced this pull request Mar 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cancelled jobs do not stop main/run container
3 participants