New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Configure proxy container for graceful termination #425

Merged

hessjcg merged 4 commits into main from gh-361-stop-hooks

Sep 19, 2023

Collaborator

hessjcg commented Sep 1, 2023 •

edited

Loading

Add the following configuration to the workload pods so that the proxy can gracefully exit
when the main container is done.

Configure the proxy container to exit 0 when it is terminated. Send a SIGTERM to the proxy container. We want
the proxy to exit with code 0, indicating a clean termination. Without this change the proxy container would
exit with exit code 140 (meaning terminated), which would cause kubernetes to report the
pod as "exited in an error state."

Configure a workload lifecycle handler so that kubernetes calls GET /quitquitquit before terminating the proxy container.
This should give the proxy container the chance to exit gracefully before kubernetes sends a SIGTERM to the
proxy process.

Always enable the /quitquitquit proxy api.

Always set the CSQL_QUIT_URLS environment variable to a space-separated list of proxy quitquitquit urls. This way, when
the main workload container is ready to exit, it can on workload pods. When a job or cronjob's main process
is done, that container can iterate over

echo Starting job

# execute the job process
run_job

# Tell proxy containers to shut down gracefully
for url in $CSQL_QUIT_URLS ; do 
   wget --post-data '' $url
done

Fixes #361

hessjcg marked this pull request as ready for review

September 1, 2023 16:53

hessjcg requested a review from a team as a code owner

September 1, 2023 16:53

blunderbuss-gcf bot assigned enocom

hessjcg added the tests: run label

github-actions bot removed the tests: run label

Collaborator Author

hessjcg commented Sep 1, 2023

Labeled tests: run to invoke e2e tests on this PR.

enocom reviewed

View reviewed changes

Member

enocom left a comment

Do we need to add something to the documentation for this?

hessjcg mentioned this pull request

Update documentation #400

Open

Collaborator Author

hessjcg commented Sep 1, 2023

Yes, we need to update the docs, but I think that will be a bigger task. Issue #400 asks us to create a new doc that describes the behavior in detail. I added a requirement to describe the behavior introduced by this PR to #400.

Collaborator Author

hessjcg commented Sep 1, 2023

E2E tests passed: https://github.com/GoogleCloudPlatform/cloud-sql-proxy-operator/actions/runs/6052015286/job/16424854308?pr=425

hessjcg force-pushed the gh-361-stop-hooks branch from c429d91 to 27ff556 Compare

September 1, 2023 17:31

hessjcg requested a review from enocom

September 1, 2023 17:31

enocom reviewed

View reviewed changes

Member

enocom left a comment

Sounds good on #400. Could we update the title? I'm not sure exactly what the scope is as written.

internal/workload/podspec_updates.go

+              	s.addProxyContainerEnvVar(p, "CSQL_PROXY_EXIT_ZERO_ON_SIGTERM", "true")
+              	// Configure the pre-stop hook for /quitquitquit
+              	c.Lifecycle = &corev1.Lifecycle{

Member

enocom Sep 1, 2023

Should we add a test that verifies this behavior?

Collaborator Author

hessjcg Sep 13, 2023

done.

hessjcg changed the title ~~feat: Configure graceful exit 0 on proxy container termination~~ feat: Configure proxy container for graceful termination

hessjcg force-pushed the gh-361-stop-hooks branch 2 times, most recently from 55cf55b to 35e33ff Compare

September 13, 2023 23:04

hessjcg requested review from enocom and ttosta-google

September 13, 2023 23:05

hessjcg force-pushed the gh-361-stop-hooks branch from 35e33ff to 73ea5ad Compare

September 14, 2023 17:23

enocom reviewed

View reviewed changes

internal/workload/podspec_updates_test.go Outdated

		@@ -911,6 +915,91 @@ func TestPodTemplateAnnotations(t *testing.T) {

		}

		func TestQuitUrlEnvVar(t *testing.T) {

Member

enocom Sep 14, 2023

URL.

Watch out for initialisms.

Collaborator Author

hessjcg Sep 14, 2023

Fixed.

internal/workload/podspec_updates_test.go Outdated

+              	csqls[2].ObjectMeta.Generation = 3
+              	var wantQuitUrlsEnv = fmt.Sprintf(
+              		"http://localhost:%d/quitquitquit http://localhost:%d/quitquitquit http://localhost:%d/quitquitquit", workload.DefaultAdminPort,

Member

enocom Sep 14, 2023

nit: shorter line length is easier to read. how about this?

var wantQuitUrlsEnv = strings.Join(
  []string{
    fmt.Sprinf("http://localhost:%d/quitquitquit", workload.DefaultAdminPort),
    fmt.Sprinf("http://localhost:%d/quitquitquit", workload.DefaultAdminPort),
    fmt.Sprinf("http://localhost:%d/quitquitquit", workload.DefaultAdminPort),
  },
  " ",
)

Collaborator Author

hessjcg Sep 14, 2023

Fixed.

internal/workload/podspec_updates_test.go Outdated


		func TestPreStopHook(t *testing.T) {

		var (

Member

enocom Sep 14, 2023

var u = ...

Collaborator Author

hessjcg Sep 14, 2023

Fixed.

hessjcg added a commit that referenced this pull request


          fix: code review comments. related to gh-361-stop-hooks PR #425

9a92643

hessjcg force-pushed the gh-361-stop-hooks branch from 73ea5ad to 9a92643 Compare

September 14, 2023 21:08

hessjcg commented

View reviewed changes

Collaborator Author

hessjcg left a comment

Updated based on Eno's comments.

internal/workload/podspec_updates_test.go Outdated

		@@ -911,6 +915,91 @@ func TestPodTemplateAnnotations(t *testing.T) {

		}

		func TestQuitUrlEnvVar(t *testing.T) {

Collaborator Author

hessjcg Sep 14, 2023

Fixed.

internal/workload/podspec_updates_test.go Outdated

+              	csqls[2].ObjectMeta.Generation = 3
+              	var wantQuitUrlsEnv = fmt.Sprintf(
+              		"http://localhost:%d/quitquitquit http://localhost:%d/quitquitquit http://localhost:%d/quitquitquit", workload.DefaultAdminPort,

Collaborator Author

hessjcg Sep 14, 2023

Fixed.

internal/workload/podspec_updates_test.go Outdated


		func TestPreStopHook(t *testing.T) {

		var (

Collaborator Author

hessjcg Sep 14, 2023

Fixed.

hessjcg added 3 commits

September 14, 2023 15:17


          feat: Configure proxy pod to exit 0 on SIGTERM

46070ad


          feat: configure /quitquitquit and lifecycle events and set the CSQL_Q…

1f736ec

…UIT_URLS envvar


          fix: code review comments. related to gh-361-stop-hooks PR #425

38c4a9a

hessjcg force-pushed the gh-361-stop-hooks branch from 9a92643 to 38c4a9a Compare

September 14, 2023 21:17

hessjcg enabled auto-merge (squash)

September 14, 2023 21:18

enocom reviewed

View reviewed changes

Member

enocom left a comment

LGTM

We need to pin the next release of the proxy before we can merge this, right?

Collaborator Author

hessjcg commented Sep 18, 2023 •

edited

Loading

No, this will work with the current proxy version and represents a significant improvement on the operator behavior.

It will work even better when the proxy is released with PR# 1947

We can merge this PR as-is. And then when the proxy is released, we will update the default proxy image version in a separate PR.

enocom approved these changes

View reviewed changes


          Merge branch 'main' into gh-361-stop-hooks

cc88627

hessjcg merged commit 0e0bb40 into main

8 checks passed

hessjcg deleted the gh-361-stop-hooks branch

September 19, 2023 15:36

release-please bot mentioned this pull request

chore(main): release 1.2.0 #430

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment