Migration to sealed secrets with existing kube secrets

[marc-ostan]

We are trying to migrate from existing(unsealed) kubernetes secrets to sealed secrets, and we found that a sealed secret doesn't replace an unsealed one. kubectl apply of the sealed secret succeeds, but we see this error message in the sealed-secrets-controller log:

2019/10/01 18:53:56 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"kubelife-test", Name:"sealed-created-secret", UID:"bc83859a-e47c-11e9-8489-0a02f9f20c90", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"132849818", FieldPath:""}): type: 'Warning' reason: 'ErrUpdateFailed' Resource "sealed-created-secret" already exists and is not managed by SealedSecret

This makes for a bit of a racy migration. If we delete the old/unsealed secret first, a pod that uses the secret could fail to start in the time it takes to apply the corresponding sealed secret.

What do you think of adding an annotation to the sealed secret to indicate that it's OK to take ownership of the underlying secret? Or maybe some other way to handle this?

[mkmik]

Yep, makes sense. Perhaps an annotation

[marc-ostan]

We'd submit a PR with implementation of this functionality, but we are failing to build docker image of controller that would run.
I’m using go1.12 on MacOs Mojave, and after I build the image and trying to run it, it crashes with following error:
standard_init_linux.go:211: exec user process caused "exec format error"

[Crevil]

That sounds like your building it for darwin and run it on linux. I think the docker image uses an Alpine base.

[mkmik]

$ GOOS=linux make controller.image CONTROLLER_IMAGE=yourimage:latest

[mkmik]

Another possibility: in #224 we discuss whether to improve sealed-secrets so that it reacts to when a secret is deleted. The main scenario there was to remove the confusion that arises when somebody accidentally removes a secret.

However, if we had that feature it could come handy in this scenario too: all you have to do is to first add your new sealed secrets resources and gradually delete the old ones.

WDYT?

[dbyron0]

I like #224, but isn't there still a race if that's all we do? Maybe there's no way to close the hole and #224 really is enough? Like, once #224 is done, isn't there a window in time after deleting the secret and before the sealed secrets controller makes the new one where a pod might fail to start because the secret doesn't exist?

[mkmik]

yes, but such a race is always possible in k8s due to asynchronous nature of k8s controllers and k8s is designed to converge to a good state (e.g. pods will be restarted until the secret exists)

[dbyron0]

Yup, thanks for confirming. Then I suspect #224 is enough. You agree @mkmik ?

[mkmik]

Yes.

[dbyron0]

Great. Any chance you have plans for work on #224 soon :) ?

[mkmik]

With this use case the feature graduated from nice to have to something potentially blocking adoption, so yes

[dbyron0]

Lovely. Thank you.

[dbyron0]

I guess I commented on the wrong issue the first time (#224 (comment)), so trying again here. Do https://github.com/bitnami-labs/sealed-secrets/blob/master/RELEASE-NOTES.md#allow-overwriting-existing-secrets and #331 fix this issue?

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[juan131]

We'll address this use case once we address #224