SealedSecret status stuck "Progressing" even when unsealed successfully #5991

mbouchenoire · 2021-04-08T10:31:13Z

Checklist:

I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
I've included steps to reproduce the bug.
I've pasted the output of argocd version.

Describe the bug

After upgrading to v2.0.0, SealedSecret resources are stuck in the "Progressing" state, even when the secret was unsealed successfully by the controller.

To Reproduce

Install quay.io/bitnami/sealed-secrets-controller:v0.15.0
Create an Argo CD Application containing a SealedSecret resource
Wait for the resource to be synced by Argo CD and unsealed by the controller
The SealedSecret managed by Argo CD has a "Progressing" health and "Waiting for Sealed Secret to be decrypted" as health details

Expected behavior

The SealedSecret health should be "Healthy"

Screenshots

The SealedSecret manifest:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"bitnami.com/v1alpha1","kind":"SealedSecret","metadata":{"annotations":{},"creationTimestamp":null,"labels":{"app.kubernetes.io/instance":"xxxx"},"name":"xxxxx","namespace":"xxxxx"},"spec":{"encryptedData":{"xxxxx"}}}}
  creationTimestamp: '2021-03-29T17:40:00Z'
  generation: 1
  labels:
    app.kubernetes.io/instance: secrets-v3-1
  name: xxxxx
  namespace: xxxxx
  resourceVersion: '239233797'
  selfLink: /apis/bitnami.com/v1alpha1/namespaces/xxxxx/sealedsecrets/xxxxx
  uid: e66a6863-4b1d-43f2-96e0-fc687cdaabd6
spec:
  encryptedData:
    xxxxxxx
  template:
    metadata:
      creationTimestamp: null
      name: xxxxx
      namespace: xxxxx

Version

{
    "Version": "v2.0.0+f5119c0",
    "BuildDate": "2021-04-07T06:00:33Z",
    "GitCommit": "f5119c06686399134b3f296d44445bcdbc778d42",
    "GitTreeState": "clean",
    "GoVersion": "go1.16",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KsonnetVersion": "v0.13.1",
    "KustomizeVersion": "v3.9.4 2021-02-09T19:22:10Z",
    "HelmVersion": "v3.5.1+g32c2223",
    "KubectlVersion": "v0.20.4",
    "JsonnetVersion": "v0.17.0"
}

Logs

The text was updated successfully, but these errors were encountered:

queil · 2021-04-08T10:33:22Z

@mbouchenoire Thanks for that. We've upgraded to v2.0 2 minutes ago and facing the same issue.

queil · 2021-04-08T10:43:43Z

It looks like the lua script checks the status field. However, in our case status is empty.

Edit: In our case this will definitely fail because of an old version of Sealed Secrets. Will try to upgrade and see.

jannfis · 2021-04-08T11:18:14Z

Hey - you checked "I've searched the FAQ" but you seem to have missed https://argo-cd.readthedocs.io/en/stable/faq/#why-are-resources-of-type-sealedsecret-stuck-in-the-progressing-state :)

queil · 2021-04-08T11:25:33Z

@jannfis Yeah, we also have just found it in Sealed Secrets release notes. Thanks

mbouchenoire · 2021-04-08T11:58:14Z

Hey - you checked "I've searched the FAQ" but you seem to have missed https://argo-cd.readthedocs.io/en/stable/faq/#why-are-resources-of-type-sealedsecret-stuck-in-the-progressing-state :)

Indeed ! I think I made this mistake because I assumed the answer would be included in this chapter : Why is my application stuck in Progressing state?. Is it worth considering merging these two ?

Tank you for the link :)

jannfis · 2021-04-08T12:15:01Z

@mbouchenoire Yeah, good suggestion about merging them - or at least, linking to the SealedSecrets FAQ entry from the "Why is my application stuck..." one. Do you mind sending a PR?

Also, we hoped for SealedSecrets to release v0.16.0 (which is supposed to have the status update enabled by default again) before our 2.0 release. Unfortunately, we were quicker :)

jef · 2021-04-13T05:17:32Z

Related: bitnami-labs/sealed-secrets#555, #5971

YevheniiPokhvalii · 2021-05-25T15:42:02Z

This might help:

Set the SEALED_SECRETS_UPDATE_STATUS variable to "1" in the sealed-secrets deployment yaml file https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/controller.yaml

apiVersion: apps/v1
kind: Deployment

    spec:
      containers:
      - args: []
        command:
        - controller
        env:
        - name: SEALED_SECRETS_UPDATE_STATUS
          value: "1"

jef · 2021-05-27T15:22:53Z

Yep, looks like this is fixed now in the latest v0.16.0 release. Either use the env var like @YevheniiPokhvalii mentioned or the --update-status flag. It should be on by default in v0.17.0.

haim-ari · 2021-08-01T08:41:33Z

I'm using flux to install sealed-secrets with Helm Release.

helm search repo sealed-secrets/sealed-secrets --versions

NAME                         	CHART VERSION	APP VERSION	DESCRIPTION
sealed-secrets/sealed-secrets	1.16.1       	v0.16.0    	Helm chart for the sealed-secrets controller.
sealed-secrets/sealed-secrets	1.16.0       	v0.16.0    	Helm chart for the sealed-secrets controller.
sealed-secrets/sealed-secrets	1.13.2       	0.13.1     	A Helm chart for Sealed Secrets

I upgraded to 1.16.1 (app v.0.16.0) and added the following:

  chart:
    spec:
      chart: sealed-secrets
      version: "1.16.1"
      sourceRef:
        kind: HelmRepository
        name: sealed-secrets
        namespace: flux-system
  interval: 10m
  values:
    fullnameOverride: sealed-secrets-controller
    controller:
      env:
      - name: SEALED_SECRETS_UPDATE_STATUS
        value: "1"

but when I'm checking the deployment I don't see the Env Var added even though I can see that the controller was restarted:

pod/sealed-secrets-controller-68f997cf86-xg2gs   1/1     Running   0          37m
service/sealed-secrets-controller            ClusterIP   10.43.123.203   <none>        8080/TCP                       24d
deployment.apps/sealed-secrets-controller   1/1     1            1           24d
replicaset.apps/sealed-secrets-controller-5c56b54f8f   0         0         0       24d
replicaset.apps/sealed-secrets-controller-68f997cf86   1         1         1       37m

Should I manually delete the crd :

sealedsecrets.bitnami.com 2021-05-25T07:44:35Z

What is the right way to set --update-status with Helm ?

Update

I've also tried (bad practice, i know but for testing) to manually edit the deployment .
I see the new controller pod has the Env var set, but even after I delete a Sealed-Secret, It still does not shown OK in argocd, even though it was decrypted successfully and the new secret was created.

from running pod:

spec:
  containers:
  - args:
    - --key-prefix
    - sealed-secrets-key
    command:
    - controller
    env:
    - name: SEALED_SECRETS_UPDATE_STATUS
      value: "1"
    image: quay.io/bitnami/sealed-secrets-controller:v0.16.0

ismailyenigul · 2021-08-01T15:43:33Z

Hi @haim-ari
I had the same issue on argo-cd.
Just upgraded sealed-secrets to 1.16.1 helm chart with the following value commandArgs. then argocd stop progressing and reports Healthy.

# secretName: The name of the TLS secret containing the key used to encrypt secrets
secretName: "sealed-secrets-key"
commandArgs:
  - "--update-status"

haim-ari · 2021-08-01T16:22:16Z

Hi @haim-ari
I had the same issue on argo-cd.
Just upgraded sealed-secrets to 1.16.1 helm chart with the following value commandArgs. then argocd stop progressing and reports Healthy.
# secretName: The name of the TLS secret containing the key used to encrypt secrets
secretName: "sealed-secrets-key"
commandArgs:
  - "--update-status"

Yes, I tried that as well, same result:

Waiting for Sealed Secret to be decrypted

But a new secret was created successfully:

type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully

haim-ari · 2021-08-01T17:04:03Z

Seems that this was related to CRD somehow (not sure why but could be related to the fact that this was an upgrade of Helm)
removed the CRD and manually added it back.

Now ArgoCD shows the correct state.

haim-ari · 2021-08-01T17:15:47Z

I can confirm this also worked on another cluster.
After deleting and re-creating CRD.

shibumi · 2021-08-05T14:45:54Z

@haim-ari you said you added a new command flag to sealed-secrets version 1.16.1. However, I cannot find it in the sealed secrets repository:

https://github.com/bitnami-labs/sealed-secrets/tree/main/helm/sealed-secrets

Did you speak about a private fork?

EDIT: Sorry, looks like I meant @ismailyenigul

EDIT2: Ah okay.. the commandArgs value is just not mentioned in the values.yaml of version 1.16.1. The value exists in the templates directory.

ismailyenigul · 2021-08-05T15:38:36Z

Hi @shibumi
yes simply we are adding value to commandArgs in values.yml of helm chart
it is defined at https://github.com/bitnami-labs/sealed-secrets/blob/main/helm/sealed-secrets/templates/deployment.yaml#L49

pmorch · 2021-09-03T10:27:49Z

I can confirm that this works:

$  helm upgrade --install --namespace kube-system sealed-secrets-controller sealed-secrets/sealed-secrets \
     --version 1.16.1 --set 'commandArgs[0]=--update-status'

My SealedSecret-s are no longer progressing

argoproj/argo-cd#5991

ismailyenigul · 2021-12-10T13:42:45Z

removed the CRD and manually added it back.

I experienced the same issue. I deleted helm chart that already had --update-status and created again
but argo-cd was still reporting Waiting for Sealed Secret to be decrypted
deleting manually CRD did not help.
deleted with helm uninstall did not help too.

I deleted everything about sealed-secret with
kubectl delete -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/controller.yaml
and created with
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/controller.yaml
not argo-cd reports Healthy and stop infinitive progressing!

deleted again with kubectl and installed with helm and it is still good.
I don't understand what is going on with CRD but it is definitely a CRD issue!

dirien · 2021-12-31T19:03:45Z

Hi @ismailyenigul @haim-ari @shibumi,

can confirm with the commandArgs is working fine! Finally no Lua stuff needed :) thanks!

anggutie-dev · 2022-01-10T18:10:14Z

upgrading sealed secrets to v0.17.1 did the trick, Argocd is now showing "Healthy"

dirien · 2022-01-10T18:18:02Z

Hi @anggutie-dev, so its safe to use the latest version of the chart?

vaibhavjain882 · 2022-01-18T13:59:48Z

Hi @ismailyenigul @haim-ari @shibumi,

can confirm with the commandArgs is working fine! Finally no Lua stuff needed :) thanks!

What are you getting in your sealed secret object events in the agro UI? I can see it is healthy but in the events, its shows "Failed to unseal: no key could decrypt secret".

ismailyenigul · 2022-01-18T14:08:15Z

Hi @anggutie-dev, so its safe to use the latest version of the chart?

Yes. because it is not sealed-secret code update. it is just helm update.

ismailyenigul · 2022-01-18T14:09:39Z

Hi @ismailyenigul @haim-ari @shibumi,
can confirm with the commandArgs is working fine! Finally no Lua stuff needed :) thanks!

What are you getting in your sealed secret object events in the agro UI? I can see it is healthy but in the events, its shows "Failed to unseal: no key could decrypt secret".

This is not something about sealed-secrets configuration.
Maybe you signed secret with non-existed key or sealed for different namespace.
How did you create this sealedsecret?

MaxWinterstein · 2022-01-29T14:37:19Z

Same here, after few rollouts it was stuck while waiting for something of sealed sealed secrets.

Updating from 1.6.x to 1.7.3 seems to fix it for me.

TobiasGoerke · 2022-03-31T12:20:39Z

Same here, after few rollouts it was stuck while waiting for something of sealed sealed secrets.

Updating from 1.6.x to 1.7.3 seems to fix it for me.

Can confirm, the current version of SealedSecrets does not cause this issue (tested with v0.17.3)

crenshaw-dev · 2023-07-07T15:04:22Z

Looks like there's a documented workaround for <=1.16 and everything Just Works in >=0.17. Let me know if I need to reopen!

mbouchenoire added the bug Something isn't working label Apr 8, 2021

mbouchenoire mentioned this issue Apr 8, 2021

docs: Link issues related to health checks in FAQ #5993

Merged

10 tasks

SaigyoujiYuyuko233 mentioned this issue Jun 25, 2021

App sync stuck in "waiting for health state" but actually the resources is already in ready state #6572

Closed

3 tasks

joaofnds added a commit to joaofnds/foobar-infra that referenced this issue Nov 8, 2021

add update-status command arg to sealed secrets

feb5d37

argoproj/argo-cd#5991

blackandred mentioned this issue Nov 30, 2021

Resolve: SealedSecret status stuck "Progressing" even when unsealed successfully riotkit-org/k3s-space-harbor#6

Closed

crenshaw-dev closed this as completed Jul 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SealedSecret status stuck "Progressing" even when unsealed successfully #5991

SealedSecret status stuck "Progressing" even when unsealed successfully #5991

mbouchenoire commented Apr 8, 2021

queil commented Apr 8, 2021

queil commented Apr 8, 2021 •

edited

jannfis commented Apr 8, 2021

queil commented Apr 8, 2021

mbouchenoire commented Apr 8, 2021

jannfis commented Apr 8, 2021

jef commented Apr 13, 2021

YevheniiPokhvalii commented May 25, 2021

jef commented May 27, 2021

haim-ari commented Aug 1, 2021 •

edited

ismailyenigul commented Aug 1, 2021 •

edited

haim-ari commented Aug 1, 2021

haim-ari commented Aug 1, 2021 •

edited

haim-ari commented Aug 1, 2021

shibumi commented Aug 5, 2021 •

edited

ismailyenigul commented Aug 5, 2021

pmorch commented Sep 3, 2021

ismailyenigul commented Dec 10, 2021

dirien commented Dec 31, 2021

anggutie-dev commented Jan 10, 2022 •

edited

dirien commented Jan 10, 2022

vaibhavjain882 commented Jan 18, 2022

ismailyenigul commented Jan 18, 2022

ismailyenigul commented Jan 18, 2022 •

edited

MaxWinterstein commented Jan 29, 2022

TobiasGoerke commented Mar 31, 2022 •

edited

crenshaw-dev commented Jul 7, 2023

SealedSecret status stuck "Progressing" even when unsealed successfully #5991

SealedSecret status stuck "Progressing" even when unsealed successfully #5991

Comments

mbouchenoire commented Apr 8, 2021

queil commented Apr 8, 2021

queil commented Apr 8, 2021 • edited

jannfis commented Apr 8, 2021

queil commented Apr 8, 2021

mbouchenoire commented Apr 8, 2021

jannfis commented Apr 8, 2021

jef commented Apr 13, 2021

YevheniiPokhvalii commented May 25, 2021

jef commented May 27, 2021

haim-ari commented Aug 1, 2021 • edited

from running pod:

ismailyenigul commented Aug 1, 2021 • edited

haim-ari commented Aug 1, 2021

haim-ari commented Aug 1, 2021 • edited

haim-ari commented Aug 1, 2021

shibumi commented Aug 5, 2021 • edited

ismailyenigul commented Aug 5, 2021

pmorch commented Sep 3, 2021

ismailyenigul commented Dec 10, 2021

dirien commented Dec 31, 2021

anggutie-dev commented Jan 10, 2022 • edited

dirien commented Jan 10, 2022

vaibhavjain882 commented Jan 18, 2022

ismailyenigul commented Jan 18, 2022

ismailyenigul commented Jan 18, 2022 • edited

MaxWinterstein commented Jan 29, 2022

TobiasGoerke commented Mar 31, 2022 • edited

crenshaw-dev commented Jul 7, 2023

queil commented Apr 8, 2021 •

edited

haim-ari commented Aug 1, 2021 •

edited

ismailyenigul commented Aug 1, 2021 •

edited

haim-ari commented Aug 1, 2021 •

edited

shibumi commented Aug 5, 2021 •

edited

anggutie-dev commented Jan 10, 2022 •

edited

ismailyenigul commented Jan 18, 2022 •

edited

TobiasGoerke commented Mar 31, 2022 •

edited