Secret key can not be omitted #1936

dtaniwaki · 2019-07-14T15:08:48Z

Describe the bug

Although the comment in secret.yaml indicates server.secretkey can be auto-generated if it's missing.

argo-cd/docs/operator-manual/argocd-secret.yaml

Line 19 in d3c850b

# Autogenerated when missing.

However, the code doesn't allow empty server.secretkey.

argo-cd/util/settings/settings.go

Line 499 in d3c850b

    
           errs = append(errs, &incompleteSettingsError{message: "server.secretkey is missing"})

Which is the expected behavior?

To Reproduce

Just after installing ArgoCD and set admin.password (although this is not mentioned in the docs).

Expected behavior

Asking the expected behavior.

Screenshots

N/A

Version

$ argocd version
argocd: v1.0.2+e0bd546.dirty
  BuildDate: 2019-06-14T17:15:36Z
  GitCommit: e0bd546a07818ec06a27c2b3033454e3eb1c4152
  GitTreeState: dirty
  GoVersion: go1.11.4
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.0.2+e0bd546.dirty
  BuildDate: 2019-06-14T17:15:03Z
  GitCommit: e0bd546a07818ec06a27c2b3033454e3eb1c4152
  GitTreeState: dirty
  GoVersion: go1.11.4
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: 0.13.1

Logs

time="2019-07-14T11:45:39Z" level=info msg="Starting configmap/secret informers"
time="2019-07-14T11:45:39Z" level=info msg="Configmap/secret informer synced"
time="2019-07-14T11:45:39Z" level=fatal msg="server.secretkey is missing"

Have you thought about contributing a fix yourself?

I'm not sure which is the expected behavior.

The text was updated successfully, but these errors were encountered:

jessesuen · 2019-07-20T00:29:09Z

The API server is supposed to generate this. During new installations, we will sometimes see application-controller crash because it started before the api-server had a chance to generate it.

Which service are your logs coming from?

dtaniwaki · 2019-08-29T04:12:39Z

The log came from the application controller.

dtaniwaki · 2019-08-29T04:15:34Z

Uh, the api server was not running because of PSP. I'll try it and update you again.

dtaniwaki · 2019-08-29T09:22:59Z

I confirmed it is created automatically if the api server is running.

emidevi · 2022-01-24T15:46:58Z

Issue: The server.secretkey was missing / did not generate on its own.

What we tried: We took the value of server.secretkey from our production environment and assigned it on our E1 environment where it was missing.

Output: We were getting error as unauthorized and we had no idea why.

Ultimate Solution: Uninstalled and reinstalled argo from scratch and then it generated the server.secretkey on its own.

So the server.secretkey has to have a unique value and should be generated on its own. We cannot assign a value of our own.

alkdese · 2022-06-21T23:10:52Z

Issue: The server.secretkey was missing / did not generate on its own.

What we tried: We took the value of server.secretkey from our production environment and assigned it on our E1 environment where it was missing.

Output: We were getting error as unauthorized and we had no idea why.

Ultimate Solution: Uninstalled and reinstalled argo from scratch and then it generated the server.secretkey on its own.

So the server.secretkey has to have a unique value and should be generated on its own. We cannot assign a value of our own.

In my situation I started having CrashLoopBackoff on argocd-dex-server in the logs I have "server.secretkey is missing" after deleting argocd namespace and creating it from scratch. I just now deleted already 3 times and recreated and still same problem. Any advice?

hyerim-gn · 2022-07-10T13:18:22Z

I solved the problem with node scale out due to the problem caused by the API server not being executed.

emirot · 2022-07-20T22:51:39Z

Seeing this issue after doing:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.4.7/manifests/install.yaml

Then CrashLoopBackOff on argocd-dex-server

kubectl logs argocd-dex-server-56d495844c-l2lxc -n argocd
time="2022-07-20T22:45:06Z" level=info msg="Starting configmap/secret informers"
time="2022-07-20T22:45:06Z" level=info msg="Configmap/secret informer synced"
time="2022-07-20T22:45:06Z" level=fatal msg="server.secretkey is missing"

crenshaw-dev · 2022-07-25T18:04:54Z

@emirot can you confirm that the API server is successfully coming up?

emirot · 2022-07-26T21:48:46Z

@crenshaw-dev sorry about the noise, I figured it out, it's not related, it is due to Open Policy Agent

brianpooe · 2022-10-03T15:59:16Z

@emirot i have the exact same issue how did you resolve yours?

emirot · 2022-10-03T17:51:00Z

@brianpooe In my case I had to do add to add --set global.networkPolicy.create=true in my helm install

winston0410 · 2022-10-12T20:14:10Z

I have the same issue when I install argocd with Core Install. @dtaniwaki Can we reopen this issue?

yevhen-harmonizehr · 2022-10-14T11:26:51Z

i do have same issue after my redis pods restarted. My argo redis do not have PVC, so this might be related.

TeamDman · 2022-11-01T19:54:07Z

#10831 might be related

I had this

$ kubectl get all -n argocd
NAME                                                 READY   STATUS             RESTARTS      AGE
pod/argocd-argo-cd-app-controller-6fbb4ccf8d-7dpnx   1/1     Running            0             17m
pod/argocd-argo-cd-repo-server-9565b9f86-5956t       1/1     Running            0             17m
pod/argocd-argo-cd-server-685dfc4dd7-kgl8r           0/1     CrashLoopBackOff   7 (33s ago)   11m
pod/argocd-redis-master-0                            1/1     Running            0             12m

NAME                                    TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
service/argocd-argo-cd-app-controller   ClusterIP   10.0.189.90   <none>        8082/TCP         17m
service/argocd-argo-cd-repo-server      ClusterIP   10.0.80.110   <none>        8081/TCP         17m
service/argocd-argo-cd-server           ClusterIP   10.0.42.249   <none>        80/TCP,443/TCP   17m
service/argocd-redis-headless           ClusterIP   None          <none>        6379/TCP         17m
service/argocd-redis-master             ClusterIP   10.0.51.188   <none>        6379/TCP         17m

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/argocd-argo-cd-app-controller   1/1     1            1           17m
deployment.apps/argocd-argo-cd-repo-server      1/1     1            1           17m
deployment.apps/argocd-argo-cd-server           0/1     1            0           17m

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/argocd-argo-cd-app-controller-6fbb4ccf8d   1         1         1       17m
replicaset.apps/argocd-argo-cd-repo-server-9565b9f86       1         1         1       17m
replicaset.apps/argocd-argo-cd-server-685dfc4dd7           1         1         0       17m

NAME                                   READY   AGE
statefulset.apps/argocd-redis-master   1/1     17m

$ kubectl logs -n argocd pod/argocd-argo-cd-server-685dfc4dd7-kgl8r
Defaulted container "argocd-server" out of: argocd-server, wait-for-redis (init)
 19:37:13.78 
 19:37:13.79 Welcome to the Bitnami argo-cd container
 19:37:13.79 Subscribe to project updates by watching https://github.com/bitnami/containers
 19:37:13.79 Submit issues and feature requests at https://github.com/bitnami/containers/issues
 19:37:13.79 
 19:37:13.80 INFO  ==> Configuring libnss_wrapper

time="2022-11-01T19:37:13Z" level=info msg="ArgoCD API Server is starting" built="1970-01-01T00:00:00Z" commit= namespace=argocd port=8080 version=v99.99.99+unknown
time="2022-11-01T19:37:13Z" level=info msg="Starting configmap/secret informers"
time="2022-11-01T19:37:14Z" level=info msg="Configmap/secret informer synced"
time="2022-11-01T19:37:14Z" level=info msg="Initialized server signature"
time="2022-11-01T19:37:14Z" level=info msg="admin disabled"
time="2022-11-01T19:37:14Z" level=info msg="Starting configmap/secret informers"
time="2022-11-01T19:37:14Z" level=info msg="configmap informer cancelled"
time="2022-11-01T19:37:14Z" level=info msg="Configmap/secret informer synced"
time="2022-11-01T19:37:14Z" level=info msg="secrets informer cancelled"
panic: server.secretkey is missing

goroutine 1 [running]:
github.com/argoproj/argo-cd/v2/util/session.NewSessionManager(0xc00004d980, {0x44c4aa0, 0xc000fcc580}, {0x34aec23, 0x16}, 0xc000aee5e0?, {0x44ddf10, 0xc0005a1f80})
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/src/github.com/argoproj/argo-cd/util/session/sessionmanager.go:124 +0x3fc
github.com/argoproj/argo-cd/v2/server.NewServer({_, _}, {0x0, 0x0, 0x1, {0x7ffc91ef54aa, 0x18}, 0x1f90, 0x1f93, {0xc000aee5e0, ...}, ...})
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/src/github.com/argoproj/argo-cd/server/server.go:267 +0x5af
github.com/argoproj/argo-cd/v2/cmd/argocd-server/commands.NewCommand.func1(0xc000afea00?, {0x347c98d?, 0xb?, 0xb?})
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/src/github.com/argoproj/argo-cd/cmd/argocd-server/commands/argocd_server.go:192 +0xfa8
github.com/spf13/cobra.(*Command).execute(0xc000afea00, {0xc0000ba010, 0xb, 0xb})
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876 +0x67b
github.com/spf13/cobra.(*Command).ExecuteC(0xc000afea00)
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918
main.main()
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/src/github.com/argoproj/argo-cd/cmd/main.go:57 +0x2c5

$ kubectl logs -n argocd pod/argocd-argo-cd-app-controller-6fbb4ccf8d-7dpnx
Defaulted container "controller" out of: controller, wait-for-redis (init)
 19:21:17.44 
Welcome to the Bitnami argo-cd container
 19:21:17.44 Subscribe to project updates by watching https://github.com/bitnami/containers
 19:21:17.45 Submit issues and feature requests at https://github.com/bitnami/containers/issues
 19:21:17.45 
INFO  ==> Configuring libnss_wrapper
level=info msg="ArgoCD Application Controller is starting" built="1970-01-01T00:00:00Z" commit= namespace=argocd version=v99.99.99+unknown
level=info msg="Processing all cluster shards"
level=info msg="appResyncPeriod=3m0s, appHardResyncPeriod=0s"
level=info msg="Starting configmap/secret informers"
level=info msg="Configmap/secret informer synced"
level=warning msg="Failed to save clusters info: secret \"argocd-secret\" not found"
level=info msg="0xc000ff02a0 subscribed to settings updates"
level=info msg="Starting secretInformer forcluster"
level=warning msg="Failed to save clusters info: secret \"argocd-secret\" not found"
level=warning msg="Unable to parse updated settings: secret \"argocd-secret\" not found"
level=warning msg="Unable to parse updated settings: server.secretkey is missing"
level=warning msg="Unable to parse updated settings: server.secretkey is missing"
level=warning msg="Failed to save clusters info: server.secretkey is missing"
level=info msg="Notifying 1 settings subscribers: [0xc000ff02a0]"
level=warning msg="Unable to parse updated settings: server.secretkey is missing"

Workaround

Manually specifying server.secretKey as an empty string in argocd-secret seems to have fixed the problem without having to manually restart anything after helm applying...
idk if everything is actually working, but the pod stopped crashing and logs look healthy now

gaeljw · 2022-11-17T16:28:37Z

For anyone coming here for similar issue where server.secretKey is not defined, here's what I did to make it work back:

delete the secret
re-create the secret but empty
restart argocd-server

ali-de7 · 2023-08-09T14:16:26Z

I fixed it by simply restarting the server.
Here's what I did to fix it:

kubectl rollout restart deploy/argocd-server

crenshaw-dev · 2023-08-09T14:19:03Z

Reopening because I think this is almost definitely a persistent problem with core install.

pgpx · 2023-09-13T17:41:10Z

Could we split argocd-secret into 2 or more secrets? Leave one for the autogenerated values, another for webhook secrets, and maybe another for the admin password (which is optionally autogenerated)? I'm encountering the same issue trying to introduce a webhook.gitlab.secret via Helm. And/or use server-side-apply for the autogenerated values?

GuillaumeTC1 · 2023-12-11T14:12:26Z

Hello, I face a related issue on ArgoCD 2.9 managed with Helm Charts. Argo goes out of sync because of required secrets config. If I sync I loose access right away and need to re add manually secrets in Azure.

Am I the only one to encounter this issue ? Do you think it would be possible to fix this without having to ignore argocd-secret sync in argo-cm ?

jdubs · 2024-01-04T20:20:53Z

I'm also having this same issue with self managed argo. This virtually makes the api/ui not usable, it makes me think that it should fail the health check so that it can be restarted automatically with the new secret.

fstr · 2024-01-26T15:18:43Z

I ran into the same issue as #1936 (comment) It's also simple to reproduce together with the ArgoCD Helm chart and ArgoCD managing itself.

Add a github.webhook.secret:

argo-cd:
  configs:
    secret:
       githubSecret: foobar

Sync it using ArgoCD. Successfully setup. Now your secret will look like this or similar:

apiVersion: v1
data:
  accounts.argoreadonly.tokens: +++++
  admin.password: +++++
  admin.passwordMtime: +++++
  server.secretkey: +++++
  webhook.github.secret: +++++

Now remove the github.webhook.secret by removing the configs: block from your value file again. The argocd-secret will now have an empty data: block when you run helm template because you removed the only value managed by the chart. If you now go into ArgoCD and sync, it will remove all the properties within data property of the argocd-secret. Including all those auto-generated values from the server. After that, ArgoCD stops working. You need to follow the steps from #1936 (comment) to resurrect it.

I agree with @pgpx that the webhook secrets should be moved into their own Kubernetes Secret. This will not only solve the issue that many people in here are facing, but also make workarounds like the one currently built and proposed by @MrFreezeex in #16262 obsolete.

MrFreezeex · 2024-01-26T21:03:34Z

I agree with @pgpx that the webhook secrets should be moved into their own Kubernetes Secret. This will not only solve the issue that many people in here are facing, but also make workarounds like the one currently built and proposed by @MrFreezeex in #16262 obsolete.

Note that my PR is not fully related to your current issue my use case was more about using https://external-secrets.io/latest/ to supply webhook secrets because that's what we use in our env but that's pretty nice if it solves other similar-ish issues as well! I also believe that having what you are proposing (a separate secret) and my PR are fully compatible and should allow even better flexibility combined :D.

fstr · 2024-01-29T09:53:25Z

Note that my PR is not fully related to your current issue my use case was more about using https://external-secrets.io/latest/ to supply webhook secrets because that's what we use in our env but that's pretty nice if it solves other similar-ish issues as well! I also believe that having what you are proposing (a separate secret) and my PR are fully compatible and should allow even better flexibility combined :D.

How I got into this topic is also by the need to provide the secret via external-secrets. If the webhook secret would be stored in a separate secret, then we could simply provision that via external-secrets and there's no more need to reference it via the special $<secret-name>.<property> syntax. Your PR will bridge the gap until then though.

RobinsonZ · 2024-05-28T21:21:36Z

This is also causing an issue for us when argocd-secret is itself managed by argocd, and (as mentioned above) it's compounded by the fact that the argocd-server healthcheck will still succeed even when this error is occurring and making the interface unusable.

gerardnico · 2024-09-18T15:35:29Z

I got this problem after using external-secrets to provide a github webhook token

In case, you are in the same situation, the workaround is to:

use the creationPolicy: 'Merge'. Example:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: argocd-secret
spec:
  secretStoreRef:
    name: vault-external-secret-store
    kind: ClusterSecretStore
  target:
    name: argocd-secret # Secret name in Kubernetes
    creationPolicy: 'Merge'
    template:
      data:
        webhook.github.secret: "{{ .github_webhook_token }}"
      metadata:
        annotations:
          description: "The WebHook Secret that permits to authenticate the GitHub Webhook"
  # Mapping to local secret from remote secret
  data:
    # https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/#2-configure-argo-cd-with-the-webhook-secret-optional
    - secretKey: github_webhook_token # Prop Name in the secret
      remoteRef:
        key: argocd # Name of the remote secret
        property: github-webhook-token # Prop Name in the remote secret

Rerun the install to get a clean argocd-secret
Restart the argo-server deployment to get the missing secrets

dtaniwaki added the bug Something isn't working label Jul 14, 2019

lcostea mentioned this issue Aug 20, 2019

add admin password and tls values to argocd admin-secret.yaml argoproj/argo-helm#97

Closed

dtaniwaki closed this as completed Aug 29, 2019

acelinkio mentioned this issue May 25, 2022

Webhook server crashes when Argo CD option server.insecure is set. argoproj/applicationset#581

Open

crenshaw-dev reopened this Aug 9, 2023

jkleinlercher mentioned this issue Apr 8, 2024

[argocd] server.secretkey is missing suxess-it/kubriX#48

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Secret key can not be omitted #1936

Secret key can not be omitted #1936

dtaniwaki commented Jul 14, 2019

jessesuen commented Jul 20, 2019

dtaniwaki commented Aug 29, 2019

dtaniwaki commented Aug 29, 2019

dtaniwaki commented Aug 29, 2019 •

edited

Loading

emidevi commented Jan 24, 2022

alkdese commented Jun 21, 2022

hyerim-gn commented Jul 10, 2022 •

edited

Loading

emirot commented Jul 20, 2022

crenshaw-dev commented Jul 25, 2022

emirot commented Jul 26, 2022

brianpooe commented Oct 3, 2022

emirot commented Oct 3, 2022

winston0410 commented Oct 12, 2022

yevhen-harmonizehr commented Oct 14, 2022

TeamDman commented Nov 1, 2022

gaeljw commented Nov 17, 2022

ali-de7 commented Aug 9, 2023 •

edited

Loading

crenshaw-dev commented Aug 9, 2023

pgpx commented Sep 13, 2023

GuillaumeTC1 commented Dec 11, 2023 •

edited

Loading

jdubs commented Jan 4, 2024

fstr commented Jan 26, 2024 •

edited

Loading

MrFreezeex commented Jan 26, 2024

fstr commented Jan 29, 2024

RobinsonZ commented May 28, 2024

gerardnico commented Sep 18, 2024

Secret key can not be omitted #1936

Secret key can not be omitted #1936

Comments

dtaniwaki commented Jul 14, 2019

jessesuen commented Jul 20, 2019

dtaniwaki commented Aug 29, 2019

dtaniwaki commented Aug 29, 2019

dtaniwaki commented Aug 29, 2019 • edited Loading

emidevi commented Jan 24, 2022

alkdese commented Jun 21, 2022

hyerim-gn commented Jul 10, 2022 • edited Loading

emirot commented Jul 20, 2022

crenshaw-dev commented Jul 25, 2022

emirot commented Jul 26, 2022

brianpooe commented Oct 3, 2022

emirot commented Oct 3, 2022

winston0410 commented Oct 12, 2022

yevhen-harmonizehr commented Oct 14, 2022

TeamDman commented Nov 1, 2022

Workaround

gaeljw commented Nov 17, 2022

ali-de7 commented Aug 9, 2023 • edited Loading

crenshaw-dev commented Aug 9, 2023

pgpx commented Sep 13, 2023

GuillaumeTC1 commented Dec 11, 2023 • edited Loading

jdubs commented Jan 4, 2024

fstr commented Jan 26, 2024 • edited Loading

MrFreezeex commented Jan 26, 2024

fstr commented Jan 29, 2024

RobinsonZ commented May 28, 2024

gerardnico commented Sep 18, 2024

dtaniwaki commented Aug 29, 2019 •

edited

Loading

hyerim-gn commented Jul 10, 2022 •

edited

Loading

ali-de7 commented Aug 9, 2023 •

edited

Loading

GuillaumeTC1 commented Dec 11, 2023 •

edited

Loading

fstr commented Jan 26, 2024 •

edited

Loading