ApplicationSet controller receives x509: certificate signed by unknown authority, when using SCM, PR or Git generator with private bitbucket server #9607

LanceR1303 · 2022-06-08T09:33:00Z

Slack conversation thread : https://cloud-native.slack.com/archives/C01U45M2SVB/p1654569552211179
Checklist:

I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
I've included steps to reproduce the bug.
I've pasted the output of argocd version.

Describe the bug
Receive x509: certificate signed by unknown authority when application set controller with an SCM generator attempts to connect to a private bitbucket server.

To Reproduce

Follow installation for release v2.4.0-rc4 under https://github.com/argoproj/argo-cd/releases
Follow steps to setup TLS certificates as documented https://argo-cd.readthedocs.io/en/release-1.8/user-guide/private-repositories/#self-signed-untrusted-tls-certificates
Create an ApplicationSet CRD with an SCM generator for the Bitbucket Server SCM. The bitbucket server needs to be a private repository with TLS connectivity via self signed certificates which should have been registered in the above step

Expected behavior

Expected the application set controller to be able to communicate with bitbucket server API's to list repositories and automate ArgoCD application generation.

Screenshots

Version

{
    "Version": "v2.4.0-rc2+cd5b2af",
    "BuildDate": "2022-05-18T12:42:45Z",
    "GitCommit": "cd5b2af358f3574ea151fr8c24b1d52dafe59c85",
    "GitTreeState": "clean",
    "GoVersion": "go1.18.2",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v4.4.1 2021-11-11T23:36:27Z",
    "HelmVersion": "v3.8.1+g5cb9af4",
    "KubectlVersion": "v0.23.1",
    "JsonnetVersion": "v0.18.0"
}

Logs

applicationset-controller logs

time="2022-06-08T00:45:28Z" level=info msg="Starting configmap/secret informers"
time="2022-06-08T00:45:28Z" level=info msg="Configmap/secret informer synced"
time="2022-06-08T00:45:28Z" level=info msg="Starting webhook server"
time="2022-06-08T00:45:28Z" level=info msg="Starting manager"
time="2022-06-08T00:45:29Z" level=error msg="error generating params" error="error listing repos: error listing repositories for: Get \"https://<myBitBucketServerHost>/rest/api/1.0/projects/<myProject>/repos?limit=100\": x509: certificate signed by unknown authority" generator="&{0xc000c6c1e0 <nil>}"

Volume and Volume Mounts of the applicationset-controller

 Mounts:
      /app/config/ssh from ssh-known-hosts (rw)
      /app/config/tls from tls-certs (rw)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from argocd-applicationset-controller-token-59cjk (ro)


Volumes:
  tls-certs:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      argocd-tls-certs-cm
    Optional:  false
  ssh-known-hosts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      argocd-ssh-known-hosts-cm
    Optional:  false

The text was updated successfully, but these errors were encountered:

LanceR1303 · 2022-06-08T09:49:03Z

Observations:

We attempted to deploy our application with an App of Apps deployment pattern and we observed no issues. We are using the same argocd-tls-certs-cm configmap for the App of Apps deployment strategy and we observe that the argocd-application-controller and argocd-repo-server are able to communicate with our internal bitbucket server correctly.
While looking through the code we can see that
a) In argocd_repo_server.go , the system makes use of tlsConfigCustomizer class to instantiate the TLS certificates. The implementation of which is in tls.go
b) In argocd_application_controller.go, the system uses the util function tls.LoadX509CertPool to setup the TLS trust store
c) In argocd_applicationset_controller.go, we couldn't locate code for loading the certificates and suspect that this could be the problem?
d) The bitbucket SCM generator bitbucket_server.go uses https://github.com/gfleury/go-bitbucket-v1 . On running through the code in this library we did not find any specific instantiation of TLS certificates as well.

LanceR1303 · 2022-06-08T10:00:49Z

ApplicationSet CRD with status events

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  name: appset
  namespace: argocd
spec:
  generators:
    - scmProvider:
        bitbucketServer:
          allBranches: false
          api: 'MyBitBucketHost'
          basicAuth:
            passwordRef:
              key: MyPass
              secretName: argocd-generator-token
            username: MyName
          project: Apps
  template:
    metadata:
      name: 'My Application'
    spec:
      destination:
        namespace: apps
        server: 'https://kubernetes.default.svc'
      project: apps
      source:
        helm:
          parameters:
            - name: namespace
              value: default 
          releaseName: releaseName
        path: ''{{ path }}'
        repoURL: '{{ url }}'
        targetRevision: '{{ branch }}'
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
status:
  conditions:
    - lastTransitionTime: '2022-06-01T10:28:16Z'
      message: >-
        error listing repos: error listing repositories for : Get
        "https://<myBitBucketServerHost>/rest/api/1.0/projects/<myProject>/repos?limit=100":
        x509: certificate signed by unknown authority
      reason: ApplicationGenerationFromParamsError
      status: 'True'
      type: ErrorOccurred
    - lastTransitionTime: '2022-06-01T10:28:16Z'
      message: >-
        error listing repos: error listing repositories for : Get
        "https://<myBitBucketServerHost>/rest/api/1.0/projects/<myProject>/repos?limit=100":
        x509: certificate signed by unknown authority
      reason: ErrorOccurred
      status: 'False'
      type: ParametersGenerated
    - lastTransitionTime: '2022-06-01T10:28:16Z'
      message: >-
        error listing repos: error listing repositories for: Get
        "https://<myBitBucketServerHost>/rest/api/1.0/projects/<myProject>/repos?limit=100":
        x509: certificate signed by unknown authority
      reason: ApplicationGenerationFromParamsError
      status: 'False'
      type: ResourcesUpToDate

rishabh625 · 2022-06-08T10:52:11Z

Yeah it's a bug

flaviohrocha · 2022-06-14T15:44:22Z

+1 same problem using GitLab on-premisses

D-BlindSide · 2022-06-14T15:51:02Z

+1 same problem...

PiePra · 2022-09-01T15:01:07Z

+1 same

zetti12345 · 2022-11-30T06:26:43Z

Is there a workaround possible?

crenshaw-dev · 2022-11-30T23:09:51Z

I'm not aware of any workaround. We need to provide a way to set TLS configurations for various endpoints.

For certain generators (PR generator?) it might make sense to pull the TLS config from any already-configured repo creds in the argocd namespace. For others (SCM generator), we'll probably need a separate ConfigMap (maybe reuse argocd-cm?) to store the TLS config.

As always, PRs welcome. If a core contributor manages to pick this up, they'll drop a note here. 🙂

tabnul · 2022-12-21T06:49:24Z

a blocker for us to use applicationsets. is there a way to allow unsafe certificates instead of trusting them?

hnikt-jonasfh · 2023-01-04T14:55:52Z

+1, this is a showstopper for us. We're experiencing this issue when using a Helm chart source with a cert signed by our custom CA, despite having explicitly mounted the cert chain in /etc/ssl/certs on the filesystem of the pods of the controller, server, repo server and applicationset controller. Argo CD 2.5.3+0c7de21 installed with helm chart v5.9.1

guydolinksy · 2023-01-05T10:10:38Z

Ran into this issue today, this is the workaround I found (keep in mind we're using an on premise setup of argocd and gitlab and thus don't need the usual certificates):

In the values.yaml of the argocd deployment add:

applicationSet:
extraVolumeMounts:
- mountPath: /etc/ssl/certs
name: tls-certs

This will mount the tls certificates from argocd's config-map into the default ssl certificate folder of golang's http client, getting around the issue. It does however override the other certificates so this isn't ideal, maybe someone can further improve on it in the future, at least until an option is made available to append these tls certificates.

Best of luck!
Guy

hnikt-jonasfh · 2023-01-05T12:55:42Z

Ran into this issue today, this is the workaround I found (keep in mind we're using an on premise setup of argocd and gitlab and thus don't need the usual certificates):

In the values.yaml of the argocd deployment add:

applicationSet: extraVolumeMounts: - mountPath: /etc/ssl/certs name: tls-certs

This will mount the tls certificates from argocd's config-map into the default ssl certificate folder of golang's http client, getting around the issue. It does however override the other certificates so this isn't ideal, maybe someone can further improve on it in the future, at least until an option is made available to append these tls certificates.

Best of luck! Guy

Hi,
Thank you! That solved the issue for us :) We have the certs saved in a secret (in the same namespace as argo) named "certificates", so for us the full spec is something along these lines:

applicationSet:
extraVolumeMounts:
- name: certificates
mountPath: /etc/ssl/certs/cert1.crt
subPath: cert1.crt
- name: certificates
mountPath: /etc/ssl/certs/cert2.crt
subPath: cert2.crt
extraVolumes:
- name: certificates
secret:
secretName: certificates
defaultMode: 420

This way we also get to keep whatever default certificates argo throws in /etc/ssl/certs! Hope this helps others who are having the same issues.

I noticed there are quite a few issues registered here with similar or directly related problems:

I still find it odd that certificates added to argo proper (e.g. via the UI or declaratively in the configmap for trusted tls certificates) do not register as valid when it comes to Argo CD using the Helm client. That seems to be a common thread with all these issues.

rumstead · 2023-01-18T17:04:00Z

We have a self-signed CA that needs to apply for all URLs so we BYOI.

Isn't as clean as a volume mount but also works.

FROM quay.io/argoproj/argocd:$ARGOCD_VERSION
# Become the root user for installing the certs
USER root
# We have a base image that grabs certs for us, you can get them however
COPY --from=cacerts /tmp/corpcert.cer /usr/local/share/ca-certificates/internal-ca-bundle.crt
RUN update-ca-certificates
# Fallback to the argo user
USER $ARGOCD_USER_ID

gaeljw · 2023-04-07T14:26:10Z

I'm not entirely sure but I believe I'm having the same issue when using Helm through Kustomize. But not when using Helm directly.

#13154

sarsatis · 2023-05-30T21:38:40Z

I'm not aware of any workaround. We need to provide a way to set TLS configurations for various endpoints.

For certain generators (PR generator?) it might make sense to pull the TLS config from any already-configured repo creds in the argocd namespace. For others (SCM generator), we'll probably need a separate ConfigMap (maybe reuse argocd-cm?) to store the TLS config.

As always, PRs welcome. If a core contributor manages to pick this up, they'll drop a note here. 🙂

Ran into this issue today, this is the workaround I found (keep in mind we're using an on premise setup of argocd and gitlab and thus don't need the usual certificates):
In the values.yaml of the argocd deployment add:
applicationSet: extraVolumeMounts: - mountPath: /etc/ssl/certs name: tls-certs
This will mount the tls certificates from argocd's config-map into the default ssl certificate folder of golang's http client, getting around the issue. It does however override the other certificates so this isn't ideal, maybe someone can further improve on it in the future, at least until an option is made available to append these tls certificates.
Best of luck! Guy

Hi, Thank you! That solved the issue for us :) We have the certs saved in a secret (in the same namespace as argo) named "certificates", so for us the full spec is something along these lines:

applicationSet:
extraVolumeMounts:

name: certificates
mountPath: /etc/ssl/certs/cert1.crt
subPath: cert1.crt

name: certificates
mountPath: /etc/ssl/certs/cert2.crt
subPath: cert2.crt
extraVolumes:

name: certificates
secret:
secretName: certificates
defaultMode: 420

This way we also get to keep whatever default certificates argo throws in /etc/ssl/certs! Hope this helps others who are having the same issues.

I noticed there are quite a few issues registered here with similar or directly related problems:

Adding Helm https repo needs a ca-file optional input. #3539

Helm repositories with self signed certs doesn't work when used as umbrella charts #3693

x509: certificate signed by unknown authority #6101

Failed to create application due to error "x509: certificate signed by unknown authority" #6376

helm chart dependencies fail to download from private repo #6477

I still find it odd that certificates added to argo proper (e.g. via the UI or declaratively in the configmap for trusted tls certificates) do not register as valid when it comes to Argo CD using the Helm client. That seems to be a common thread with all these issues.

Hi @guydolinksy @crenshaw-dev @D-BlindSide @zetti12345 @flaviohrocha

We are also facing the same issue. We are using Openshift operator to install ArogoCd i tried to update applicationSet by creating the secret as mentioned above but after i add volume section operator reloads and removes the volumes which i added. Please help if there is any other workaround or how to solve this issue

ThomasVitt · 2023-08-08T11:39:29Z

We have the same issue under Openshift 4.13 running the Openshift GitOps Operator (ArgoCD v2.6.7).
I connected repos from both BitBucket and GitLab on-prem servers with HTTPS and successfully created applicationsets with the "git"-generator. When doing the same with the "scmProvider" generator, I get the message:

  conditions:
  - lastTransitionTime: "2023-08-08T11:31:09Z"
    message: 'error listing repos: error listing repositories for REPO: Get "https://<url>/rest/api/1.0/projects/REPO/repos?limit=100":
      x509: certificate signed by unknown authority'
    reason: ApplicationGenerationFromParamsError
    status: "True"
    type: ErrorOccurred
  - lastTransitionTime: "2023-08-08T11:31:09Z"
    message: 'error listing repos: error listing repositories for REPO: Get "https://<url>/rest/api/1.0/projects/REPO/repos?limit=100":
      x509: certificate signed by unknown authority'
    reason: ErrorOccurred
    status: "False"
    type: ParametersGenerated
  - lastTransitionTime: "2023-08-08T11:31:09Z"
    message: 'error listing repos: error listing repositories for REPO: Get "https://<url>/rest/api/1.0/projects/REPO/repos?limit=100":
      x509: certificate signed by unknown authority'
    reason: ApplicationGenerationFromParamsError
    status: "False"
    type: ResourcesUpToDate

The repositories inside ArgoCD are not "insecure", so the certificates are implemented correct. But the scmProvider generator obviously has problems with them.

ThomasVitt · 2023-08-08T11:53:12Z

I just noticed that a pull request has been merged into release-2.8:
#14462
#14348
Do they solve the problem?

crenshaw-dev · 2023-08-08T15:06:01Z

They might solve it for GitLab. We'd need equivalent PRs for other SCM providers.

I do think it's a miss that ApplicationSet doesn't know how to use the existing configured certificates. There's a similar problem with repositories, because there's incongruence between accessing an SCM's git interface and accessing its broader API.

PiePra · 2023-08-23T09:17:58Z

My simple workaround for the Pull Request Generator was to add the Cert at settings / Repository certificates and known hosts.
Those certificates will be present in the configmap argocd-tls-certs-cm in namespace argocd. I then added an additional volumeMount to Applicationset Deployment at /etc/ssl/certs like

        volumeMounts:
        ...
        - mountPath: /etc/ssl/certs
          name: tls-certs

jdoylei · 2023-09-05T20:06:42Z

I had the same experience as @ThomasVitt with ApplicationSet generators - the git generator connects successfully, but the scmProvider generator (with bitbucketServer) fails with "x509: certificate signed by unknown authority". Both generators are configured with the same protocol (https:) and the same Bitbucket Server hostname. The Bitbucket Server hostname matches an Argo CD "repository certificate" stored in argocd-tls-certs-cm, which must be sufficient for the git generator.

appiepollo14 · 2023-11-13T14:43:28Z

@ThomasVitt still running into this error on 2.8 with BitBucket Server. No workaround available for us so this is an impediment.

Anyone working on this? Any planning for this one?

LanceR1303 added the bug Something isn't working label Jun 8, 2022

rishabh625 added the component:applications-set Bulk application management related label Jun 8, 2022

gaeljw mentioned this issue Apr 7, 2023

Helm through Kustomize with a private Helm repository - x509 certificate signed by unknown authority #13154

Closed

3 tasks

appiepollo14 mentioned this issue Nov 15, 2023

(Extra)VolumeMount for appset controller redhat-developer/gitops-operator#620

Open

arnouthoebreckx mentioned this issue Jun 4, 2024

feat(appset): Add support for self-signed certificates / TLS on Azure DevOps Server #9607 #18501

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ApplicationSet controller receives x509: certificate signed by unknown authority, when using SCM, PR or Git generator with private bitbucket server #9607

ApplicationSet controller receives x509: certificate signed by unknown authority, when using SCM, PR or Git generator with private bitbucket server #9607

LanceR1303 commented Jun 8, 2022 •

edited

LanceR1303 commented Jun 8, 2022

LanceR1303 commented Jun 8, 2022

rishabh625 commented Jun 8, 2022

flaviohrocha commented Jun 14, 2022

D-BlindSide commented Jun 14, 2022

PiePra commented Sep 1, 2022

zetti12345 commented Nov 30, 2022

crenshaw-dev commented Nov 30, 2022

tabnul commented Dec 21, 2022

hnikt-jonasfh commented Jan 4, 2023

guydolinksy commented Jan 5, 2023

hnikt-jonasfh commented Jan 5, 2023

rumstead commented Jan 18, 2023

gaeljw commented Apr 7, 2023

sarsatis commented May 30, 2023

ThomasVitt commented Aug 8, 2023

ThomasVitt commented Aug 8, 2023

crenshaw-dev commented Aug 8, 2023

PiePra commented Aug 23, 2023

jdoylei commented Sep 5, 2023

appiepollo14 commented Nov 13, 2023 •

edited

ApplicationSet controller receives x509: certificate signed by unknown authority, when using SCM, PR or Git generator with private bitbucket server #9607

ApplicationSet controller receives x509: certificate signed by unknown authority, when using SCM, PR or Git generator with private bitbucket server #9607

Comments

LanceR1303 commented Jun 8, 2022 • edited

LanceR1303 commented Jun 8, 2022

LanceR1303 commented Jun 8, 2022

rishabh625 commented Jun 8, 2022

flaviohrocha commented Jun 14, 2022

D-BlindSide commented Jun 14, 2022

PiePra commented Sep 1, 2022

zetti12345 commented Nov 30, 2022

crenshaw-dev commented Nov 30, 2022

tabnul commented Dec 21, 2022

hnikt-jonasfh commented Jan 4, 2023

guydolinksy commented Jan 5, 2023

hnikt-jonasfh commented Jan 5, 2023

rumstead commented Jan 18, 2023

gaeljw commented Apr 7, 2023

sarsatis commented May 30, 2023

ThomasVitt commented Aug 8, 2023

ThomasVitt commented Aug 8, 2023

crenshaw-dev commented Aug 8, 2023

PiePra commented Aug 23, 2023

jdoylei commented Sep 5, 2023

appiepollo14 commented Nov 13, 2023 • edited

LanceR1303 commented Jun 8, 2022 •

edited

appiepollo14 commented Nov 13, 2023 •

edited