(Corporate network) x509: certificate signed by unknown authority / x509: certificate is valid for github.com, www.github.com, not kubernetes.github.io

[brovoca]

We're having issues using public Helm repositories on our company network which does sort of man-in-the-middle SSL-stripping. One of them is:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: ingress-nginx
spec:
  interval: 30s
  url: https://kubernetes.github.io/ingress-nginx

Below are various changes tested. Where .spec.secretRef.name is specified, it simply refers to a Secret with .stringData.caFile set to our root CA.

• With neither .name.passCredentials or .spec.secretRef.name (effectively the YAML above).

  • helmrepository/ingress-nginx works fine.

  • helmchart/ingress-nginx-ingress-nginx gives:

    chart pull error: failed to download chart for remote reference: Get "https://github.com/kubernetes/ingress-nginx/releases/download/helm-chart-4.0.10/ingress-nginx-4.0.10.tgz": x509: certificate is valid for github.com, www.github.com, not kubernetes.github.io

    ... this is most likely related to the fact that https://kubernetes.github.io/ingress-nginx/index.yaml lives on kubernetes.github.io, but points to a .tgzs on github.com.

• With .name.passCredentials, but no .spec.secretRef.name.

  • helmrepository/ingress-nginx works fine.

  • helmchart/ingress-nginx-ingress-nginx gives:

    chart pull error: failed to download chart for remote reference: Get "https://github-releases.githubusercontent.com/72891330/356bf635-bdaa-42d4-8235-97556c1a1771?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211203%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211203T081453Z&X-Amz-Expires=300&X-Amz-Signature=3d8a75409f5bf877a6d89cb99b9950c0e54205cb61a51d5e76036bf2b73063e9&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=72891330&response-content-disposition=attachment%3B%20filename%3Dingress-nginx-4.0.10.tgz&response-content-type=application%2Foctet-stream": x509: certificate signed by unknown authority

• With both .name.passCredentials and .spec.secretRef.name specified.

  • helmrepository/ingress-nginx gives:

    failed to download repository index: failed to cache index to '/tmp/chart-index-3560494879.yaml': Get "https://kubernetes.github.io/ingress-nginx/index.yaml": x509: certificate signed by unknown authority

  • helmchart/ingress-nginx-ingress-nginx gives:

    chart pull error: failed to download chart for remote reference: Get "https://github-releases.githubusercontent.com/72891330/356bf635-bdaa-42d4-8235-97556c1a1771?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211203%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211203T081453Z&X-Amz-Expires=300&X-Amz-Signature=3d8a75409f5bf877a6d89cb99b9950c0e54205cb61a51d5e76036bf2b73063e9&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=72891330&response-content-disposition=attachment%3B%20filename%3Dingress-nginx-4.0.10.tgz&response-content-type=application%2Foctet-stream": x509: certificate signed by unknown authority

[brovoca]

I've applied patches to the controllers as described by ChrisJBurns in #508, but it doesn't seem to help. However I seem to modify the various parameter, I get an x509 error somewhere. I suspect this is due to our MITM/DPI (Deep Packet Inspection) server being selective and doesn't perform DPI on all the domains, so just a few URLs require the Company-Root-CA while the rest are presented without modification.

What I've got is:

• flux-system
  • Company-Root-CA.yaml
  • gotk-components.yaml
  • gotk-patches
    • helm-controller-certs-patch.yaml
    • notification-controller-certs-patch.yaml
    • source-controller-certs-patch.yaml
  • gotk-sync.yaml
  • kustomization.yaml

All 3 files under gotk-patches are effectively the same, except for the .metadata.name.

---

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - Company-Root-CA.yaml
  - gotk-components.yaml
  - gotk-sync.yaml
patchesStrategicMerge:
  - gotk-patches/helm-controller-certs-patch.yaml
  - gotk-patches/notification-controller-certs-patch.yaml
  - gotk-patches/source-controller-certs-patch.yaml

Company-Root-CA.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: Company-Root-CA
  namespace: flux-system
data:
  Company-Root-CA.crt: |+
    -----BEGIN CERTIFICATE-----
    REDACTED
    -----END CERTIFICATE-----

gotk-patches/helm-controller-certs-patch.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helm-controller
  namespace: flux-system
spec:
  template:
    spec:
      containers:
        - name: manager
          volumeMounts:
            - name: cert-volume
              mountPath: /etc/ssl/certs/Company-Root-CA.crt
              subPath: Company-Root-CA.crt
              readOnly: false
      initContainers:
        - name: init-helm-controller-certs
          image: debian
          command:
            ["sh", "-c", 'echo "$TRUSTED_CERT" > /var/tmp/Company-Root-CA.crt']
          env:
            - name: TRUSTED_CERT
              valueFrom:
                configMapKeyRef:
                  name: Company-Root-CA
                  key: Company-Root-CA.crt
          volumeMounts:
            - name: cert-volume
              mountPath: /var/tmp
      volumes:
        - name: cert-volume
          emptyDir: {}

gotk-patches/notification-controller-certs-patch.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: notification-controller
  namespace: flux-system
spec:
  template:
    spec:
      containers:
        - name: manager
          volumeMounts:
            - name: cert-volume
              mountPath: /etc/ssl/certs/Company-Root-CA.crt
              subPath: Company-Root-CA.crt
              readOnly: false
      initContainers:
        - name: init-notification-controller-certs
          image: debian
          command:
            ["sh", "-c", 'echo "$TRUSTED_CERT" > /var/tmp/Company-Root-CA.crt']
          env:
            - name: TRUSTED_CERT
              valueFrom:
                configMapKeyRef:
                  name: Company-Root-CA
                  key: Company-Root-CA.crt
          volumeMounts:
            - name: cert-volume
              mountPath: /var/tmp
      volumes:
        - name: cert-volume
          emptyDir: {}

gotk-patches/source-controller-certs-patch.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: source-controller
  namespace: flux-system
spec:
  template:
    spec:
      containers:
        - name: manager
          volumeMounts:
            - name: cert-volume
              mountPath: /etc/ssl/certs/Company-Root-CA.crt
              subPath: Company-Root-CA.crt
              readOnly: false
      initContainers:
        - name: init-source-controller-certs
          image: debian
          command:
            ["sh", "-c", 'echo "$TRUSTED_CERT" > /var/tmp/Company-Root-CA.crt']
          env:
            - name: TRUSTED_CERT
              valueFrom:
                configMapKeyRef:
                  name: Company-Root-CA
                  key: Company-Root-CA.crt
          volumeMounts:
            - name: cert-volume
              mountPath: /var/tmp
      volumes:
        - name: cert-volume
          emptyDir: {}

[brovoca]

I've successfully managed to install this Helm chart using a Debian Pod as suggested by @hiddeco over Slack. I did not have to add any certificates. Commands used:

apt-get install curl vim --yes

curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
sudo apt-get install apt-transport-https --yes
echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm --yes

mkdir .kube
vim .kube/config
chmod 600 /root/.kube/config
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
vim values-override.yaml
helm install ingress-nginx-2 ingress-nginx/ingress-nginx --namespace ingress-nginx-2 --create-namespace -f values-override.yaml

$ flux version
flux: v0.24.0
helm-controller: v0.14.0
kustomize-controller: v0.18.1
notification-controller: v0.19.0
source-controller: v0.19.0

[hiddeco]

The problem ended up being the network doing weird things with the certificates, and the user not providing the full chain set for this. Once this got resolved, it started working as expected.

[brovoca]

@hiddeco correct. However, the native method of providing a CA cert through .spec.secretRef.name did not work. My guess is that it ignores all existing certs on the system and only looks for this particular one which doesn't work when different CA certs are presented. I had to resort to patching the source-controller. Thank you for the assistance provided Hiddie.