x509: certificate signed by unknown authority in oadp-aws-registry pod

[gorantornqvist]

Describe the bug

Hi,
I had a working install for 0.2.6 and then I upgraded to 0.3.0 according to docs but after I created the velero object, the oadp-velero-name-1-aws-registry-7bfdb7b77f-bwbqr pod throws a "x509: certificate signed by unknown authority" error and then crashes:

time="2021-09-29T13:20:11.926600357Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: RequestError: send request failed\ncaused by: Get https://netapp-storagegrid-s3-endpoint/nimbus-demo-customer-ec-u12-dr01?delimiter=%2F&list-type=2&max-keys=1000&prefix=docker%2Fregistry%2Fv2%2Frepositories%2F: x509: certificate signed by unknown authority" err.message="unknown error" go.version=go1.13.8 http.request.host="172.19.16.128:5000" http.request.id=7aa63461-d0be-49eb-a4f8-ab68b0b1c914 http.request.method=GET http.request.remoteaddr="172.19.16.2:39468" http.request.uri="/v2/_catalog?n=5" http.request.useragent=kube-probe/1.21 http.response.contenttype=application/json http.response.duration=351.444579ms http.response.status=500 http.response.written=104

Has there been any change in the format for the caCert bundle?

I currently have this order in the PEM CA bundle:

• IssuingCA
• RootCA

This is my Velero CR:

apiVersion: oadp.openshift.io/v1alpha1
kind: Velero
metadata:
  name: demo-customer
  namespace: openshift-adp
spec:
  olmManaged: true
  backupStorageLocations:
    - name: nimbus-demo-customer
      credential:
        name: cloud-credentials-demo-customer
        key: cloud
        namespace: openshift-adp
      provider: aws
      objectStorage:
        bucket: nimbus-demo-customer-ec-u12-dr01
        prefix: nimbus
        caCert: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk...........BVEUtLS0tLQo=
      config:
        region: us-east-1
        profile: "default"
        insecureSkipTlsVerify: "false"
        signatureVersion: "4"
        s3Url: "https://my-netapp-storagegrid-s3-endpoint"
        s3ForcePathStyle: "true"
  defaultVeleroPlugins:
    - aws
    - csi
    - openshift
  veleroFeatureFlags:
    - EnableCSI
  enableRestic: true

[gorantornqvist]

So I cant fint my cert bundle in any fields for the oadp-velero-name-1-aws-registry-7bfdb7b77f-bwbqr pod.
I found this though:

        env:
        - name: REGISTRY_STORAGE
          value: s3
        - name: REGISTRY_STORAGE_S3_ACCESSKEY
          value: **********
        - name: REGISTRY_STORAGE_S3_BUCKET
          value: nimbus-demo-customer-ec-u12-dr01
        - name: REGISTRY_STORAGE_S3_REGION
          value: us-east-1
        - name: REGISTRY_STORAGE_S3_SECRETKEY
          value: **********
        - name: REGISTRY_STORAGE_S3_REGIONENDPOINT
          value: 'https://my-netapp-storagegrid-s3-endpoint'
        - name: REGISTRY_STORAGE_S3_SKIPVERIFY

Could that be the issue?

EDIT: tried deleting the velero and recreate it with insecureSkipTlsVerify: "true" in config but REGISTRY_STORAGE_S3_SKIPVERIFY still doesnt have a value set.

[gorantornqvist]

Mixed up the options, tried with:

s_3__url: "https://..."
s_3__force_path_style: "true"

As suggested earlier and now most of the pods started ...

The oadp-demo-customer-1-aws-registry-77bc49d446-dfsl9 still crashes though with another error now:

time="2021-09-30T11:12:29.340772757Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.\n\tstatus code: 403, request id: 6RFX11ES0KAVQA76, host id: WCGjM4ik03YRVIrKzL0qm6nIewr7UjZ4HVJYjiuE/qkp1Lecn0zB+ypFZfswiqXkD+F3gj/lAPk=" err.message="unknown error" go.version=go1.13.8 http.request.host="172.19.17.102:5000" http.request.id=43a7f673-28f7-4bfc-afd2-a1362e03c13f http.request.method=GET http.request.remoteaddr="172.19.16.2:46814" http.request.uri="/v2/_catalog?n=5" http.request.useragent=kube-probe/1.21 http.response.contenttype=application/json http.response.duration=107.48711ms http.response.status=500 http.response.written=123

EDIT: Hmmm, deleted velero CR and recreated it with insecureSkipTLSVerify: "false" and now the oadp-demo-customer-1-aws-registry pod isnt created and all pods are running.

Anyway, will do some further testing now when everything is up

[gorantornqvist]

So after some further testing:

s_3__url / s_3__force_path_style wasnt the correct parameters:

time="2021-09-30T12:49:07Z" level=error msg="Error getting backup store for this location" backupLocation=demo-customer-1 controller=backup-sync error="rpc error: code = Unknown desc = config has invalid keys [s_3__force_path_style s_3__url]; valid keys are [region s3Url publicUrl kmsKeyId s3ForcePathStyle signatureVersion credentialsFile profile serverSideEncryption insecureSkipTLSVerify bucket prefix caCert]"

It was the removal of volumeSnapshotLocations from the velero the caused it to start working again, will check that later.

Anyway, all is up and running and I was able to create a backup, closing.

[gorantornqvist]

Reopening since this issue is still there, the issue actually never went away.
We are currently on oadp-operator 0.4.1.

time="2021-11-09T08:34:44.045229482Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: RequestError: send request failed\ncaused by: Get https://netapp-storagegrid-s3-endpoint.ourdomain.com/oadp-backups-nimbus-ec-u12-dr01?delimiter=%2F&list-type=2&max-keys=1000&prefix=docker%2Fregistry%2Fv2%2Frepositories%2F: x509: certificate signed by unknown authority" err.message="unknown error" go.version=go1.13.8 http.request.host="172.19.22.32:5000" http.request.id=c3d8c853-e93f-469c-902b-0c9cdc832011 http.request.method=GET http.request.remoteaddr="172.19.22.2:47126" http.request.uri="/v2/_catalog?n=5" http.request.useragent=kube-probe/1.21 http.response.contenttype=application/json http.response.duration=458.420558ms http.response.status=500 http.response.written=104

If I check the oadp-aws-registry pod I cannot see our certficate bundle in there...and REGISTRY_STORAGE_S3_SKIPVERIFY is set to false.

spec:
  containers:
  - env:
    - name: REGISTRY_STORAGE
      value: s3
    - name: REGISTRY_STORAGE_S3_ACCESSKEY
      value: **************9GWZ4
    - name: REGISTRY_STORAGE_S3_BUCKET
      value: oadp-backups-nimbus-ec-u12-dr01
    - name: REGISTRY_STORAGE_S3_REGION
      value: us-east-1
    - name: REGISTRY_STORAGE_S3_SECRETKEY
      value: **************VOLlALYbBxc
    - name: REGISTRY_STORAGE_S3_REGIONENDPOINT
      value: https://netapp-storagegrid-s3-endpoint.ourdomain.com
    - name: REGISTRY_STORAGE_S3_SKIPVERIFY
      value: "false"

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close