Add parameter for trusted CA in backup configuration in UI

[oskapt]

Longhorn will not accept a Minio endpoint wrapped in TLS from an unknown CA. After a long conversation in Slack, we all agreed that rather than accept any CA, it would be a fair compromise to specify a CA to trust for the backup endpoint. That CA can be an internal CA or a self-signed certificate that's sitting in front of Minio.

[yasker]

We also need to add document to https://github.com/longhorn/website for this new feature.

[yasker]

We use the custom CA by default in our testing for now. No further end-to-end automation needed for now. longhorn/longhorn-tests@025b195

[meldafrawi]

• Docs Validation: PARTIALLY FAILED
  In docs here:

  1. Minio secret example is missing AWS_CERT_KEY
  2. Current Minio deploy manifest is using http endpoint, we need to update the docs to have both http/https examples, or modify the existing manifest to be consistent with using https.

• Feature Validation: PASSED

Steps to test:

1. Deploy Longhorn
2. Create a selfsign certificate using openssl req -x509 -nodes -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365
3. Convert certificate and key files data to base64 format using
   cat cert.pem | base64 | tr -d "\n"
cat key.pem | base64 | tr -d "\n"
4. Create Minio secret using with AWS_CERT & AWS_KEY data

apiVersion: v1
kind: Secret
metadata:
  name: <MINIO_SECRET_NAME>
  namespace: <MINIO_NAMESPACE>
type: Opaque
data:
  AWS_ACCESS_KEY_ID: <MINIO_ACCESS_KEY_BASE64>
  AWS_SECRET_ACCESS_KEY: <MINIO_SECRET_KEY_BASE64>   
  AWS_ENDPOINTS: <MINIO_ENDPOINT_URI_BASE64>
  AWS_CERT: <MINIO_CERT_BASE64>
  AWS_CERT_KEY: <MINIO_CERT_KEY_BASE64>

5. Deploy minio backupstore and service

---
apiVersion: v1
kind: Pod
metadata:
  name: longhorn-test-minio
  namespace: default
  labels:
    app: longhorn-test-minio
spec:
  volumes:
  - name: minio-volume
    emptyDir: {}
  - name: minio-certificates
    secret:
      secretName: minio-secret
      items:
      - key: AWS_CERT
        path: public.crt
      - key: AWS_CERT_KEY
        path: private.key

  containers:
  - name: minio
    image: minio/minio
    command: ["sh", "-c", "mkdir -p /storage/backupbucket && mkdir -p /root/.minio/certs && ln -s /root/certs/private.key /root/.minio/certs/private.key && ln -s /root/certs/public.crt /root/.minio/certs/public.crt && exec /usr/bin/minio server /storage"]
    env:
    - name: MINIO_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: minio-secret
          key: AWS_ACCESS_KEY_ID
    - name: MINIO_SECRET_KEY
      valueFrom:
        secretKeyRef:
          name: minio-secret
          key: AWS_SECRET_ACCESS_KEY
    ports:
    - containerPort: 9000
    volumeMounts:
    - name: minio-volume
      mountPath: "/storage"
    - name: minio-certificates
      mountPath: "/root/certs"
      readOnly: true
---
apiVersion: v1
kind: Service
metadata:
  name: minio-service
  namespace: default
spec:
  selector:
    app: longhorn-test-minio
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  sessionAffinity: ClientIP

6. set backupstore target in Longhorn UI to use minio service
7. Verify backup and restore operations works as expected. PASSED

[joshimoo]

The minio AWS_CERT_KEY (private key) is only necessary for our test-minio, it's up to the user to deploy their own ssl secured minio backupstore.

We currently do not provide any guidance on how to do so, we only provide instructions for the client side (longhorn) setup (AWS_CERT)

[meldafrawi]

Validation PASSED

Additional tests:

• Test Minio backupstore with a valid Let's Encrypt certificate, backup list/create/restore/delete operations should work PASSED
• Test Minio backupstore with self-signed certificate with Corrent SANS -- backup list/create/restore/delete operations should work PASSED
• Test Minio backupstore with self-signed certificate with incorrect SANS -- should fail to connect to minio PASSED