Minio S3 backup doesn't work due to certificate signed by unknown authority

[mceronja]

Describe the bug

Both backup and restore procedures do not work with a Minio S3 instance, let's call it minio.lan and the bucket name is jenkins-backup.

Result of backup cron job:

2024/01/02 14:15:01 Backup started!
2024/01/02 14:15:01 Getting clients
2024/01/02 14:15:05 RequestError: send request failed
caused by: Get https://minio.lan/jenkins-backup?max-keys=0: x509: certificate signed by unknown authority

Result of the restore job:

2024/01/03 11:42:50 RequestError: send request failed
caused by: Get https://minio.lan/jenkins-backup?max-keys=0: x509: certificate signed by unknown authority

Version of Helm and Kubernetes

- Helm: v3.9.0
- Kubernetes: v1.24.9

Chart version

jenkins-4.11.1

What happened?

1. Create a secret jenkins-aws-s3-credentials with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
2. Update values.yaml
3. Install the chart
4. Run the cron job

What you expected to happen?

For the backup job the files to be copied from Jenkins to the bucket and for the restore job from the bucket to Jenkins.
Also there is no way to alter the backup cron job via the Helm chart so we can maybe mount the certificate. AWS_S3_NO_SSL doesn't seem to do what is expected of it...

How to reproduce it

backup:
  enabled: true
  existingSecret:
    jenkins-aws-s3-credentials:
      awsaccesskey: AWS_ACCESS_KEY_ID
      awssecretkey: AWS_SECRET_ACCESS_KEY
  env:
    - name: "AWS_S3_ENDPOINT"
      value: "https://minio.lan"
    - name: "AWS_S3_NO_SSL"
      value: "true"
    - name: "AWS_S3_FORCE_PATH_STYLE"
      value: "true"
  destination: "s3://jenkins-backup"

Anything else we need to know?

Both backup and restore worked with an AWS S3 bucket. Basically all the config is the same expect the environment which has only AWS_REGION set in the AWS S3 bucket case.
Everything was done according to the official documentation: https://github.com/jenkinsci/helm-charts/tree/main/charts/jenkins#backup
Minio was configured according to: https://github.com/maorfr/skbn?tab=readme-ov-file#minio-s3-support

[kvanzuijlen]

This is likely due to the kube-tasks image being really old. Sadly, it no longer seems to be kept up to date. Alternatively, you could try using Velero instead (just a tool I have experience with personally).

@timja I don't see an easy way of updating this. It might be a better idea to drop the whole backup section altogether with the 5.0.0 upgrade I've drafted in #979.

[timja]

I'm fine with that, @torstenwalter any thoughts?

[mceronja]

@kvanzuijlen I could implement backup and restore procedure in a 100 ways, but I wanted to stay inside of the solution that the Helm chart provides. Eventually I implemented the backup and restore with an AWS S3 bucket.
What bothered me in this case are the actual images that are used for this.
Backup -> https://github.com/jenkinsci/helm-charts/tree/main/charts/jenkins#backup -> https://github.com/maorfr/kube-tasks
Restore -> https://github.com/jenkinsci/helm-charts/tree/main/charts/jenkins#restore-from-backup -> https://github.com/maorfr/skbn
Both images seem very old and out of date. Also I got error in Stream: command terminated with exit code 1 a couple of times during backup and I don't see anything I could do about it...

[mceronja]

Just to add on top of that, the issue I stumbled upon:
Backup -> maorfr/kube-tasks#6 -> Added in https://github.com/amerello/kube-tasks
Restore -> maorfr/skbn#8 -> Merged but not released maorfr/skbn#23

[kvanzuijlen]

@timja can be closed