[BUG] Unable to validate S3 backup target configuration - Editing S3 Custom CA Configuration

[CamZie]

Rancher Server Setup

• Rancher version: 2.8.2
• Installation option (Docker install/Helm Chart): Helm Chart
  • If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): RKE1

Information about the Cluster

• Kubernetes version: v1.27.11
• Cluster Type (Local/Downstream): local

User Information

• What is the role of the user logged in? Admin

Describe the bug
Our S3 certificate expired and when adding the new Custom CA Certificate we get always get this error:

Unable to validate S3 backup target configuration: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: no route to host

To Reproduce

• Login to rancher webui
• Go to Cluster Management
• Edit Config on the cluster with expired S3 certificate
• Add new certificate
• Save

Result

Unable to validate S3 backup target configuration: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: no route to host

Expected Result
Successfully save the settings and new certificate is active.

[vtrenton]

Where is Rancher installed? 169.254.169.254 is an internal amazon only IP address that cannot be used outside of AWS.

That error is not related to a certificate issue - rather the backup operator cannot reach the endpoint specified at that address.

[CamZie]

Where is Rancher installed? 169.254.169.254 is an internal amazon only IP address that cannot be used outside of AWS.

That error is not related to a certificate issue - rather the backup operator cannot reach the endpoint specified at that address.

Rancher is installed on our own server with Ubuntu OS. Do you know where it is taking this IP address from? Because we cannot find it anywhere on our configuration. This is what our s3 bucket configuration is:

s3_backup_config:
          access_key: <ACCESS_KEY>
          bucket_name: s3bucket
          custom_ca: |-
            -----BEGIN CERTIFICATE-----
            -----END CERTIFICATE-----
          endpoint: '<IPADDRESS>:9000'
          folder: s3bucket-backup
          region: <REGION>

[CamZie]

We found a workaround by editing the YAML configuration directly using "kubectl" commands instead of the rancher web and it works.