Unable to enable GCP CAPIProvider with Rancher CloudCredentials #510

cpinjani · 2024-04-25T08:11:37Z

What steps did you take and what happened?

GCP CAPIProvider fails to enable using Rancher CloudCredentials. It enables when only ConfigSecret is provided along with GCP_B64ENCODED_CREDENTIALS variable.

Create GCP Cloud credentials In Rancher
Create CAPIProvider type GCP

apiVersion: turtles-capi.cattle.io/v1alpha1
kind: CAPIProvider
metadata:
  name: gcp
  namespace: capg-system
spec:
  configSecret:
    name: gcp-credentials-qmyxt
  credentials:
      rancherCloudCredentialNamespaceName: cattle-global-data:cc-4w8r5
  name: gcp
  variables:
    CLUSTER_TOPOLOGY: "true"
    EXP_CLUSTER_RESOURCE_SET: "true"
  type: infrastructure

Logs

E0425 07:36:38.279512       1 controller.go:329]  "msg"="Reconciler error" "error"="action failed after 10 attempts: failed to create provider object /v1, Kind=Secret, capg-system/capg-manager-bootstrap-credentials: Secret in version \"v1\" cannot be handled as a Secret: json: cannot unmarshal object into Go struct field Secret.data of type []uint8" "InfrastructureProvider"={"name":"gcp","namespace":"capg-system"} "controller"="infrastructureprovider" "controllerGroup"="operator.cluster.x-k8s.io" "controllerKind"="InfrastructureProvider" "name"="gcp" "namespace"="capg-system" "reconcileID"="bd6cd108-c507-4983-a48e-cd60f1947495"
I0425 07:36:38.279573       1 genericprovider_controller.go:62]  "msg"="Reconciling provider" "InfrastructureProvider"={"name":"gcp","namespace":"capg-system"} "controller"="infrastructureprovider" "controllerGroup"="operator.cluster.x-k8s.io" "controllerKind"="InfrastructureProvider" "name"="gcp" "namespace"="capg-system" "reconcileID"="5d107bdd-f960-4453-bb36-6dac5135c9d4"

What did you expect to happen?

Enabling GCP CAPIProvider with Rancher CloudCredentials must complete successfully

How to reproduce it?

No response

Rancher Turtles version

Turtles: v0.6.0

Anything else you would like to add?

It seems GCP_B64ENCODED_CREDENTIALS from Cloud credentials is not getting encoded:

$ kubectl get secret/gcp-credentials-qmyxt -n capg-system --template={{.data.GCP_B64ENCODED_CREDENTIALS}}  | base64 -d
{
  "type": "service_account",
  "project_id": "<REDACTED>",
  "private_key_id": "<REDACTED>",
  "private_key": "-----BEGIN PRIVATE KEY-----<REDACTED>-----END PRIVATE KEY-----",
  "client_email": "<REDACTED>",
  "client_id": "<REDACTED>",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "<REDACTED>"
}

Label(s) to be applied

/kind bug

The text was updated successfully, but these errors were encountered:

Danil-Grigorev · 2024-05-07T12:51:21Z

Hi @cpinjani, I tried reproducing this issue with latest version by doing:

Create GCP credentials from rancher UI.
Apply the template specified in the issue with the only correction - point to my rancher credentials secret.
I got the infrastructure provider in Ready state and I see GCP_B64 credentials double base64 encoded in the secret data, as it should.

Output:

dgrigorev@localhost:~> kubectl get secret/gcp-credentials-qmyxt -n capg-system --template={{.data.GCP_B64ENCODED_CREDENTIALS}}  | base64 -d | base64 -d
{
  "type": "service_account",
  "project_id": "***",
  "private_key_id": "***",
  "private_key": "-----BEGIN PRIVATE KEY-----\n***-----END PRIVATE KEY-----\n",
  "client_email": "***",
  "client_id": "***",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "***",
  "universe_domain": "googleapis.com"
}

My assumption was that Rancher credentials are not B64 encoded in the original secret, but it is not true, so I'm going to close the #529 as everything works as it should. Can it be that GCP rancher credentials were created manually and have incorrect initial format?

cpinjani · 2024-05-09T10:46:23Z

Hi @Danil-Grigorev , reproduced by steps below:

Rancher version: 2.8-head
Turtles: v0.7.0

Create GCP credentials using Rancher UI
Example data:

apiVersion: v1
data:
  googlecredentialConfig-authEncodedJson: <REDACTED_BASE64_DATA>
kind: Secret
metadata:
  annotations:
    field.cattle.io/creatorId: user-vfbhj
    field.cattle.io/name: gcp
    provisioning.cattle.io/driver: gcp
  creationTimestamp: "2024-05-08T11:22:01Z"
  generateName: cc-
  labels:
    cattle.io/creator: norman
  name: cc-8rsfr
  namespace: cattle-global-data
  resourceVersion: "27484"
  uid: ddff022d-fea2-423b-8ee1-6ea2229f302f
type: Opaque

Create Google Infra provider using CAPI UI or CAPIProvider template yaml
UI sent payload:

{
  "type": "turtles-capi.cattle.io.capiprovider",
  "metadata": {
    "namespace": "capg-system",
    "name": "gcp"
  },
  "spec": {
    "name": "gcp",
    "type": "infrastructure",
    "configSecret": {
      "name": "gcp-credentials-vw8iw"
    },
    "credentials": {
      "rancherCloudCredentialNamespaceName": "cattle-global-data:cc-4bf6q"
    },
    "features": {
      "clusterResourceSet": true,
      "clusterTopology": true,
      "machinePool": true
    },
    "variables": { 
    }
  }
}

Provider status:

Name:         gcp
Namespace:    capg-system
Labels:       <none>
Annotations:  <none>
API Version:  turtles-capi.cattle.io/v1alpha1
Kind:         CAPIProvider
Metadata:
  Creation Timestamp:  2024-05-08T11:24:30Z
  Generation:          1
  Resource Version:    43744
  UID:                 80ff1a34-2527-42e7-a054-685586f91737
Spec:
  Config Secret:
    Name:  gcp-credentials-qp00n
  Credentials:
    Rancher Cloud Credential Namespace Name:  cattle-global-data:cc-8rsfr
  Features:
    Cluster Resource Set:  true
    Cluster Topology:      true
    Machine Pool:          true
  Name:                    gcp
  Type:                    infrastructure
  Variables:
Status:
  Conditions:
    Last Transition Time:  2024-05-08T11:41:30Z
    Status:                True
    Type:                  LastAppliedConfigurationTime
    Last Transition Time:  2024-05-08T11:24:30Z
    Status:                True
    Type:                  PreflightCheckPassed
    Last Transition Time:  2024-05-08T11:25:20Z
    Message:               action failed after 10 attempts: failed to create provider object /v1, Kind=Secret, capg-system/capg-manager-bootstrap-credentials: Secret in version "v1" cannot be handled as a Secret: json: cannot unmarshal object into Go struct field Secret.data of type []uint8
    Reason:                Install failed
    Severity:              Warning
    Status:                False
    Type:                  ProviderInstalled
    Last Transition Time:  2024-05-08T11:24:30Z
    Status:                True
    Type:                  RancherCredentialsSecretMapped
  Observed Generation:     1
  Phase:                   Provisioning
  Variables:
    CLUSTER_TOPOLOGY:          true
    EXP_CLUSTER_RESOURCE_SET:  true
    EXP_MACHINE_POOL:          true
Events:                        <none>

Rancher logs:

W0508 10:23:39.347986      39 reflector.go:535] pkg/mod/github.com/rancher/client-go@v1.28.6-rancher1/tools/cache/reflector.go:229: failed to list *summary.SummarizedObject: Internal error occurred: error resolving resource
E0508 10:23:39.348017      39 reflector.go:147] pkg/mod/github.com/rancher/client-go@v1.28.6-rancher1/tools/cache/reflector.go:229: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Internal error occurred: error resolving resource
W0508 10:23:48.285788      39 reflector.go:535] pkg/mod/github.com/rancher/client-go@v1.28.6-rancher1/tools/cache/reflector.go:229: failed to list *summary.SummarizedObject: Internal error occurred: error resolving resource
E0508 10:23:48.285828      39 reflector.go:147] pkg/mod/github.com/rancher/client-go@v1.28.6-rancher1/tools/cache/reflector.go:229: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Internal error occurred: error resolving resource
W0508 10:23:53.308747      39 reflector.go:535] pkg/mod/github.com/rancher/client-go@v1.28.6-rancher1/tools/cache/reflector.go:229: failed to list *summary.SummarizedObject: Internal error occurred: error resolving resource
E0508 10:23:53.308776      39 reflector.go:147] pkg/mod/github.com/rancher/client-go@v1.28.6-rancher1/tools/cache/reflector.go:229: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Internal error occurred: error resolving resource

furkatgofurov7 · 2024-05-13T13:15:27Z

This should need testing, @cpinjani ?

cpinjani · 2024-05-14T05:12:17Z

@furkatgofurov7 Yes!

cpinjani · 2024-05-16T07:27:52Z

Validated as fixed on chart with nightly build with commit ID - 354df4c
Enabling GCP CAPIProvider with Rancher CloudCredentials completes successfully.

apiVersion: turtles-capi.cattle.io/v1alpha1
kind: CAPIProvider
metadata:
  creationTimestamp: '2024-05-16T07:21:16Z'
  generation: 1
  managedFields:
    - apiVersion: turtles-capi.cattle.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:finalizers:
            v:"foregroundDeletion": {}
        f:status:
          f:conditions: {}
          f:contract: {}
          f:installedVersion: {}
          f:observedGeneration: {}
          f:phase: {}
          f:variables:
            f:CLUSTER_TOPOLOGY: {}
            f:EXP_CLUSTER_RESOURCE_SET: {}
            f:EXP_MACHINE_POOL: {}
      manager: capi-provider-operator
      operation: Apply
      subresource: status
      time: '2024-05-16T07:23:45Z'
    - apiVersion: turtles-capi.cattle.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          .: {}
          f:configSecret:
            .: {}
            f:name: {}
          f:credentials: {}
          f:features:
            .: {}
            f:clusterResourceSet: {}
            f:clusterTopology: {}
            f:machinePool: {}
          f:name: {}
          f:type: {}
          f:variables: {}
      manager: rancher
      operation: Update
      time: '2024-05-16T07:21:16Z'
  name: google
  namespace: capg-system
  resourceVersion: '55804'
  uid: e53d1fdb-d432-4212-8fb1-2b2019ef45a6
spec:
  configSecret:
    name: gcp-credentials-vlim8
  credentials:
    rancherCloudCredentialNamespaceName: cattle-global-data:cc-vfrtk
  features:
    clusterResourceSet: true
    clusterTopology: true
    machinePool: true
  name: gcp
  type: infrastructure
  variables: {}
status:
  conditions:
    - lastTransitionTime: '2024-05-16T07:21:44Z'
      reason: MinimumReplicasAvailable
      status: 'True'
      type: Ready
    - lastTransitionTime: '2024-05-16T07:23:16Z'
      status: 'True'
      type: LastAppliedConfigurationTime
    - lastTransitionTime: '2024-05-16T07:21:16Z'
      status: 'True'
      type: PreflightCheckPassed
    - lastTransitionTime: '2024-05-16T07:21:23Z'
      status: 'True'
      type: ProviderInstalled
    - lastTransitionTime: '2024-05-16T07:21:16Z'
      status: 'True'
      type: RancherCredentialsSecretMapped
  contract: v1beta1
  installedVersion: v1.6.0
  observedGeneration: 2
  phase: Ready
  variables:
    CLUSTER_TOPOLOGY: 'true'
    EXP_CLUSTER_RESOURCE_SET: 'true'
    EXP_MACHINE_POOL: 'true'

cpinjani added the kind/bug Something isn't working label Apr 25, 2024

richardcase added this to the May release milestone May 3, 2024

Danil-Grigorev self-assigned this May 7, 2024

Danil-Grigorev mentioned this issue May 7, 2024

🐛 B64 encode gcp credentials for credentials sync #529

Merged

4 tasks

furkatgofurov7 closed this as completed in #529 May 13, 2024

furkatgofurov7 reopened this May 13, 2024

cpinjani self-assigned this May 14, 2024

cpinjani closed this as completed May 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to enable GCP CAPIProvider with Rancher CloudCredentials #510

Unable to enable GCP CAPIProvider with Rancher CloudCredentials #510

cpinjani commented Apr 25, 2024

Danil-Grigorev commented May 7, 2024

cpinjani commented May 9, 2024

furkatgofurov7 commented May 13, 2024

cpinjani commented May 14, 2024

cpinjani commented May 16, 2024

Unable to enable GCP CAPIProvider with Rancher CloudCredentials #510

Unable to enable GCP CAPIProvider with Rancher CloudCredentials #510

Comments

cpinjani commented Apr 25, 2024

What steps did you take and what happened?

What did you expect to happen?

How to reproduce it?

Rancher Turtles version

Anything else you would like to add?

Label(s) to be applied

Danil-Grigorev commented May 7, 2024

cpinjani commented May 9, 2024

furkatgofurov7 commented May 13, 2024

cpinjani commented May 14, 2024

cpinjani commented May 16, 2024