DHCP Server create call failed but Server got created!!

[Karthik-K-N]

/kind bug
/area provider/ibmcloud

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

Tried creating a PowerVS cluster with create infra workflow, Noticed that the controllerCreated was set to false for network in IBMVSCluster object

Spec:
  Control Plane Endpoint:
    Host:  6688d18e-us-east.lb.appdomain.cloud
    Port:  6443
  Load Balancers:
    Name:    capi-karthik-19041-loadbalancer
    Public:  true
  Network:
  Resource Group:
    Name:  ibm-hypershift-dev
  Service Instance:
    Name:               capi-karthik-19041-serviceInstance
  Service Instance ID:
  Transit Gateway:
    Name:  capi-karthik-19041-transitgateway
  Vpc:
    Name:    capi-karthik-19041-vpc
    Region:  us-east
  Vpc Subnets:
    Name:  capi-karthik-19041-vpcsubnet
  Zone:    wdc06
Status:
  Conditions:
    Last Transition Time:  2024-04-23T12:03:57Z
    Status:                True
    Type:                  LoadBalancerReady
    Last Transition Time:  2024-04-23T11:52:56Z
    Status:                True
    Type:                  NetworkReady
    Last Transition Time:  2024-04-23T11:52:26Z
    Status:                True
    Type:                  ServiceInstanceReady
    Last Transition Time:  2024-04-23T11:57:30Z
    Status:                True
    Type:                  TransitGatewayReady
    Last Transition Time:  2024-04-23T11:53:00Z
    Status:                True
    Type:                  VPCReady
    Last Transition Time:  2024-04-23T11:53:08Z
    Status:                True
    Type:                  VPCSubnetReady
  Load Balancers:
    capi-karthik-19041-loadbalancer:
      Controller Created:  true
      Hostname:            6688d18e-us-east.lb.appdomain.cloud
      Id:                  r014-6688d18e-0f1b-4e90-8881-43d311940b39
      State:               active
  Network:
    Controller Created:  false
    Id:                  2ac7e26c-a2e6-4c8f-affa-0dc02ae9d6db
  Ready:                 true
  Resource Group ID:
    Controller Created:  false
    Id:                  08fe0ad0ec9b45aab2cb6d7a4d6817ba
  Service Instance:
    Controller Created:  true
    Id:                  ba762cca-5446-4790-a120-a4ddcd1b33ac
  Transit Gateway:
    Controller Created:  true
    Id:                  50dcc5f4-513b-4991-b1c1-12ae77dd5aa1
  Vpc:
    Controller Created:  true
    Id:                  r014-65d221c3-0abe-49ab-a78a-5223ba6159bf
  Vpc Subnet:
    capi-karthik-19041-vpcsubnet:
      Controller Created:  true
      Id:                  0757-541a3c50-033d-46b7-a91e-26f916717edf

Upon checking logs, found a unexpected behaviour,

1. On first attempt DHCP server creation returned error
2. Second attempt the controller was able to find the network, It means the first attempt was succesfull!!

I0423 11:52:26.052626       1 ibmpowervscluster_controller.go:157] "Reconciling network" controller="ibmpowervscluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="IBMPowerVSCluster" IBMPowerVSCluster="default/capi-karthik-19041" namespace="default" name="capi-karthik-19041" reconcileID="cb38564c-2113-4542-825a-94ddd3648de2" cluster="default/capi-karthik-19041"
I0423 11:52:27.458668       1 leaderelection.go:281] successfully renewed lease capi-ibmcloud-system/effcf9b8.cluster.x-k8s.io
I0423 11:52:27.472131       1 powervs_cluster.go:800] "Not able to find PowerVS network in IBM Cloud" controller="ibmpowervscluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="IBMPowerVSCluster" IBMPowerVSCluster="default/capi-karthik-19041" namespace="default" name="capi-karthik-19041" reconcileID="cb38564c-2113-4542-825a-94ddd3648de2" cluster="default/capi-karthik-19041" network={}
I0423 11:52:27.472179       1 powervs_cluster.go:779] "Creating DHCP server" controller="ibmpowervscluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="IBMPowerVSCluster" IBMPowerVSCluster="default/capi-karthik-19041" namespace="default" name="capi-karthik-19041" reconcileID="cb38564c-2113-4542-825a-94ddd3648de2" cluster="default/capi-karthik-19041"
I0423 11:52:29.476507       1 leaderelection.go:281] successfully renewed lease capi-ibmcloud-system/effcf9b8.cluster.x-k8s.io
E0423 11:52:46.046682       1 powervs_cluster.go:782] "Error creating DHCP server" err="failed to perform Create DHCP Operation for cloud instance ba762cca-5446-4790-a120-a4ddcd1b33ac with error [POST /pcloud/v1/cloud-instances/{cloud_instance_id}/services/dhcp][500] pcloudDhcpPostInternalServerError  &{Code:0 Description:create dhcp server could not retrieve image Error:internal server error Message:}" controller="ibmpowervscluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="IBMPowerVSCluster" IBMPowerVSCluster="default/capi-karthik-19041" namespace="default" name="capi-karthik-19041" reconcileID="cb38564c-2113-4542-825a-94ddd3648de2" cluster="default/capi-karthik-19041"
E0423 11:52:46.046905       1 ibmpowervscluster_controller.go:159] "failed to reconcile PowerVS network" err="failed to perform Create DHCP Operation for cloud instance ba762cca-5446-4790-a120-a4ddcd1b33ac with error [POST /pcloud/v1/cloud-instances/{cloud_instance_id}/services/dhcp][500] pcloudDhcpPostInternalServerError  &{Code:0 Description:create dhcp server could not retrieve image Error:internal server error Message:}" controller="ibmpowervscluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="IBMPowerVSCluster" IBMPowerVSCluster="default/capi-karthik-19041" namespace="default" name="capi-karthik-19041" reconcileID="cb38564c-2113-4542-825a-94ddd3648de2" cluster="default/capi-karthik-19041"
E0423 11:52:46.072249       1 controller.go:329] "Reconciler error" err="failed to perform Create DHCP Operation for cloud instance ba762cca-5446-4790-a120-a4ddcd1b33ac with error [POST /pcloud/v1/cloud-instances/{cloud_instance_id}/services/dhcp][500] pcloudDhcpPostInternalServerError  &{Code:0 Description:create dhcp server could not retrieve image Error:internal server error Message:}" controller="ibmpowervscluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="IBMPowerVSCluster" IBMPowerVSCluster="default/capi-karthik-19041" namespace="default" name="capi-karthik-19041" reconcileID="cb38564c-2113-4542-825a-94ddd3648de2"
I0423 11:52:47.683259       1 leaderelection.go:281] successfully renewed lease capi-ibmcloud-system/effcf9b8.cluster.x-k8s.io
I0423 11:52:48.208891       1 ibmpowervscluster_controller.go:135] "Reconciling resource group" controller="ibmpowervscluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="IBMPowerVSCluster" IBMPowerVSCluster="default/capi-karthik-19041" namespace="default" name="capi-karthik-19041" reconcileID="379721b4-2074-4e88-9d49-484d73477aa5" cluster="default/capi-karthik-19041"
I0423 11:52:48.209204       1 ibmpowervscluster_controller.go:143] "Reconciling PowerVS service instance" controller="ibmpowervscluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="IBMPowerVSCluster" IBMPowerVSCluster="default/capi-karthik-19041" namespace="default" name="capi-karthik-19041" reconcileID="379721b4-2074-4e88-9d49-484d73477aa5" cluster="default/capi-karthik-19041"
I0423 11:52:48.209234       1 powervs_cluster.go:639] "Service instance id is set" controller="ibmpowervscluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="IBMPowerVSCluster" IBMPowerVSCluster="default/capi-karthik-19041" namespace="default" name="capi-karthik-19041" reconcileID="379721b4-2074-4e88-9d49-484d73477aa5" cluster="default/capi-karthik-19041" id="ba762cca-5446-4790-a120-a4ddcd1b33ac"
I0423 11:52:49.478389       1 powervs_cluster.go:689] "PowerVS service instance is in active state" controller="ibmpowervscluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="IBMPowerVSCluster" IBMPowerVSCluster="default/capi-karthik-19041" namespace="default" name="capi-karthik-19041" reconcileID="379721b4-2074-4e88-9d49-484d73477aa5" cluster="default/capi-karthik-19041"
I0423 11:52:49.478558       1 ibmpowervscluster_controller.go:157] "Reconciling network" controller="ibmpowervscluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="IBMPowerVSCluster" IBMPowerVSCluster="default/capi-karthik-19041" namespace="default" name="capi-karthik-19041" reconcileID="379721b4-2074-4e88-9d49-484d73477aa5" cluster="default/capi-karthik-19041"
I0423 11:52:49.709738       1 leaderelection.go:281] successfully renewed lease capi-ibmcloud-system/effcf9b8.cluster.x-k8s.io
I0423 11:52:50.301457       1 powervs_cluster.go:774] "Found PowerVS network in IBM Cloud" controller="ibmpowervscluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="IBMPowerVSCluster" IBMPowerVSCluster="default/capi-karthik-19041" namespace="default" name="capi-karthik-19041" reconcileID="379721b4-2074-4e88-9d49-484d73477aa5" cluster="default/capi-karthik-19041" id="2ac7e26c-a2e6-4c8f-affa-0dc02ae9d6db"

What did you expect to happen:

DHCP server creation should have been failed

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api version:
• Minikube/KIND version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[Karthik-K-N]

Not sure what we can do to avoid this, Created this issue for reference.

[mkumatag]

Not sure what we can do to avoid this, Created this issue for reference.

May be we need to set Controller Created true irrespective of success or failure when create path is hit.

[Karthik-K-N]

Yeah, Whats one option, May be based on the number of occurance of these kind of error we can take the approach.

For now we try to create resource followed by updating status. https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/blob/main/cloud/scope/powervs_cluster.go#L889-L901.

[mkumatag]

not seen this anymore, hence closing this bug