subctl diagnose failing when hostname does not match with nodename

[sridhargaddam]

What happened:
As part of subctl diagnose firewall tunnel, we try to find the active gateway node in the cluster so that we can schedule the testPod on that node. In an IBMCluster it was seen that hostname (as stored in the endpoint info) does not match with the nodeName, because of this, the diagnose command is failing.

subctl diagnose firewall tunnel /home/ubuntu/submariner/cluster-c.config /home/ubuntu/submariner/cluster-d.config
⚠ Checking if tunnels can be setup on Gateway node of cluster "c3fvuprd0vobcoseot80".
⚠ Could not find the active Gateway nodeName in local cluster

k describe gateways.submariner.io -n submariner-operator kube-c3fvuprd0vobcoseot80-submarinerc-submari-00000146
Name: kube-c3fvuprd0vobcoseot80-submarinerc-submari-00000146
Namespace: submariner-operator
Labels:
Annotations: update-timestamp: 1625435263
API Version: submariner.io/v1
Kind: Gateway
Metadata:
Creation Timestamp: 2021-07-04T21:47:33Z
Generation: 3
Managed Fields:
API Version: submariner.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:update-timestamp:
f:status:
.:
f:connections:
f:haStatus:
f:localEndpoint:
.:
f:backend:
f:backend_config:
.:
f:preferred-server:
f:cable_name:
f:cluster_id:
f:healthCheckIP:
f:hostname:
f:nat_enabled:
f:private_ip:
f:public_ip:
f:subnets:
f:statusFailure:
f:version:
Manager: submariner-gateway
Operation: Update
Time: 2021-07-04T21:47:33Z
Resource Version: 302733
UID: eef3a1b7-29d7-4628-abde-72520696fec0
Status:
Connections:
Endpoint:
Backend: libreswan
backend_config:
Preferred - Server: false
cable_name: submariner-cable-c3g1045d0j7fl990v4q0-10-240-64-12
cluster_id: c3g1045d0j7fl990v4q0
Health Check IP: 172.17.35.0
Hostname: kube-c3g1045d0j7fl990v4q0-submarinerd-default-000001fb
nat_enabled: true
private_ip: 10.240.64.12
public_ip: 150.239.164.46
Subnets:
172.22.0.0/16
172.17.0.0/18
Latency RTT:
Average: 0s
Last: 0s
Max: 0s
Min: 0s
Std Dev: 0s
Status: connecting
Status Message:
Using IP: 150.239.164.46
Using NAT: true
Ha Status: active
Local Endpoint:
Backend: libreswan
backend_config:
Preferred - Server: false
cable_name: submariner-cable-c3fvuprd0vobcoseot80-10-240-128-4
cluster_id: c3fvuprd0vobcoseot80
Health Check IP: 172.17.111.0
Hostname: kube-c3fvuprd0vobcoseot80-submarinerc-submari-00000146
nat_enabled: true
private_ip: 10.240.128.4
public_ip: 52.118.79.113
Subnets:
172.21.0.0/16
172.17.64.0/18
Status Failure:
Version: release-0.9-c0e2e61
Events:

In GKE cluster the hostname was coming as the actual nodename. IN IBM cluster it is not name but ibm-cloud.kubernetes.io/worker-id

k get nodes --show-labels --selector=submariner.io/gateway=true
NAME STATUS ROLES AGE VERSION LABELS
10.240.128.4 Ready 47h v1.20.7+IKS arch=amd64,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=bx2.4x16,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-south,failure-domain.beta.kubernetes.io/zone=us-south-3,ibm-cloud.kubernetes.io/ha-worker=true,ibm-cloud.kubernetes.io/iaas-provider=g2,ibm-cloud.kubernetes.io/instance-id=0737_c65a86db-e7a9-4d83-84d1-d2108ba062f3,ibm-cloud.kubernetes.io/internal-ip=10.240.128.4,ibm-cloud.kubernetes.io/machine-type=bx2.4x16,ibm-cloud.kubernetes.io/os=UBUNTU_18_64,ibm-cloud.kubernetes.io/region=us-south,ibm-cloud.kubernetes.io/sgx-enabled=false,ibm-cloud.kubernetes.io/subnet-id=0737-f612d62f-e946-4d24-8e12-2dc25b171efa,ibm-cloud.kubernetes.io/worker-id=kube-c3fvuprd0vobcoseot80-submarinerc-submari-00000146,ibm-cloud.kubernetes.io/worker-pool-id=c3fvuprd0vobcoseot80-3e04804,ibm-cloud.kubernetes.io/worker-pool-name=submariner-c-test,ibm-cloud.kubernetes.io/worker-version=1.20.7_1543,ibm-cloud.kubernetes.io/zone=us-south-3,kubernetes.io/arch=amd64,kubernetes.io/hostname=10.240.128.4,kubernetes.io/os=linux,node.kubernetes.io/instance-type=bx2.4x16,submariner.io/gateway=true,topology.kubernetes.io/region=us-south,topology.kubernetes.io/zone=us-south-3