AWS: All submariner-routeagent pods got into CrashLoopBackOff

[manosnoam]

What happened:

Right after a new submariner deploy and join, I see:

pod/submariner-routeagent-8554g                      0/1     CrashLoopBackOff   1          25s   app=submariner-routeagent,component=routeagent,controller-revision-hash=c677d9594,pod-template-generation=1
pod/submariner-routeagent-c7494                      0/1     CrashLoopBackOff   1          25s   app=submariner-routeagent,component=routeagent,controller-revision-hash=c677d9594,pod-template-generation=1
pod/submariner-routeagent-fmg5c                      0/1     CrashLoopBackOff   1          25s   app=submariner-routeagent,component=routeagent,controller-revision-hash=c677d9594,pod-template-generation=1
pod/submariner-routeagent-mf2f5                      0/1     CrashLoopBackOff   1          25s   app=submariner-routeagent,component=routeagent,controller-revision-hash=c677d9594,pod-template-generation=1

In routeagent pods logs I see:

Command:
      submariner-route-agent.sh
    State:          Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Wed, 22 Jul 2020 10:54:12 +0300
      Finished:     Wed, 22 Jul 2020 10:54:12 +0300

F0722 07:54:09.225159       1 main.go:97] Error running route controller: error annotating the node "ip-10-166-56-196" with cniIfaceIP: error updatating node "ip-10-166-56-196", err: unable to get node info for node ip-10-166-56-196, err: nodes "ip-10-166-56-196" not found

This issue is reproduced, so it's a blocker.

Please see full test report, including pod logs collection (last step):

• https://qe-jenkins-csb-skynet.cloud.paas.psi.redhat.com/job/debug_job/103/Test-Report/
• https://qe-jenkins-csb-skynet.cloud.paas.psi.redhat.com/job/Submariner-OSP-AWS/760/Test-Report/

Environment:

• subctl version: v0.5.0-6-gad8f10a

• Cluster A (AWS public):
  Client Version: 4.5.2
  Server Version: 4.5.2
  Kubernetes Version: v1.18.3+b74c5ed

• Cluster B (OSP on-prem):
  Client Version: 4.5.2
  Server Version: 4.4.3
  Kubernetes Version: v1.17.1

[manosnoam]

According to @sridhargaddam this bug is related to recent change:
#729

Its something to do with hostnames. Which are constructed in a different manner on AWS only (it does not occur on OSP).
Basically its a new issue and looks like we did not validate hostNetworking to remoteService use-case where the sourcePod was on AWS Cluster.

[sridhargaddam]

On AWS Clusters, it was seen that nodes are named with FQDN (i.e., hostname.domainname). However, on many of the other
K8s clusters we tested so far (like KIND, OSP etc), the hostnames were matching with the nodeNames.

In order to support connectivity from HostNetwork to remoteClusters (when using Globalnet), routeagent Pod annotates the respective nodes with the cniIfaceIP annotation. In the current code, where this annotation is added, we were using the hostname to query the node info. This was failing on AWS Clusters for the issue mentioned above.

This is now fixed via the following PR: #737