TestWireGuard/testServiceConnectivity failing in IPv6 cluster

[antoninbas]

Describe the bug
After merging #3336, the e2e test TestWireGuard/testServiceConnectivity is failing for the jenkins-ipv6-only-e2e CI job.

It seems that the new IPv6 route reconciliation logic prevents correct gateway route configuration on the Node.

If we deploy Antrea without Wireguard, the gateway routes look like this:

fd02::1 dev antrea-gw0 metric 1024 pref medium # peer Pod gateway
fd02::/64 via fd02::1 dev antrea-gw0 metric 1024 pref medium # peer Pod CIDR
fd02:0:0:1::/64 dev antrea-gw0 proto kernel metric 256 pref medium # local Pod CIDR

If we deploy Antrea with Wireguard, the routes (antrea-gw0 + antrea-wg0) look like this:

fd02::/64 dev antrea-wg0 src fd02:0:0:1::1 metric 1024 pref medium
fd02:0:0:1::1 dev antrea-wg0 proto kernel metric 256 pref medium
fd02:0:0:1::/64 dev antrea-gw0 proto kernel metric 256 pref medium

However, if we first deploy Antrea without Wireguard, then re-deploy it (apply the yaml again) with Wireguard enabled (which is what the e2e test case does), we get the following routes:

fd02::1 dev antrea-gw0 metric 1024 pref medium
fd02::/64 dev antrea-wg0 src fd02:0:0:1::1 metric 1024 pref medium
fd02:0:0:1::1 dev antrea-wg0 proto kernel metric 256 pref medium
fd02:0:0:1::/64 dev antrea-gw0 proto kernel metric 256 pref medium

Notice how there is an "extra" route (the first one). This could explain why the test is failing.

Versions:
Antrea: top of tree

[antoninbas]

Test failure:

=== RUN   TestWireGuard/testServiceConnectivity
    service_test.go:183: Created service Pod IPs [fd74:ca9b:172:16:1::1545]
    wireguard_test.go:150: 
        	Error Trace:	wireguard_test.go:150
        	            				wireguard_test.go:75
        	Error:      	Received unexpected error:
        	            	nc stdout: <>, stderr: <nc: fd74:ca9b:172:17::e92d ([fd74:ca9b:172:17::e92d]:80): Connection timed out
        	            	nc: fd74:ca9b:172:17::e92d ([fd74:ca9b:172:17::e92d]:80): Connection timed out
        	            	nc: fd74:ca9b:172:17::e92d ([fd74:ca9b:172:17::e92d]:80): Connection timed out
        	            	nc: fd74:ca9b:172:17::e92d ([fd74:ca9b:172:17::e92d]:80): Connection timed out
        	            	nc: fd74:ca9b:172:17::e92d ([fd74:ca9b:172:17::e92d]:80): Connection timed out
        	            	>, err: <command terminated with exit code 1>
        	Test:       	TestWireGuard/testServiceConnectivity
        	Messages:   	Pod hostnetwork-pod should be able to connect the service's ClusterIP [fd74:ca9b:172:17::e92d]:80, but was not able to connect
=== CONT  TestWireGuard
    fixtures.go:269: Exporting test logs to '/var/lib/jenkins/workspace/antrea-ipv6-only-e2e-for-pull-request/antrea-test-logs/TestWireGuard/beforeTeardown.Mar10-20-48-30'
    fixtures.go:375: Error when exporting kubelet logs: error when running journalctl on Node 'antrea-ipv6-2-0', is it available? Error: <nil>
    fixtures.go:396: Deleting 'antrea-test' K8s Namespace
I0310 20:48:45.848635    6521 framework.go:630] Deleting Namespace antrea-test took 6.005761736s
--- FAIL: TestWireGuard (137.90s)
    --- PASS: TestWireGuard/testPodConnectivity (15.19s)
    --- FAIL: TestWireGuard/testServiceConnectivity (58.18s)