CI: Suite-runtime.RuntimeFQDNPolicies Implements matchPattern: "*": DNS resolves but curl times out: Process exited with status 28

[joestringer]

Suite-runtime.RuntimeFQDNPolicies Implements matchPattern: "*"

https://jenkins.cilium.io/job/cilium-master-runtime-kernel-4.9/2247/testReport/junit/(root)/Suite-runtime/RuntimeFQDNPolicies_Implements_matchPattern_____/
ad391c17_RuntimeFQDNPolicies_Implements_matchPattern___.zip

Possibly related to #16724.

Failing for the past 1 build (Since #2247 )

Took 18 sec.

Stacktrace

/home/jenkins/workspace/cilium-master-runtime-kernel-4.9/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:518
Curl to world3.cilium.test failed when in deny-all due to toFQDNs
Expected command: docker exec -i  app1 curl --path-as-is -s -D /dev/stderr --fail --connect-timeout 5 --max-time 20 world3.cilium.test -w "time-> DNS: '%{time_namelookup}(%{remote_ip})', Connect: '%{time_connect}',Transfer '%{time_starttransfer}', total '%{time_total}'" 
To succeed, but it failed:
Exitcode: 28 
Err: Process exited with status 28
Stdout:
 	 time-> DNS: '0.009256()', Connect: '0.000000',Transfer '0.000000', total '5.000660'
Stderr:
/home/jenkins/workspace/cilium-master-runtime-kernel-4.9/runtime-gopath/src/github.com/cilium/cilium/test/runtime/fqdn.go:784

Standard Output

Number of "context deadline exceeded" in logs: 0

Number of "level=error" in logs: 0

Number of "level=warning" in logs: 0

Number of "Cilium API handler panicked" in logs: 0

Number of "Goroutine took lock for more than" in logs: 1

No errors/warnings found in logs

Standard Error

17:31:58 STEP: Importing policy with matchPattern: "*" rule

17:31:58 STEP: Setting up policy: /home/vagrant/go/src/github.com/cilium/cilium/test/policy_0f5df029.json

17:31:59 STEP: Denying egress to any IPs or domains

FAIL: Curl to world3.cilium.test failed when in deny-all due to toFQDNs

Expected command: docker exec -i  app1 curl --path-as-is -s -D /dev/stderr --fail --connect-timeout 5 --max-time 20 world3.cilium.test -w "time-> DNS: '%{time_namelookup}(%{remote_ip})', Connect: '%{time_connect}',Transfer '%{time_starttransfer}', total '%{time_total}'"

To succeed, but it failed:

Exitcode: 28

Err: Process exited with status 28

Stdout:

time-> DNS: '0.009256()', Connect: '0.000000',Transfer '0.000000', total '5.000660'

Stderr:

=== Test Finished at 2021-06-29T17:32:06Z====

17:32:06 STEP: Running JustAfterEach block for EntireTestsuite RuntimeFQDNPolicies

===================== TEST FAILED =====================

17:32:06 STEP: Running AfterFailed block for EntireTestsuite RuntimeFQDNPolicies

10.15.176.177 httpd1

10.15.167.1 app3

10.15.217.239 app2

10.15.97.48 app1

10.15.161.68 httpd3

10.15.186.57 httpd2

172.17.0.5 bind

172.18.0.7 OutsideHttpd2

172.18.0.6 OutsideHttpd1

172.18.0.5 OutsideHttpd3

172.18.0.4 WorldHttpd3

172.18.0.3 WorldHttpd2

172.18.0.2 WorldHttpd1

172.17.0.4 cilium-etcd

172.17.0.3 cilium-consul

172.17.0.2 registry
cmd: sudo cilium endpoint list

Exitcode: 0

Stdout:

ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4            STATUS

ENFORCEMENT        ENFORCEMENT

374        Disabled           Disabled          2184       container:app=test            f00d::a0f:0:0:3b15   10.15.217.239   ready

container:id.app2

479        Disabled           Disabled          7762       container:app=test            f00d::a0f:0:0:9d7d   10.15.186.57    ready

container:id.httpd2

container:id.service1

578        Disabled           Disabled          283        container:app=test            f00d::a0f:0:0:aea1   10.15.161.68    ready

container:id.httpd3

container:id.service1

661        Disabled           Disabled          26095      container:app=test            f00d::a0f:0:0:ae24   10.15.167.1     ready

container:id.app3

1325       Disabled           Disabled          1          reserved:host                                                      ready

1678       Disabled           Disabled          21355      container:app=test            f00d::a0f:0:0:357a   10.15.176.177   ready

container:id.httpd1

container:id.service1

1970       Disabled           Disabled          4          reserved:health               f00d::a0f:0:0:5d8a   10.15.240.239   ready

3727       Disabled           Enabled           7233       container:app=test            f00d::a0f:0:0:7779   10.15.97.48     ready

container:id.app1
Stderr:
cmd: sudo cilium policy get

Exitcode: 0

Stdout:

[

{

"endpointSelector": {

"matchLabels": {

"container:id.app1": ""

}

},

"egress": [

{

"toPorts": [

{

"ports": [

{

"port": "53",

"protocol": "ANY"

}

],

"rules": {

"dns": [

{

"matchPattern": ""

}

]

}

}

]

},

{

"toFQDNs": [

{

"matchPattern": "world1.cilium.test"

},

{

"matchPattern": "world.cilium.test"

},

{

"matchPattern": "level*CNAME.cilium.test"

}

]

}

],

"labels": [

{

"key": "toFQDNs-runtime-test-policy",

"source": ""

}

]

}

]

Revision: 22
Stderr:
===================== Exiting AfterFailed =====================

17:32:15 STEP: Running AfterEach for block EntireTestsuite RuntimeFQDNPolicies

17:32:16 STEP: Running AfterEach for block EntireTestsuite
[[ATTACHMENT|ad391c17_RuntimeFQDNPolicies_Implements_matchPattern:_"*".zip]]

[pchaigno]

Possibly related to #16724.

Once #16754 (missing bugtool) is fixed, we can check if it's the same root cause.