Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: K8sAgentHubbleTest Hubble Observe Test L3/L4 Flow #25068

Closed
maintainer-s-little-helper bot opened this issue Apr 23, 2023 · 5 comments
Closed

CI: K8sAgentHubbleTest Hubble Observe Test L3/L4 Flow #25068

maintainer-s-little-helper bot opened this issue Apr 23, 2023 · 5 comments
Labels
ci/flake This is a known failure that occurs in the tree. Please investigate me! stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.

Comments

@maintainer-s-little-helper
Copy link

maintainer-s-little-helper bot commented Apr 23, 2023 •
edited by glibsm

Test Name

K8sAgentHubbleTest Hubble Observe Test L3/L4 Flow

Failure Output

FAIL: hubble observe query timed out on "Exitcode: 0 
Stdout:
{
  "flow": {
    "time": "2023-04-20T20:21:15.653707777Z",
    "verdict": "FORWARDED",
    "ethernet": {
      "source": "ca:47:d0:56:4d:b3",
      "destination": "36:52:b0:81:06:a6"
    },
    "IP": {
      "source": "10.0.1.49",
      "destination": "10.0.1.173",
      "ipVersion": "IPv4"
    },
    "l4": {
      "TCP": {
        "source_port": 60870,
        "destination_port": 80,
        "flags": {
          "SYN": true
        }
      }
    },
    "source": {
      "ID": 174,
      "identity": 7471,
      "namespace": "202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
      "labels": [
        "k8s:appSecond=true",
        "k8s:id=app2",
        "k8s:io.cilium.k8s.policy.cluster=default",
        "k8s:io.cilium.k8s.policy.serviceaccount=app2-account",
        "k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
        "k8s:zgroup=testapp"
      ],
      "pod_name": "app2-6dfff7bfbd-djx8s",
      "workloads": [
        {
          "name": "app2",
          "kind": "Deployment"
        }
      ]
    },
    "destination": {
      "ID": 3994,
      "identity": 19661,
      "namespace": "202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
      "labels": [
        "k8s:id=app1",
        "k8s:io.cilium.k8s.policy.cluster=default",
        "k8s:io.cilium.k8s.policy.serviceaccount=app1-account",
        "k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
        "k8s:zgroup=testapp"
      ],
      "pod_name": "app1-755788dd65-lxq77",
      "workloads": [
        {
          "name": "app1",
          "kind": "Deployment"
        }
      ]
    },
    "Type": "L3_L4",
    "node_name": "k8s1",
    "event_type": {
      "type": 4
    },
    "traffic_direction": "INGRESS",
    "trace_observation_point": "TO_ENDPOINT",
    "is_reply": false,
    "interface": {
      "index": 26
    },
    "Summary": "TCP Flags: SYN"
  },
  "node_name": "k8s1",
  "time": "2023-04-20T20:21:15.653707777Z"
}
{
  "flow": {
    "time": "2023-04-20T20:21:15.653722598Z",
    "verdict": "FORWARDED",
    "ethernet": {
      "source": "ca:47:d0:56:4d:b3",
      "destination": "36:52:b0:81:06:a6"
    },
    "IP": {
      "source": "10.0.1.49",
      "destination": "10.0.1.173",
      "ipVersion": "IPv4"
    },
    "l4": {
      "TCP": {
        "source_port": 60870,
        "destination_port": 80,
        "flags": {
          "ACK": true
        }
      }
    },
    "source": {
      "ID": 174,
      "identity": 7471,
      "namespace": "202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
      "labels": [
        "k8s:appSecond=true",
        "k8s:id=app2",
        "k8s:io.cilium.k8s.policy.cluster=default",
        "k8s:io.cilium.k8s.policy.serviceaccount=app2-account",
        "k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
        "k8s:zgroup=testapp"
      ],
      "pod_name": "app2-6dfff7bfbd-djx8s",
      "workloads": [
        {
          "name": "app2",
          "kind": "Deployment"
        }
      ]
    },
    "destination": {
      "ID": 3994,
      "identity": 19661,
      "namespace": "202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
      "labels": [
        "k8s:id=app1",
        "k8s:io.cilium.k8s.policy.cluster=default",
        "k8s:io.cilium.k8s.policy.serviceaccount=app1-account",
        "k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
        "k8s:zgroup=testapp"
      ],
      "pod_name": "app1-755788dd65-lxq77",
      "workloads": [
        {
          "name": "app1",
          "kind": "Deployment"
        }
      ]
    },
    "Type": "L3_L4",
    "node_name": "k8s1",
    "event_type": {
      "type": 4
    },
    "traffic_direction": "INGRESS",
    "trace_observation_point": "TO_ENDPOINT",
    "is_reply": false,
    "interface": {
      "index": 26
    },
    "Summary": "TCP Flags: ACK"
  },
  "node_name": "k8s1",
  "time": "2023-04-20T20:21:15.653722598Z"
}
{
  "flow": {
    "time": "2023-04-20T20:21:15.654641146Z",
    "verdict": "FORWARDED",
    "ethernet": {
      "source": "ca:47:d0:56:4d:b3",
      "destination": "36:52:b0:81:06:a6"
    },
    "IP": {
      "source": "10.0.1.49",
      "destination": "10.0.1.173",
      "ipVersion": "IPv4"
    },
    "l4": {
      "TCP": {
        "source_port": 60870,
        "destination_port": 80,
        "flags": {
          "PSH": true,
          "ACK": true
        }
      }
    },
    "source": {
      "ID": 174,
      "identity": 7471,
      "namespace": "202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
      "labels": [
        "k8s:appSecond=true",
        "k8s:id=app2",
        "k8s:io.cilium.k8s.policy.cluster=default",
        "k8s:io.cilium.k8s.policy.serviceaccount=app2-account",
        "k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
        "k8s:zgroup=testapp"
      ],
      "pod_name": "app2-6dfff7bfbd-djx8s",
      "workloads": [
        {
          "name": "app2",
          "kind": "Deployment"
        }
      ]
    },
    "destination": {
      "ID": 3994,
      "identity": 19661,
      "namespace": "202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
      "labels": [
        "k8s:id=app1",
        "k8s:io.cilium.k8s.policy.cluster=default",
        "k8s:io.cilium.k8s.policy.serviceaccount=app1-account",
        "k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
        "k8s:zgroup=testapp"
      ],
      "pod_name": "app1-755788dd65-lxq77",
      "workloads": [
        {
          "name": "app1",
          "kind": "Deployment"
        }
      ]
    },
    "Type": "L3_L4",
    "node_name": "k8s1",
    "event_type": {
      "type": 4
    },
    "traffic_direction": "INGRESS",
    "trace_observation_point": "TO_ENDPOINT",
    "is_reply": false,
    "interface": {
      "index": 26
    },
    "Summary": "TCP Flags: ACK, PSH"
  },
  "node_name": "k8s1",
  "time": "2023-04-20T20:21:15.654641146Z"
}
{
  "flow": {
    "time": "2023-04-20T20:21:15.654659891Z",
    "verdict": "FORWARDED",
    "ethernet": {
      "source": "ca:47:d0:56:4d:b3",
      "destination": "36:52:b0:81:06:a6"
    },
    "IP": {
      "source": "10.0.1.49",
      "destination": "10.0.1.173",
      "ipVersion": "IPv4"
    },
    "l4": {
      "TCP": {
        "source_port": 60870,
        "destination_port": 80,
        "flags": {
          "ACK": true
        }
      }
    },
    "source": {
      "ID": 174,
      "identity": 7471,
      "namespace": "202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
      "labels": [
        "k8s:appSecond=true",
        "k8s:id=app2",
        "k8s:io.cilium.k8s.policy.cluster=default",
        "k8s:io.cilium.k8s.policy.serviceaccount=app2-account",
        "k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
        "k8s:zgroup=testapp"
      ],
      "pod_name": "app2-6dfff7bfbd-djx8s",
      "workloads": [
        {
          "name": "app2",
          "kind": "Deployment"
        }
      ]
    },
    "destination": {
      "ID": 3994,
      "identity": 19661,
      "namespace": "202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
      "labels": [
        "k8s:id=app1",
        "k8s:io.cilium.k8s.policy.cluster=default",
        "k8s:io.cilium.k8s.policy.serviceaccount=app1-account",
        "k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
        "k8s:zgroup=testapp"
      ],
      "pod_name": "app1-755788dd65-lxq77",
      "workloads": [
        {
          "name": "app1",
          "kind": "Deployment"
        }
      ]
    },
    "Type": "L3_L4",
    "node_name": "k8s1",
    "event_type": {
      "type": 4
    },
    "traffic_direction": "INGRESS",
    "trace_observation_point": "TO_ENDPOINT",
    "is_reply": false,
    "interface": {
      "index": 26
    },
    "Summary": "TCP Flags: ACK"
  },
  "node_name": "k8s1",
  "time": "2023-04-20T20:21:15.654659891Z"
}
{
  "flow": {
    "time": "2023-04-20T20:21:15.655084625Z",
    "verdict": "FORWARDED",
    "ethernet": {
      "source": "ca:47:d0:56:4d:b3",
      "destination": "36:52:b0:81:06:a6"
    },
    "IP": {
      "source": "10.0.1.49",
      "destination": "10.0.1.173",
      "ipVersion": "IPv4"
    },
    "l4": {
      "TCP": {
        "source_port": 60870,
        "destination_port": 80,
        "flags": {
          "FIN": true,
          "ACK": true
        }
      }
    },
    "source": {
      "ID": 174,
      "identity": 7471,
      "namespace": "202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
      "labels": [
        "k8s:appSecond=true",
        "k8s:id=app2",
        "k8s:io.cilium.k8s.policy.cluster=default",
        "k8s:io.cilium.k8s.policy.serviceaccount=app2-account",
        "k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
        "k8s:zgroup=testapp"
      ],
      "pod_name": "app2-6dfff7bfbd-djx8s",
      "workloads": [
        {
          "name": "app2",
          "kind": "Deployment"
        }
      ]
    },
    "destination": {
      "ID": 3994,
      "identity": 19661,
      "namespace": "202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
      "labels": [
        "k8s:id=app1",
        "k8s:io.cilium.k8s.policy.cluster=default",
        "k8s:io.cilium.k8s.policy.serviceaccount=app1-account",
        "k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow",
        "k8s:zgroup=testapp"
      ],
      "pod_name": "app1-755788dd65-lxq77",
      "workloads": [
        {
          "name": "app1",
          "kind": "Deployment"
        }
      ]
    },
    "Type": "L3_L4",
    "node_name": "k8s1",
    "event_type": {
      "type": 4
    },
    "traffic_direction": "INGRESS",
    "trace_observation_point": "TO_ENDPOINT",
    "is_reply": false,
    "interface": {
      "index": 26
    },
    "Summary": "TCP Flags: ACK, FIN"
  },
  "node_name": "k8s1",
  "time": "2023-04-20T20:21:15.655084625Z"
}```

### Stacktrace
<details><summary>Click to show.</summary>

```stack-trace
/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
hubble observe query timed out on "Exitcode: 0 \nStdout:\n \t {\"flow\":{\"time\":\"2023-04-20T20:21:15.653707777Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"ca:47:d0:56:4d:b3\",\"destination\":\"36:52:b0:81:06:a6\"},\"IP\":{\"source\":\"10.0.1.49\",\"destination\":\"10.0.1.173\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":60870,\"destination_port\":80,\"flags\":{\"SYN\":true}}},\"source\":{\"ID\":174,\"identity\":7471,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-djx8s\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":3994,\"identity\":19661,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-lxq77\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"event_type\":{\"type\":4},\"traffic_direction\":\"INGRESS\",\"trace_observation_point\":\"TO_ENDPOINT\",\"is_reply\":false,\"interface\":{\"index\":26},\"Summary\":\"TCP Flags: SYN\"},\"node_name\":\"k8s1\",\"time\":\"2023-04-20T20:21:15.653707777Z\"}\n\t {\"flow\":{\"time\":\"2023-04-20T20:21:15.653722598Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"ca:47:d0:56:4d:b3\",\"destination\":\"36:52:b0:81:06:a6\"},\"IP\":{\"source\":\"10.0.1.49\",\"destination\":\"10.0.1.173\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":60870,\"destination_port\":80,\"flags\":{\"ACK\":true}}},\"source\":{\"ID\":174,\"identity\":7471,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-djx8s\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":3994,\"identity\":19661,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-lxq77\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"event_type\":{\"type\":4},\"traffic_direction\":\"INGRESS\",\"trace_observation_point\":\"TO_ENDPOINT\",\"is_reply\":false,\"interface\":{\"index\":26},\"Summary\":\"TCP Flags: ACK\"},\"node_name\":\"k8s1\",\"time\":\"2023-04-20T20:21:15.653722598Z\"}\n\t {\"flow\":{\"time\":\"2023-04-20T20:21:15.654641146Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"ca:47:d0:56:4d:b3\",\"destination\":\"36:52:b0:81:06:a6\"},\"IP\":{\"source\":\"10.0.1.49\",\"destination\":\"10.0.1.173\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":60870,\"destination_port\":80,\"flags\":{\"PSH\":true,\"ACK\":true}}},\"source\":{\"ID\":174,\"identity\":7471,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-djx8s\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":3994,\"identity\":19661,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-lxq77\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"event_type\":{\"type\":4},\"traffic_direction\":\"INGRESS\",\"trace_observation_point\":\"TO_ENDPOINT\",\"is_reply\":false,\"interface\":{\"index\":26},\"Summary\":\"TCP Flags: ACK, PSH\"},\"node_name\":\"k8s1\",\"time\":\"2023-04-20T20:21:15.654641146Z\"}\n\t {\"flow\":{\"time\":\"2023-04-20T20:21:15.654659891Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"ca:47:d0:56:4d:b3\",\"destination\":\"36:52:b0:81:06:a6\"},\"IP\":{\"source\":\"10.0.1.49\",\"destination\":\"10.0.1.173\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":60870,\"destination_port\":80,\"flags\":{\"ACK\":true}}},\"source\":{\"ID\":174,\"identity\":7471,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-djx8s\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":3994,\"identity\":19661,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-lxq77\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"event_type\":{\"type\":4},\"traffic_direction\":\"INGRESS\",\"trace_observation_point\":\"TO_ENDPOINT\",\"is_reply\":false,\"interface\":{\"index\":26},\"Summary\":\"TCP Flags: ACK\"},\"node_name\":\"k8s1\",\"time\":\"2023-04-20T20:21:15.654659891Z\"}\n\t {\"flow\":{\"time\":\"2023-04-20T20:21:15.655084625Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"ca:47:d0:56:4d:b3\",\"destination\":\"36:52:b0:81:06:a6\"},\"IP\":{\"source\":\"10.0.1.49\",\"destination\":\"10.0.1.173\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":60870,\"destination_port\":80,\"flags\":{\"FIN\":true,\"ACK\":true}}},\"source\":{\"ID\":174,\"identity\":7471,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-djx8s\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":3994,\"identity\":19661,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-lxq77\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"event_type\":{\"type\":4},\"traffic_direction\":\"INGRESS\",\"trace_observation_point\":\"TO_ENDPOINT\",\"is_reply\":false,\"interface\":{\"index\":26},\"Summary\":\"TCP Flags: ACK, FIN\"},\"node_name\":\"k8s1\",\"time\":\"2023-04-20T20:21:15.655084625Z\"}\n\t \nStderr:\n \t \n"
Expected
    <*errors.errorString | 0xc000d5cdb0>: 
    could not parse "{\"flow\":{\"time\":\"2023-04-20T20:21:15.655084625Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"ca:47:d0:56:4d:b3\",\"destination\":\"36:52:b0:81:06:a6\"},\"IP\":{\"source\":\"10.0.1.49\",\"destination\":\"10.0.1.173\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":60870,\"destination_port\":80,\"flags\":{\"FIN\":true,\"ACK\":true}}},\"source\":{\"ID\":174,\"identity\":7471,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-djx8s\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":3994,\"identity\":19661,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-lxq77\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_n" as JSON (line 4 of "kubectl exec -n kube-system cilium-mzq5q -c cilium-agent -- hubble observe --follow --output=jsonpb --last 1 --type trace --from-pod 202304202021k8sagenthubbletesthubbleobservetestl3l4flow/app2-6dfff7bfbd-djx8s --to-namespace 202304202021k8sagenthubbletesthubbleobservetestl3l4flow --to-label id=app1,zgroup=testapp --to-port 80")
    {
        s: "could not parse \"{\\\"flow\\\":{\\\"time\\\":\\\"2023-04-20T20:21:15.655084625Z\\\",\\\"verdict\\\":\\\"FORWARDED\\\",\\\"ethernet\\\":{\\\"source\\\":\\\"ca:47:d0:56:4d:b3\\\",\\\"destination\\\":\\\"36:52:b0:81:06:a6\\\"},\\\"IP\\\":{\\\"source\\\":\\\"10.0.1.49\\\",\\\"destination\\\":\\\"10.0.1.173\\\",\\\"ipVersion\\\":\\\"IPv4\\\"},\\\"l4\\\":{\\\"TCP\\\":{\\\"source_port\\\":60870,\\\"destination_port\\\":80,\\\"flags\\\":{\\\"FIN\\\":true,\\\"ACK\\\":true}}},\\\"source\\\":{\\\"ID\\\":174,\\\"identity\\\":7471,\\\"namespace\\\":\\\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\\\",\\\"labels\\\":[\\\"k8s:appSecond=true\\\",\\\"k8s:id=app2\\\",\\\"k8s:io.cilium.k8s.policy.cluster=default\\\",\\\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\\\",\\\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\\\",\\\"k8s:zgroup=testapp\\\"],\\\"pod_name\\\":\\\"app2-6dfff7bfbd-djx8s\\\",\\\"workloads\\\":[{\\\"name\\\":\\\"app2\\\",\\\"kind\\\":\\\"Deployment\\\"}]},\\\"destination\\\":{\\\"ID\\\":3994,\\\"identity\\\":19661,\\\"namespace\\\":\\\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\\\",\\\"labels\\\":[\\\"k8s:id=app1\\\",\\\"k8s:io.cilium.k8s.policy.cluster=default\\\",\\\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\\\",\\\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\\\",\\\"k8s:zgroup=testapp\\\"],\\\"pod_name\\\":\\\"app1-755788dd65-lxq77\\\",\\\"workloads\\\":[{\\\"name\\\":\\\"app1\\\",\\\"kind\\\":\\\"Deployment\\\"}]},\\\"Type\\\":\\\"L3_L4\\\",\\\"node_n\" as JSON (line 4 of \"kubectl exec -n kube-system cilium-mzq5q -c cilium-agent -- hubble observe --follow --output=jsonpb --last 1 --type trace --from-pod 202304202021k8sagenthubbletesthubbleobservetestl3l4flow/app2-6dfff7bfbd-djx8s --to-namespace 202304202021k8sagenthubbletesthubbleobservetestl3l4flow --to-label id=app1,zgroup=testapp --to-port 80\")",
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19/src/github.com/cilium/cilium/test/k8s/hubble.go:191

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 1
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 1
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 1 errors/warnings:
Failed to create peer client for peers synchronization; will try again after the timeout has expired
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 1
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 1 errors/warnings:
Network status error received, restarting client connections
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 5
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
Unable to get node resource
Waiting for k8s node information
UpdateIdentities: Skipping Delete of a non-existing identity
Cilium pods: [cilium-mzq5q cilium-z6xv6]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                             Ingress   Egress
hubble-relay-7ff9f4db57-mf5j7   false     false
app1-755788dd65-lxq77           false     false
app1-755788dd65-sb7zt           false     false
app2-6dfff7bfbd-djx8s           false     false
app3-68f6b87dd5-6kwhx           false     false
grafana-585bb89877-xcxr9        false     false
prometheus-8885c5888-w65sg      false     false
coredns-758664cbbf-rp67r        false     false
Cilium agent 'cilium-mzq5q': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 48 Failed 0
Cilium agent 'cilium-z6xv6': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 24 Failed 0

Standard Error

Click to show. 
20:18:58 STEP: Running BeforeAll block for EntireTestsuite
20:18:58 STEP: Starting tests: command line parameters: {Reprovision:false HoldEnvironment:false PassCLIEnvironment:true SSHConfig: ShowCommands:false TestScope: SkipLogGathering:false CiliumImage:quay.io/cilium/cilium-ci CiliumTag:4ebf90ed3fa553910659bdfdc38466b255422a40 CiliumOperatorImage:quay.io/cilium/operator CiliumOperatorTag:4ebf90ed3fa553910659bdfdc38466b255422a40 CiliumOperatorSuffix:-ci HubbleRelayImage:quay.io/cilium/hubble-relay-ci HubbleRelayTag:4ebf90ed3fa553910659bdfdc38466b255422a40 ProvisionK8s:true Timeout:2h50m0s Kubeconfig:/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19/src/github.com/cilium/cilium/test/vagrant-kubeconfig KubectlPath:/tmp/kubectl RegistryCredentials: Multinode:true RunQuarantined:false Help:false} environment variables: [JENKINS_HOME=/var/jenkins_home ghprbSourceBranch=pr/christarazi/operator-version VM_MEMORY=8192 MAIL=/var/mail/root SSH_CLIENT=54.148.123.155 40410 22 USER=root PROJ_PATH=src/github.com/cilium/cilium RUN_CHANGES_DISPLAY_URL=https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/1028/display/redirect?page=changes ghprbPullDescription=GitHub pull request #25018 of commit 4ebf90ed3fa553910659bdfdc38466b255422a40, no merge conflicts. NETNEXT=0 ghprbActualCommit=4ebf90ed3fa553910659bdfdc38466b255422a40 SHLVL=1 CILIUM_TAG=4ebf90ed3fa553910659bdfdc38466b255422a40 NODE_LABELS=baremetal ginkgo nightly node-brave-midge vagrant HUDSON_URL=https://jenkins.cilium.io/ GIT_COMMIT=05a35dae526f021723cc23e2bd5cf698e88c952f OLDPWD=/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19 GINKGO_TIMEOUT=170m HOME=/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19 INTEGRATION_TESTS=true ghprbTriggerAuthorLoginMention=@christarazi BUILD_URL=https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/1028/ ghprbPullAuthorLoginMention=@christarazi HUDSON_COOKIE=2ff23d1a-a9b9-474c-a046-f1443d002612 JENKINS_SERVER_COOKIE=durable-963ac66a0522387382e0af9c3b4bb35c ghprbGhRepository=cilium/cilium DOCKER_TAG=4ebf90ed3fa553910659bdfdc38466b255422a40 JobKernelVersion=419 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus KERNEL=419 CONTAINER_RUNTIME=docker WORKSPACE=/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19 ghprbPullLongDescription=To be consistent with where the version log is printed in the Agent,\nmove the Operator's log statement accordingly to be after the options\nare printed.\n\nThis is a very minor change, but I found it much easier to find the\nversion when it's in the usual spot that we are used to when looking\nat the Agent. Without this change, the version is printed after the hive\nframework initialization. If that fails, we wouldn't be able to see the\nversion log anyway.\n\nSigned-off-by: Chris Tarazi <chris@isovalent.com>\n K8S_NODES=2 TESTDIR=/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19/src/github.com/cilium/cilium/test LOGNAME=root NODE_NAME=node-brave-midge ghprbCredentialsId=ciliumbot _=/usr/bin/java HUBBLE_RELAY_IMAGE=quay.io/cilium/hubble-relay-ci STAGE_NAME=BDD-Test-PR GIT_BRANCH=origin/pr/25018/merge EXECUTOR_NUMBER=0 ghprbTriggerAuthorLogin=christarazi TERM=xterm XDG_SESSION_ID=5 HOST_FIREWALL=0 CILIUM_OPERATOR_TAG=4ebf90ed3fa553910659bdfdc38466b255422a40 BUILD_DISPLAY_NAME=operator/cmd: Move Cilium Operator version log earlier  https://github.com/cilium/cilium/pull/25018  #1028 ghprbPullAuthorLogin=christarazi HUDSON_HOME=/var/jenkins_home JOB_BASE_NAME=Cilium-PR-K8s-1.16-kernel-4.19 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/go/bin:/root/go/bin sha1=origin/pr/25018/merge KUBECONFIG=/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19/src/github.com/cilium/cilium/test/vagrant-kubeconfig FOCUS=K8s BUILD_ID=1028 XDG_RUNTIME_DIR=/run/user/0 BUILD_TAG=jenkins-Cilium-PR-K8s-1.16-kernel-4.19-1028 RUN_QUARANTINED=false CILIUM_IMAGE=quay.io/cilium/cilium-ci JENKINS_URL=https://jenkins.cilium.io/ LANG=C.UTF-8 ghprbCommentBody=/test JOB_URL=https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/ ghprbPullTitle=operator/cmd: Move Cilium Operator version log earlier GIT_URL=https://github.com/cilium/cilium ghprbPullLink=https://github.com/cilium/cilium/pull/25018 BUILD_NUMBER=1028 JENKINS_NODE_COOKIE=21643618-bad1-4ee5-a3d8-73f4ebd3c674 SHELL=/bin/bash GOPATH=/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19 RUN_DISPLAY_URL=https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/1028/display/redirect IMAGE_REGISTRY=quay.io/cilium ghprbAuthorRepoGitUrl=https://github.com/christarazi/cilium.git FAILFAST=false HUDSON_SERVER_COOKIE=693c250bfb7e85bf ghprbTargetBranch=main JOB_DISPLAY_URL=https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/display/redirect K8S_VERSION=1.16 JOB_NAME=Cilium-PR-K8s-1.16-kernel-4.19 SSH_CONNECTION=54.148.123.155 40410 145.40.80.47 22 ghprbPullId=25018 CILIUM_OPERATOR_IMAGE=quay.io/cilium/operator HUBBLE_RELAY_TAG=4ebf90ed3fa553910659bdfdc38466b255422a40 JobK8sVersion=1.16 VM_CPUS=3 PWD=/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19/src/github.com/cilium/cilium/test CILIUM_OPERATOR_SUFFIX=-ci]
20:18:58 STEP: Ensuring the namespace kube-system exists
20:18:58 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
20:19:01 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
20:19:01 STEP: Preparing cluster
20:19:01 STEP: Labelling nodes
20:19:01 STEP: Cleaning up Cilium components
20:19:02 STEP: Running BeforeAll block for EntireTestsuite K8sAgentHubbleTest Hubble Observe
20:19:02 STEP: Ensuring the namespace kube-system exists
20:19:02 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
20:19:02 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
20:19:02 STEP: Installing Cilium
20:19:03 STEP: Waiting for Cilium to become ready
20:20:12 STEP: Restarting unmanaged pods coredns-758664cbbf-qxgmm in namespace kube-system
20:20:18 STEP: Validating if Kubernetes DNS is deployed
20:20:18 STEP: Checking if deployment is ready
20:20:18 STEP: Checking if kube-dns service is plumbed correctly
20:20:18 STEP: Checking if pods have identity
20:20:18 STEP: Checking if DNS can resolve
20:20:28 STEP: Kubernetes DNS is not ready: 10s timeout expired
20:20:28 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
20:20:28 STEP: Waiting for Kubernetes DNS to become operational
20:20:28 STEP: Checking if deployment is ready
20:20:28 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
20:20:29 STEP: Checking service kube-system/kube-dns plumbing in cilium pod cilium-mzq5q: unable to find service backend 10.0.0.176:53 in datapath of cilium pod cilium-mzq5q
20:20:29 STEP: Checking if deployment is ready
20:20:29 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
20:20:30 STEP: Checking if deployment is ready
20:20:30 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
20:20:31 STEP: Checking if deployment is ready
20:20:31 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
20:20:32 STEP: Checking if deployment is ready
20:20:32 STEP: Kubernetes DNS is not ready yet: only 0 of 1 replicas are available
20:20:33 STEP: Checking if deployment is ready
20:20:33 STEP: Checking if kube-dns service is plumbed correctly
20:20:33 STEP: Checking if pods have identity
20:20:33 STEP: Checking if DNS can resolve
20:20:37 STEP: Validating Cilium Installation
20:20:37 STEP: Performing Cilium controllers preflight check
20:20:37 STEP: Performing Cilium health check
20:20:37 STEP: Checking whether host EP regenerated
20:20:37 STEP: Performing Cilium status preflight check
20:20:44 STEP: Performing Cilium service preflight check
20:20:50 STEP: Cilium is not ready yet: cilium services are not set up correctly: Error validating Cilium service on pod {cilium-mzq5q [{0xc0004b7880 0xc000015378} {0xc0004b7a80 0xc0000153a8} {0xc0004b7c80 0xc0000153b8} {0xc0004b7e00 0xc0000153d8} {0xc0004b7fc0 0xc000015410} {0xc000bc6180 0xc000015428} {0xc000bc6340 0xc000015438}] map[10.100.193.224:443:[0.0.0.0:0 (3) (0) [ClusterIP, non-routable] 192.168.56.12:4244 (3) (2) 192.168.56.11:4244 (3) (1)] 10.101.139.19:3000:[0.0.0.0:0 (6) (0) [ClusterIP, non-routable] 10.0.1.84:3000 (6) (1)] 10.101.197.12:80:[0.0.0.0:0 (2) (0) [ClusterIP, non-routable] 10.0.1.15:4245 (2) (1)] 10.107.18.216:9090:[10.0.1.39:9090 (7) (1) 0.0.0.0:0 (7) (0) [ClusterIP, non-routable]] 10.96.0.10:53:[10.0.0.254:53 (5) (1) 0.0.0.0:0 (5) (0) [ClusterIP, non-routable]] 10.96.0.10:9153:[10.0.0.254:9153 (4) (1) 0.0.0.0:0 (4) (0) [ClusterIP, non-routable]] 10.96.0.1:443:[192.168.56.11:6443 (1) (1) 0.0.0.0:0 (1) (0) [ClusterIP, non-routable]]]}: Could not match cilium service backend address 10.0.1.84:3000 with k8s endpoint
20:20:50 STEP: Performing Cilium status preflight check
20:20:50 STEP: Performing Cilium controllers preflight check
20:20:50 STEP: Checking whether host EP regenerated
20:20:50 STEP: Performing Cilium health check
20:20:57 STEP: Performing Cilium service preflight check
20:20:57 STEP: Performing K8s service preflight check
20:21:04 STEP: Waiting for cilium-operator to be ready
20:21:04 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
20:21:04 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
20:21:04 STEP: Waiting for hubble-relay to be ready
20:21:04 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=hubble-relay")
20:21:04 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=hubble-relay") => <nil>
20:21:04 STEP: Deleting namespace 202304202021k8sagenthubbletesthubbleobservetestl3l4flow
20:21:05 STEP: Creating namespace 202304202021k8sagenthubbletesthubbleobservetestl3l4flow
20:21:05 STEP: WaitforPods(namespace="202304202021k8sagenthubbletesthubbleobservetestl3l4flow", filter="-l zgroup=testapp")
20:21:15 STEP: WaitforPods(namespace="202304202021k8sagenthubbletesthubbleobservetestl3l4flow", filter="-l zgroup=testapp") => <nil>
FAIL: hubble observe query timed out on "Exitcode: 0 \nStdout:\n \t {\"flow\":{\"time\":\"2023-04-20T20:21:15.653707777Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"ca:47:d0:56:4d:b3\",\"destination\":\"36:52:b0:81:06:a6\"},\"IP\":{\"source\":\"10.0.1.49\",\"destination\":\"10.0.1.173\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":60870,\"destination_port\":80,\"flags\":{\"SYN\":true}}},\"source\":{\"ID\":174,\"identity\":7471,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-djx8s\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":3994,\"identity\":19661,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-lxq77\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"event_type\":{\"type\":4},\"traffic_direction\":\"INGRESS\",\"trace_observation_point\":\"TO_ENDPOINT\",\"is_reply\":false,\"interface\":{\"index\":26},\"Summary\":\"TCP Flags: SYN\"},\"node_name\":\"k8s1\",\"time\":\"2023-04-20T20:21:15.653707777Z\"}\n\t {\"flow\":{\"time\":\"2023-04-20T20:21:15.653722598Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"ca:47:d0:56:4d:b3\",\"destination\":\"36:52:b0:81:06:a6\"},\"IP\":{\"source\":\"10.0.1.49\",\"destination\":\"10.0.1.173\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":60870,\"destination_port\":80,\"flags\":{\"ACK\":true}}},\"source\":{\"ID\":174,\"identity\":7471,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-djx8s\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":3994,\"identity\":19661,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-lxq77\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"event_type\":{\"type\":4},\"traffic_direction\":\"INGRESS\",\"trace_observation_point\":\"TO_ENDPOINT\",\"is_reply\":false,\"interface\":{\"index\":26},\"Summary\":\"TCP Flags: ACK\"},\"node_name\":\"k8s1\",\"time\":\"2023-04-20T20:21:15.653722598Z\"}\n\t {\"flow\":{\"time\":\"2023-04-20T20:21:15.654641146Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"ca:47:d0:56:4d:b3\",\"destination\":\"36:52:b0:81:06:a6\"},\"IP\":{\"source\":\"10.0.1.49\",\"destination\":\"10.0.1.173\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":60870,\"destination_port\":80,\"flags\":{\"PSH\":true,\"ACK\":true}}},\"source\":{\"ID\":174,\"identity\":7471,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-djx8s\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":3994,\"identity\":19661,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-lxq77\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"event_type\":{\"type\":4},\"traffic_direction\":\"INGRESS\",\"trace_observation_point\":\"TO_ENDPOINT\",\"is_reply\":false,\"interface\":{\"index\":26},\"Summary\":\"TCP Flags: ACK, PSH\"},\"node_name\":\"k8s1\",\"time\":\"2023-04-20T20:21:15.654641146Z\"}\n\t {\"flow\":{\"time\":\"2023-04-20T20:21:15.654659891Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"ca:47:d0:56:4d:b3\",\"destination\":\"36:52:b0:81:06:a6\"},\"IP\":{\"source\":\"10.0.1.49\",\"destination\":\"10.0.1.173\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":60870,\"destination_port\":80,\"flags\":{\"ACK\":true}}},\"source\":{\"ID\":174,\"identity\":7471,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-djx8s\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":3994,\"identity\":19661,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-lxq77\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"event_type\":{\"type\":4},\"traffic_direction\":\"INGRESS\",\"trace_observation_point\":\"TO_ENDPOINT\",\"is_reply\":false,\"interface\":{\"index\":26},\"Summary\":\"TCP Flags: ACK\"},\"node_name\":\"k8s1\",\"time\":\"2023-04-20T20:21:15.654659891Z\"}\n\t {\"flow\":{\"time\":\"2023-04-20T20:21:15.655084625Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"ca:47:d0:56:4d:b3\",\"destination\":\"36:52:b0:81:06:a6\"},\"IP\":{\"source\":\"10.0.1.49\",\"destination\":\"10.0.1.173\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":60870,\"destination_port\":80,\"flags\":{\"FIN\":true,\"ACK\":true}}},\"source\":{\"ID\":174,\"identity\":7471,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-djx8s\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":3994,\"identity\":19661,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-lxq77\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"event_type\":{\"type\":4},\"traffic_direction\":\"INGRESS\",\"trace_observation_point\":\"TO_ENDPOINT\",\"is_reply\":false,\"interface\":{\"index\":26},\"Summary\":\"TCP Flags: ACK, FIN\"},\"node_name\":\"k8s1\",\"time\":\"2023-04-20T20:21:15.655084625Z\"}\n\t \nStderr:\n \t \n"
Expected
    <*errors.errorString | 0xc000d5cdb0>: 
    could not parse "{\"flow\":{\"time\":\"2023-04-20T20:21:15.655084625Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"ca:47:d0:56:4d:b3\",\"destination\":\"36:52:b0:81:06:a6\"},\"IP\":{\"source\":\"10.0.1.49\",\"destination\":\"10.0.1.173\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":60870,\"destination_port\":80,\"flags\":{\"FIN\":true,\"ACK\":true}}},\"source\":{\"ID\":174,\"identity\":7471,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-djx8s\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":3994,\"identity\":19661,\"namespace\":\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-lxq77\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_n" as JSON (line 4 of "kubectl exec -n kube-system cilium-mzq5q -c cilium-agent -- hubble observe --follow --output=jsonpb --last 1 --type trace --from-pod 202304202021k8sagenthubbletesthubbleobservetestl3l4flow/app2-6dfff7bfbd-djx8s --to-namespace 202304202021k8sagenthubbletesthubbleobservetestl3l4flow --to-label id=app1,zgroup=testapp --to-port 80")
    {
        s: "could not parse \"{\\\"flow\\\":{\\\"time\\\":\\\"2023-04-20T20:21:15.655084625Z\\\",\\\"verdict\\\":\\\"FORWARDED\\\",\\\"ethernet\\\":{\\\"source\\\":\\\"ca:47:d0:56:4d:b3\\\",\\\"destination\\\":\\\"36:52:b0:81:06:a6\\\"},\\\"IP\\\":{\\\"source\\\":\\\"10.0.1.49\\\",\\\"destination\\\":\\\"10.0.1.173\\\",\\\"ipVersion\\\":\\\"IPv4\\\"},\\\"l4\\\":{\\\"TCP\\\":{\\\"source_port\\\":60870,\\\"destination_port\\\":80,\\\"flags\\\":{\\\"FIN\\\":true,\\\"ACK\\\":true}}},\\\"source\\\":{\\\"ID\\\":174,\\\"identity\\\":7471,\\\"namespace\\\":\\\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\\\",\\\"labels\\\":[\\\"k8s:appSecond=true\\\",\\\"k8s:id=app2\\\",\\\"k8s:io.cilium.k8s.policy.cluster=default\\\",\\\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\\\",\\\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\\\",\\\"k8s:zgroup=testapp\\\"],\\\"pod_name\\\":\\\"app2-6dfff7bfbd-djx8s\\\",\\\"workloads\\\":[{\\\"name\\\":\\\"app2\\\",\\\"kind\\\":\\\"Deployment\\\"}]},\\\"destination\\\":{\\\"ID\\\":3994,\\\"identity\\\":19661,\\\"namespace\\\":\\\"202304202021k8sagenthubbletesthubbleobservetestl3l4flow\\\",\\\"labels\\\":[\\\"k8s:id=app1\\\",\\\"k8s:io.cilium.k8s.policy.cluster=default\\\",\\\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\\\",\\\"k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow\\\",\\\"k8s:zgroup=testapp\\\"],\\\"pod_name\\\":\\\"app1-755788dd65-lxq77\\\",\\\"workloads\\\":[{\\\"name\\\":\\\"app1\\\",\\\"kind\\\":\\\"Deployment\\\"}]},\\\"Type\\\":\\\"L3_L4\\\",\\\"node_n\" as JSON (line 4 of \"kubectl exec -n kube-system cilium-mzq5q -c cilium-agent -- hubble observe --follow --output=jsonpb --last 1 --type trace --from-pod 202304202021k8sagenthubbletesthubbleobservetestl3l4flow/app2-6dfff7bfbd-djx8s --to-namespace 202304202021k8sagenthubbletesthubbleobservetestl3l4flow --to-label id=app1,zgroup=testapp --to-port 80\")",
    }
to be nil
=== Test Finished at 2023-04-20T20:21:15Z====
20:21:15 STEP: Running JustAfterEach block for EntireTestsuite K8sAgentHubbleTest Hubble Observe
===================== TEST FAILED =====================
20:21:16 STEP: Running AfterFailed block for EntireTestsuite K8sAgentHubbleTest Hubble Observe
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE                                                 NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 202304202021k8sagenthubbletesthubbleobservetestl3l4flow   app1-755788dd65-lxq77              2/2     Running   0          15s     10.0.1.173      k8s1   <none>           <none>
	 202304202021k8sagenthubbletesthubbleobservetestl3l4flow   app1-755788dd65-sb7zt              2/2     Running   0          15s     10.0.1.149      k8s1   <none>           <none>
	 202304202021k8sagenthubbletesthubbleobservetestl3l4flow   app2-6dfff7bfbd-djx8s              1/1     Running   0          15s     10.0.1.49       k8s1   <none>           <none>
	 202304202021k8sagenthubbletesthubbleobservetestl3l4flow   app3-68f6b87dd5-6kwhx              1/1     Running   0          15s     10.0.1.158      k8s1   <none>           <none>
	 cilium-monitoring                                         grafana-585bb89877-xcxr9           1/1     Running   0          2m18s   10.0.1.84       k8s1   <none>           <none>
	 cilium-monitoring                                         prometheus-8885c5888-w65sg         1/1     Running   0          2m18s   10.0.1.39       k8s1   <none>           <none>
	 kube-system                                               cilium-mzq5q                       1/1     Running   0          2m17s   192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               cilium-operator-7d9fc97cbb-679xt   1/1     Running   0          2m17s   192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               cilium-operator-7d9fc97cbb-85nn5   1/1     Running   0          2m17s   192.168.56.12   k8s2   <none>           <none>
	 kube-system                                               cilium-z6xv6                       1/1     Running   0          2m17s   192.168.56.12   k8s2   <none>           <none>
	 kube-system                                               coredns-758664cbbf-rp67r           1/1     Running   0          52s     10.0.0.254      k8s2   <none>           <none>
	 kube-system                                               etcd-k8s1                          1/1     Running   0          5m23s   192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               hubble-relay-7ff9f4db57-mf5j7      1/1     Running   0          2m17s   10.0.1.15       k8s1   <none>           <none>
	 kube-system                                               kube-apiserver-k8s1                1/1     Running   0          5m23s   192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               kube-controller-manager-k8s1       1/1     Running   0          5m29s   192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               kube-proxy-h7n7b                   1/1     Running   0          6m7s    192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               kube-proxy-l25vf                   1/1     Running   0          3m6s    192.168.56.12   k8s2   <none>           <none>
	 kube-system                                               kube-scheduler-k8s1                1/1     Running   0          5m39s   192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               log-gatherer-5mhq8                 1/1     Running   0          2m22s   192.168.56.12   k8s2   <none>           <none>
	 kube-system                                               log-gatherer-tjlsm                 1/1     Running   0          2m22s   192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               registry-adder-ls4dx               1/1     Running   0          3m3s    192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               registry-adder-qg8cm               1/1     Running   0          3m3s    192.168.56.12   k8s2   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-mzq5q cilium-z6xv6]
cmd: kubectl exec -n kube-system cilium-mzq5q -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                               IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                                     
	 174        Disabled           Disabled          7471       k8s:appSecond=true                                                                        fd02::161   10.0.1.49    ready   
	                                                            k8s:id=app2                                                                                                                
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app2-account                                                                       
	                                                            k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow                                    
	                                                            k8s:zgroup=testapp                                                                                                         
	 379        Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                         ready   
	                                                            k8s:node-role.kubernetes.io/master                                                                                         
	                                                            reserved:host                                                                                                              
	 2782       Disabled           Disabled          60088      k8s:id=app3                                                                               fd02::132   10.0.1.158   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                            
	                                                            k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow                                    
	                                                            k8s:zgroup=testapp                                                                                                         
	 2889       Disabled           Disabled          43216      k8s:app=grafana                                                                           fd02::141   10.0.1.84    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                            
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                          
	 3251       Disabled           Disabled          43111      k8s:app=prometheus                                                                        fd02::1cb   10.0.1.39    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                                     
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                          
	 3450       Disabled           Disabled          33761      k8s:app.kubernetes.io/name=hubble-relay                                                   fd02::131   10.0.1.15    ready   
	                                                            k8s:app.kubernetes.io/part-of=cilium                                                                                       
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=hubble-relay                                                                       
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                                
	                                                            k8s:k8s-app=hubble-relay                                                                                                   
	 3668       Disabled           Disabled          4          reserved:health                                                                           fd02::13a   10.0.1.110   ready   
	 3889       Disabled           Disabled          19661      k8s:id=app1                                                                               fd02::130   10.0.1.149   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app1-account                                                                       
	                                                            k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow                                    
	                                                            k8s:zgroup=testapp                                                                                                         
	 3994       Disabled           Disabled          19661      k8s:id=app1                                                                               fd02::1f3   10.0.1.173   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app1-account                                                                       
	                                                            k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow                                    
	                                                            k8s:zgroup=testapp                                                                                                         
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-z6xv6 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                       IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                            
	 159        Disabled           Disabled          39404      k8s:io.cilium.k8s.policy.cluster=default          fd02::1    10.0.0.254   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                   
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                       
	                                                            k8s:k8s-app=kube-dns                                                              
	 2326       Disabled           Disabled          4          reserved:health                                   fd02::5a   10.0.0.124   ready   
	 3601       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                ready   
	                                                            reserved:host                                                                     
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
20:21:27 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|e0fc63c3_K8sAgentHubbleTest_Hubble_Observe_Test_L3-L4_Flow.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19//1028/artifact/e0fc63c3_K8sAgentHubbleTest_Hubble_Observe_Test_L3-L4_Flow.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19//1028/artifact/test_results_Cilium-PR-K8s-1.16-kernel-4.19_1028_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/1028/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.
@maintainer-s-little-helper maintainer-s-little-helper bot added the ci/flake This is a known failure that occurs in the tree. Please investigate me! label Apr 23, 2023
Merged
@christarazi
Copy link
Member

The JSON hubble flow in question is

Details 

{"flow":{"time":"2023-04-20T20:21:15.653707777Z","verdict":"FORWARDED","ethernet":{"source":"ca:47:d0:56:4d:b3","destination":"36:52:b0:81:06:a6"},"IP":{"source":"10.0.1.49","destination":"10.0.1.173","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":60870,"destination_port":80,"flags":{"SYN":true}}},"source":{"ID":174,"identity":7471,"namespace":"202304202021k8sagenthubbletesthubbleobservetestl3l4flow","labels":["k8s:appSecond=true","k8s:id=app2","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=app2-account","k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow","k8s:zgroup=testapp"],"pod_name":"app2-6dfff7bfbd-djx8s","workloads":[{"name":"app2","kind":"Deployment"}]},"destination":{"ID":3994,"identity":19661,"namespace":"202304202021k8sagenthubbletesthubbleobservetestl3l4flow","labels":["k8s:id=app1","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=app1-account","k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow","k8s:zgroup=testapp"],"pod_name":"app1-755788dd65-lxq77","workloads":[{"name":"app1","kind":"Deployment"}]},"Type":"L3_L4","node_name":"k8s1","event_type":{"type":4},"traffic_direction":"INGRESS","trace_observation_point":"TO_ENDPOINT","is_reply":false,"interface":{"index":26},"Summary":"TCP Flags: SYN"},"node_name":"k8s1","time":"2023-04-20T20:21:15.653707777Z"}

as far as I can tell, it is valid JSON and both jq and hubble observe parse it correctly, so not sure what went wrong in this CI failure.

@maintainer-s-little-helper
Copy link
Author

PR #24830 hit this flake with 85.93% similarity:

Click to show.

Test Name

K8sAgentHubbleTest Hubble Observe Test L3/L4 Flow

Failure Output

FAIL: hubble observe query timed out on "Exitcode: 0 \nStdout:\n \t {\"flow\":{\"time\":\"2023-04-24T12:05:19.549857985Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"82:76:73:2d:8f:5f\",\"destination\":\"0a:55:a6:ec:5a:65\"},\"IP\":{\"source\":\"10.0.0.252\",\"destination\":\"10.0.0.196\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":35966,\"destination_port\":80,\"flags\":{\"SYN\":true}}},\"source\":{\"ID\":1516,\"identity\":22807,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-q5rnp\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":564,\"identity\":1085,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-dl8l8\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"event_type\":{\"type\":4},\"traffic_direction\":\"INGRESS\",\"trace_observation_point\":\"TO_ENDPOINT\",\"is_reply\":false,\"interface\":{\"index\":26},\"Summary\":\"TCP Flags: SYN\"},\"node_name\":\"k8s1\",\"time\":\"2023-04-24T12:05:19.549857985Z\"}\n\t {\"flow\":{\"time\":\"2023-04-24T12:05:19.549873098Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"82:76:73:2d:8f:5f\",\"destination\":\"0a:55:a6:ec:5a:65\"},\"IP\":{\"source\":\"10.0.0.252\",\"destination\":\"10.0.0.196\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":35966,\"destination_port\":80,\"flags\":{\"ACK\":true}}},\"source\":{\"ID\":1516,\"identity\":22807,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-q5rnp\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":564,\"identity\":1085,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-dl8l8\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"event_type\":{\"type\":4},\"traffic_direction\":\"INGRESS\",\"trace_observation_point\":\"TO_ENDPOINT\",\"is_reply\":false,\"interface\":{\"index\":26},\"Summary\":\"TCP Flags: ACK\"},\"node_name\":\"k8s1\",\"time\":\"2023-04-24T12:05:19.549873098Z\"}\n\t {\"flow\":{\"time\":\"2023-04-24T12:05:19.549934654Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"82:76:73:2d:8f:5f\",\"destination\":\"0a:55:a6:ec:5a:65\"},\"IP\":{\"source\":\"10.0.0.252\",\"destination\":\"10.0.0.196\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":35966,\"destination_port\":80,\"flags\":{\"PSH\":true,\"ACK\":true}}},\"source\":{\"ID\":1516,\"identity\":22807,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-q5rnp\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":564,\"identity\":1085,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-dl8l8\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"\nStderr:\n \t \n"

Stacktrace

Click to show. 
/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
hubble observe query timed out on "Exitcode: 0 \nStdout:\n \t {\"flow\":{\"time\":\"2023-04-24T12:05:19.549857985Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"82:76:73:2d:8f:5f\",\"destination\":\"0a:55:a6:ec:5a:65\"},\"IP\":{\"source\":\"10.0.0.252\",\"destination\":\"10.0.0.196\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":35966,\"destination_port\":80,\"flags\":{\"SYN\":true}}},\"source\":{\"ID\":1516,\"identity\":22807,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-q5rnp\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":564,\"identity\":1085,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-dl8l8\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"event_type\":{\"type\":4},\"traffic_direction\":\"INGRESS\",\"trace_observation_point\":\"TO_ENDPOINT\",\"is_reply\":false,\"interface\":{\"index\":26},\"Summary\":\"TCP Flags: SYN\"},\"node_name\":\"k8s1\",\"time\":\"2023-04-24T12:05:19.549857985Z\"}\n\t {\"flow\":{\"time\":\"2023-04-24T12:05:19.549873098Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"82:76:73:2d:8f:5f\",\"destination\":\"0a:55:a6:ec:5a:65\"},\"IP\":{\"source\":\"10.0.0.252\",\"destination\":\"10.0.0.196\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":35966,\"destination_port\":80,\"flags\":{\"ACK\":true}}},\"source\":{\"ID\":1516,\"identity\":22807,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-q5rnp\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":564,\"identity\":1085,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-dl8l8\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"event_type\":{\"type\":4},\"traffic_direction\":\"INGRESS\",\"trace_observation_point\":\"TO_ENDPOINT\",\"is_reply\":false,\"interface\":{\"index\":26},\"Summary\":\"TCP Flags: ACK\"},\"node_name\":\"k8s1\",\"time\":\"2023-04-24T12:05:19.549873098Z\"}\n\t {\"flow\":{\"time\":\"2023-04-24T12:05:19.549934654Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"82:76:73:2d:8f:5f\",\"destination\":\"0a:55:a6:ec:5a:65\"},\"IP\":{\"source\":\"10.0.0.252\",\"destination\":\"10.0.0.196\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":35966,\"destination_port\":80,\"flags\":{\"PSH\":true,\"ACK\":true}}},\"source\":{\"ID\":1516,\"identity\":22807,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-q5rnp\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":564,\"identity\":1085,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-dl8l8\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"\nStderr:\n \t \n"
Expected
    <*errors.errorString | 0xc000b06be0>: 
    could not parse "{\"flow\":{\"time\":\"2023-04-24T12:05:19.549934654Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"82:76:73:2d:8f:5f\",\"destination\":\"0a:55:a6:ec:5a:65\"},\"IP\":{\"source\":\"10.0.0.252\",\"destination\":\"10.0.0.196\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":35966,\"destination_port\":80,\"flags\":{\"PSH\":true,\"ACK\":true}}},\"source\":{\"ID\":1516,\"identity\":22807,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-q5rnp\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":564,\"identity\":1085,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-dl8l8\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"" as JSON (line 2 of "kubectl exec -n kube-system cilium-bhrs6 -c cilium-agent -- hubble observe --follow --output=jsonpb --last 1 --type trace --from-pod 202304241205k8sagenthubbletesthubbleobservetestl3l4flow/app2-6dfff7bfbd-q5rnp --to-namespace 202304241205k8sagenthubbletesthubbleobservetestl3l4flow --to-label id=app1,zgroup=testapp --to-port 80")
    {
        s: "could not parse \"{\\\"flow\\\":{\\\"time\\\":\\\"2023-04-24T12:05:19.549934654Z\\\",\\\"verdict\\\":\\\"FORWARDED\\\",\\\"ethernet\\\":{\\\"source\\\":\\\"82:76:73:2d:8f:5f\\\",\\\"destination\\\":\\\"0a:55:a6:ec:5a:65\\\"},\\\"IP\\\":{\\\"source\\\":\\\"10.0.0.252\\\",\\\"destination\\\":\\\"10.0.0.196\\\",\\\"ipVersion\\\":\\\"IPv4\\\"},\\\"l4\\\":{\\\"TCP\\\":{\\\"source_port\\\":35966,\\\"destination_port\\\":80,\\\"flags\\\":{\\\"PSH\\\":true,\\\"ACK\\\":true}}},\\\"source\\\":{\\\"ID\\\":1516,\\\"identity\\\":22807,\\\"namespace\\\":\\\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\\\",\\\"labels\\\":[\\\"k8s:appSecond=true\\\",\\\"k8s:id=app2\\\",\\\"k8s:io.cilium.k8s.policy.cluster=default\\\",\\\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\\\",\\\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\\\",\\\"k8s:zgroup=testapp\\\"],\\\"pod_name\\\":\\\"app2-6dfff7bfbd-q5rnp\\\",\\\"workloads\\\":[{\\\"name\\\":\\\"app2\\\",\\\"kind\\\":\\\"Deployment\\\"}]},\\\"destination\\\":{\\\"ID\\\":564,\\\"identity\\\":1085,\\\"namespace\\\":\\\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\\\",\\\"labels\\\":[\\\"k8s:id=app1\\\",\\\"k8s:io.cilium.k8s.policy.cluster=default\\\",\\\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\\\",\\\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\\\",\\\"k8s:zgroup=testapp\\\"],\\\"pod_name\\\":\\\"app1-755788dd65-dl8l8\\\",\\\"workloads\\\":[{\\\"name\\\":\\\"app1\\\",\\\"kind\\\":\\\"Deployment\\\"}]},\\\"Type\\\":\\\"L3_L4\\\",\\\"node_name\\\":\\\"k8s1\\\",\\\"\" as JSON (line 2 of \"kubectl exec -n kube-system cilium-bhrs6 -c cilium-agent -- hubble observe --follow --output=jsonpb --last 1 --type trace --from-pod 202304241205k8sagenthubbletesthubbleobservetestl3l4flow/app2-6dfff7bfbd-q5rnp --to-namespace 202304241205k8sagenthubbletesthubbleobservetestl3l4flow --to-label id=app1,zgroup=testapp --to-port 80\")",
    }
to be nil
/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19/src/github.com/cilium/cilium/test/k8s/hubble.go:191

Standard Output

Click to show. 
Number of "context deadline exceeded" in logs: 1
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 1
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 1 errors/warnings:
Failed to create peer client for peers synchronization; will try again after the timeout has expired
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
⚠️  Number of "level=warning" in logs: 7
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 5 errors/warnings:
Unable to get node resource
Waiting for k8s node information
UpdateIdentities: Skipping Delete of a non-existing identity
Key allocation attempt failed
Attempt to remove non-existing IP from ipcache layer
Cilium pods: [cilium-255gq cilium-bhrs6]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                             Ingress   Egress
coredns-758664cbbf-qgvzt        false     false
hubble-relay-7c95769877-ns8xn   false     false
app1-755788dd65-dl8l8           false     false
app1-755788dd65-sgvxf           false     false
app2-6dfff7bfbd-q5rnp           false     false
app3-68f6b87dd5-g8rl9           false     false
grafana-585bb89877-m9cdl        false     false
prometheus-8885c5888-6lcp7      false     false
Cilium agent 'cilium-255gq': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 24 Failed 0
Cilium agent 'cilium-bhrs6': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 48 Failed 0

Standard Error

Click to show. 
12:03:10 STEP: Running BeforeAll block for EntireTestsuite
12:03:10 STEP: Starting tests: command line parameters: {Reprovision:false HoldEnvironment:false PassCLIEnvironment:true SSHConfig: ShowCommands:false TestScope: SkipLogGathering:false CiliumImage:quay.io/cilium/cilium-ci CiliumTag:b796c01c83cac412005b71e44b01e25f28b89137 CiliumOperatorImage:quay.io/cilium/operator CiliumOperatorTag:b796c01c83cac412005b71e44b01e25f28b89137 CiliumOperatorSuffix:-ci HubbleRelayImage:quay.io/cilium/hubble-relay-ci HubbleRelayTag:b796c01c83cac412005b71e44b01e25f28b89137 ProvisionK8s:true Timeout:2h50m0s Kubeconfig:/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19/src/github.com/cilium/cilium/test/vagrant-kubeconfig KubectlPath:/tmp/kubectl RegistryCredentials: Multinode:true RunQuarantined:false Help:false} environment variables: [JENKINS_HOME=/var/jenkins_home ghprbSourceBranch=pr/mhofstetter/hive-config-map VM_MEMORY=8192 MAIL=/var/mail/root SSH_CLIENT=54.148.123.155 41666 22 USER=root PROJ_PATH=src/github.com/cilium/cilium RUN_CHANGES_DISPLAY_URL=https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/1081/display/redirect?page=changes ghprbPullDescription=GitHub pull request #24830 of commit b796c01c83cac412005b71e44b01e25f28b89137, no merge conflicts. NETNEXT=0 ghprbActualCommit=b796c01c83cac412005b71e44b01e25f28b89137 SHLVL=1 CILIUM_TAG=b796c01c83cac412005b71e44b01e25f28b89137 NODE_LABELS=baremetal ginkgo nightly node-nice-quagga vagrant HUDSON_URL=https://jenkins.cilium.io/ GIT_COMMIT=f9da721214fa735c472b9cc851819db55f0edb87 OLDPWD=/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19 GINKGO_TIMEOUT=170m HOME=/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19 INTEGRATION_TESTS=true ghprbTriggerAuthorLoginMention=@mhofstetter BUILD_URL=https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/1081/ ghprbPullAuthorLoginMention=@mhofstetter HUDSON_COOKIE=ae651116-ca9f-4f85-937f-c17e37c12d69 JENKINS_SERVER_COOKIE=durable-963ac66a0522387382e0af9c3b4bb35c ghprbGhRepository=cilium/cilium DOCKER_TAG=b796c01c83cac412005b71e44b01e25f28b89137 JobKernelVersion=419 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus KERNEL=419 CONTAINER_RUNTIME=docker WORKSPACE=/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19 ghprbPullLongDescription=Currently, the interaction with the BPF map configmap and the corresponding utime sync is mainly via global accessors.\r\n\r\nThis commit refactors this, by introducing hive cells for the configmap and the utime sync. K8S_NODES=2 TESTDIR=/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19/src/github.com/cilium/cilium/test LOGNAME=root NODE_NAME=node-nice-quagga ghprbCredentialsId=ciliumbot _=/usr/bin/java HUBBLE_RELAY_IMAGE=quay.io/cilium/hubble-relay-ci STAGE_NAME=BDD-Test-PR GIT_BRANCH=origin/pr/24830/merge EXECUTOR_NUMBER=0 ghprbTriggerAuthorLogin=mhofstetter TERM=xterm XDG_SESSION_ID=5 HOST_FIREWALL=0 CILIUM_OPERATOR_TAG=b796c01c83cac412005b71e44b01e25f28b89137 BUILD_DISPLAY_NAME=configmap & utime sync: provide via hive cell  https://github.com/cilium/cilium/pull/24830  #1081 ghprbPullAuthorLogin=mhofstetter HUDSON_HOME=/var/jenkins_home ghprbTriggerAuthor=Marco Hofstetter JOB_BASE_NAME=Cilium-PR-K8s-1.16-kernel-4.19 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/go/bin:/root/go/bin sha1=origin/pr/24830/merge KUBECONFIG=/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19/src/github.com/cilium/cilium/test/vagrant-kubeconfig FOCUS=K8s BUILD_ID=1081 XDG_RUNTIME_DIR=/run/user/0 BUILD_TAG=jenkins-Cilium-PR-K8s-1.16-kernel-4.19-1081 RUN_QUARANTINED=false CILIUM_IMAGE=quay.io/cilium/cilium-ci JENKINS_URL=https://jenkins.cilium.io/ LANG=C.UTF-8 ghprbCommentBody=/test JOB_URL=https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/ ghprbPullTitle=configmap & utime sync: provide via hive cell GIT_URL=https://github.com/cilium/cilium ghprbPullLink=https://github.com/cilium/cilium/pull/24830 BUILD_NUMBER=1081 JENKINS_NODE_COOKIE=4bb16b14-27ff-44dd-baeb-eada5376d25f SHELL=/bin/bash GOPATH=/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19 RUN_DISPLAY_URL=https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/1081/display/redirect IMAGE_REGISTRY=quay.io/cilium ghprbAuthorRepoGitUrl=https://github.com/mhofstetter/cilium.git FAILFAST=false HUDSON_SERVER_COOKIE=693c250bfb7e85bf ghprbTargetBranch=main JOB_DISPLAY_URL=https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/display/redirect K8S_VERSION=1.16 JOB_NAME=Cilium-PR-K8s-1.16-kernel-4.19 SSH_CONNECTION=54.148.123.155 41666 145.40.77.233 22 ghprbPullId=24830 CILIUM_OPERATOR_IMAGE=quay.io/cilium/operator HUBBLE_RELAY_TAG=b796c01c83cac412005b71e44b01e25f28b89137 JobK8sVersion=1.16 VM_CPUS=3 PWD=/home/jenkins/workspace/Cilium-PR-K8s-1.16-kernel-4.19/src/github.com/cilium/cilium/test CILIUM_OPERATOR_SUFFIX=-ci]
12:03:10 STEP: Ensuring the namespace kube-system exists
12:03:10 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
12:03:25 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
12:03:25 STEP: Preparing cluster
12:03:25 STEP: Labelling nodes
12:03:26 STEP: Cleaning up Cilium components
12:03:26 STEP: Running BeforeAll block for EntireTestsuite K8sAgentHubbleTest Hubble Observe
12:03:26 STEP: Ensuring the namespace kube-system exists
12:03:26 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
12:03:26 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
12:03:27 STEP: Installing Cilium
12:03:28 STEP: Waiting for Cilium to become ready
12:04:38 STEP: Restarting unmanaged pods coredns-758664cbbf-9wjjp in namespace kube-system
12:04:49 STEP: Validating if Kubernetes DNS is deployed
12:04:49 STEP: Checking if deployment is ready
12:04:49 STEP: Checking if kube-dns service is plumbed correctly
12:04:49 STEP: Checking if pods have identity
12:04:49 STEP: Checking if DNS can resolve
12:04:52 STEP: Kubernetes DNS is up and operational
12:04:52 STEP: Validating Cilium Installation
12:04:52 STEP: Performing Cilium controllers preflight check
12:04:52 STEP: Performing Cilium health check
12:04:52 STEP: Checking whether host EP regenerated
12:04:52 STEP: Performing Cilium status preflight check
12:05:00 STEP: Performing Cilium service preflight check
12:05:00 STEP: Performing K8s service preflight check
12:05:06 STEP: Waiting for cilium-operator to be ready
12:05:06 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator")
12:05:06 STEP: WaitforPods(namespace="kube-system", filter="-l name=cilium-operator") => <nil>
12:05:06 STEP: Waiting for hubble-relay to be ready
12:05:06 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=hubble-relay")
12:05:06 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=hubble-relay") => <nil>
12:05:06 STEP: Deleting namespace 202304241205k8sagenthubbletesthubbleobservetestl3l4flow
12:05:06 STEP: Creating namespace 202304241205k8sagenthubbletesthubbleobservetestl3l4flow
12:05:07 STEP: WaitforPods(namespace="202304241205k8sagenthubbletesthubbleobservetestl3l4flow", filter="-l zgroup=testapp")
12:05:19 STEP: WaitforPods(namespace="202304241205k8sagenthubbletesthubbleobservetestl3l4flow", filter="-l zgroup=testapp") => <nil>
FAIL: hubble observe query timed out on "Exitcode: 0 \nStdout:\n \t {\"flow\":{\"time\":\"2023-04-24T12:05:19.549857985Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"82:76:73:2d:8f:5f\",\"destination\":\"0a:55:a6:ec:5a:65\"},\"IP\":{\"source\":\"10.0.0.252\",\"destination\":\"10.0.0.196\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":35966,\"destination_port\":80,\"flags\":{\"SYN\":true}}},\"source\":{\"ID\":1516,\"identity\":22807,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-q5rnp\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":564,\"identity\":1085,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-dl8l8\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"event_type\":{\"type\":4},\"traffic_direction\":\"INGRESS\",\"trace_observation_point\":\"TO_ENDPOINT\",\"is_reply\":false,\"interface\":{\"index\":26},\"Summary\":\"TCP Flags: SYN\"},\"node_name\":\"k8s1\",\"time\":\"2023-04-24T12:05:19.549857985Z\"}\n\t {\"flow\":{\"time\":\"2023-04-24T12:05:19.549873098Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"82:76:73:2d:8f:5f\",\"destination\":\"0a:55:a6:ec:5a:65\"},\"IP\":{\"source\":\"10.0.0.252\",\"destination\":\"10.0.0.196\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":35966,\"destination_port\":80,\"flags\":{\"ACK\":true}}},\"source\":{\"ID\":1516,\"identity\":22807,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-q5rnp\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":564,\"identity\":1085,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-dl8l8\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"event_type\":{\"type\":4},\"traffic_direction\":\"INGRESS\",\"trace_observation_point\":\"TO_ENDPOINT\",\"is_reply\":false,\"interface\":{\"index\":26},\"Summary\":\"TCP Flags: ACK\"},\"node_name\":\"k8s1\",\"time\":\"2023-04-24T12:05:19.549873098Z\"}\n\t {\"flow\":{\"time\":\"2023-04-24T12:05:19.549934654Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"82:76:73:2d:8f:5f\",\"destination\":\"0a:55:a6:ec:5a:65\"},\"IP\":{\"source\":\"10.0.0.252\",\"destination\":\"10.0.0.196\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":35966,\"destination_port\":80,\"flags\":{\"PSH\":true,\"ACK\":true}}},\"source\":{\"ID\":1516,\"identity\":22807,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-q5rnp\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":564,\"identity\":1085,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-dl8l8\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"\nStderr:\n \t \n"
Expected
    <*errors.errorString | 0xc000b06be0>: 
    could not parse "{\"flow\":{\"time\":\"2023-04-24T12:05:19.549934654Z\",\"verdict\":\"FORWARDED\",\"ethernet\":{\"source\":\"82:76:73:2d:8f:5f\",\"destination\":\"0a:55:a6:ec:5a:65\"},\"IP\":{\"source\":\"10.0.0.252\",\"destination\":\"10.0.0.196\",\"ipVersion\":\"IPv4\"},\"l4\":{\"TCP\":{\"source_port\":35966,\"destination_port\":80,\"flags\":{\"PSH\":true,\"ACK\":true}}},\"source\":{\"ID\":1516,\"identity\":22807,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:appSecond=true\",\"k8s:id=app2\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app2-6dfff7bfbd-q5rnp\",\"workloads\":[{\"name\":\"app2\",\"kind\":\"Deployment\"}]},\"destination\":{\"ID\":564,\"identity\":1085,\"namespace\":\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"labels\":[\"k8s:id=app1\",\"k8s:io.cilium.k8s.policy.cluster=default\",\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\",\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\",\"k8s:zgroup=testapp\"],\"pod_name\":\"app1-755788dd65-dl8l8\",\"workloads\":[{\"name\":\"app1\",\"kind\":\"Deployment\"}]},\"Type\":\"L3_L4\",\"node_name\":\"k8s1\",\"" as JSON (line 2 of "kubectl exec -n kube-system cilium-bhrs6 -c cilium-agent -- hubble observe --follow --output=jsonpb --last 1 --type trace --from-pod 202304241205k8sagenthubbletesthubbleobservetestl3l4flow/app2-6dfff7bfbd-q5rnp --to-namespace 202304241205k8sagenthubbletesthubbleobservetestl3l4flow --to-label id=app1,zgroup=testapp --to-port 80")
    {
        s: "could not parse \"{\\\"flow\\\":{\\\"time\\\":\\\"2023-04-24T12:05:19.549934654Z\\\",\\\"verdict\\\":\\\"FORWARDED\\\",\\\"ethernet\\\":{\\\"source\\\":\\\"82:76:73:2d:8f:5f\\\",\\\"destination\\\":\\\"0a:55:a6:ec:5a:65\\\"},\\\"IP\\\":{\\\"source\\\":\\\"10.0.0.252\\\",\\\"destination\\\":\\\"10.0.0.196\\\",\\\"ipVersion\\\":\\\"IPv4\\\"},\\\"l4\\\":{\\\"TCP\\\":{\\\"source_port\\\":35966,\\\"destination_port\\\":80,\\\"flags\\\":{\\\"PSH\\\":true,\\\"ACK\\\":true}}},\\\"source\\\":{\\\"ID\\\":1516,\\\"identity\\\":22807,\\\"namespace\\\":\\\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\\\",\\\"labels\\\":[\\\"k8s:appSecond=true\\\",\\\"k8s:id=app2\\\",\\\"k8s:io.cilium.k8s.policy.cluster=default\\\",\\\"k8s:io.cilium.k8s.policy.serviceaccount=app2-account\\\",\\\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\\\",\\\"k8s:zgroup=testapp\\\"],\\\"pod_name\\\":\\\"app2-6dfff7bfbd-q5rnp\\\",\\\"workloads\\\":[{\\\"name\\\":\\\"app2\\\",\\\"kind\\\":\\\"Deployment\\\"}]},\\\"destination\\\":{\\\"ID\\\":564,\\\"identity\\\":1085,\\\"namespace\\\":\\\"202304241205k8sagenthubbletesthubbleobservetestl3l4flow\\\",\\\"labels\\\":[\\\"k8s:id=app1\\\",\\\"k8s:io.cilium.k8s.policy.cluster=default\\\",\\\"k8s:io.cilium.k8s.policy.serviceaccount=app1-account\\\",\\\"k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow\\\",\\\"k8s:zgroup=testapp\\\"],\\\"pod_name\\\":\\\"app1-755788dd65-dl8l8\\\",\\\"workloads\\\":[{\\\"name\\\":\\\"app1\\\",\\\"kind\\\":\\\"Deployment\\\"}]},\\\"Type\\\":\\\"L3_L4\\\",\\\"node_name\\\":\\\"k8s1\\\",\\\"\" as JSON (line 2 of \"kubectl exec -n kube-system cilium-bhrs6 -c cilium-agent -- hubble observe --follow --output=jsonpb --last 1 --type trace --from-pod 202304241205k8sagenthubbletesthubbleobservetestl3l4flow/app2-6dfff7bfbd-q5rnp --to-namespace 202304241205k8sagenthubbletesthubbleobservetestl3l4flow --to-label id=app1,zgroup=testapp --to-port 80\")",
    }
to be nil
=== Test Finished at 2023-04-24T12:05:19Z====
12:05:19 STEP: Running JustAfterEach block for EntireTestsuite K8sAgentHubbleTest Hubble Observe
===================== TEST FAILED =====================
12:05:19 STEP: Running AfterFailed block for EntireTestsuite K8sAgentHubbleTest Hubble Observe
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE                                                 NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 202304241205k8sagenthubbletesthubbleobservetestl3l4flow   app1-755788dd65-dl8l8              2/2     Running   0          17s     10.0.0.196      k8s1   <none>           <none>
	 202304241205k8sagenthubbletesthubbleobservetestl3l4flow   app1-755788dd65-sgvxf              2/2     Running   0          17s     10.0.0.191      k8s1   <none>           <none>
	 202304241205k8sagenthubbletesthubbleobservetestl3l4flow   app2-6dfff7bfbd-q5rnp              1/1     Running   0          17s     10.0.0.252      k8s1   <none>           <none>
	 202304241205k8sagenthubbletesthubbleobservetestl3l4flow   app3-68f6b87dd5-g8rl9              1/1     Running   0          17s     10.0.0.50       k8s1   <none>           <none>
	 cilium-monitoring                                         grafana-585bb89877-m9cdl           1/1     Running   0          118s    10.0.0.137      k8s1   <none>           <none>
	 cilium-monitoring                                         prometheus-8885c5888-6lcp7         1/1     Running   0          118s    10.0.0.54       k8s1   <none>           <none>
	 kube-system                                               cilium-255gq                       1/1     Running   0          116s    192.168.56.12   k8s2   <none>           <none>
	 kube-system                                               cilium-bhrs6                       1/1     Running   0          116s    192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               cilium-operator-789bfff5c4-dmpsl   1/1     Running   0          116s    192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               cilium-operator-789bfff5c4-v5p6t   1/1     Running   0          116s    192.168.56.12   k8s2   <none>           <none>
	 kube-system                                               coredns-758664cbbf-qgvzt           1/1     Running   0          46s     10.0.1.240      k8s2   <none>           <none>
	 kube-system                                               etcd-k8s1                          1/1     Running   0          5m22s   192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               hubble-relay-7c95769877-ns8xn      1/1     Running   0          116s    10.0.0.138      k8s1   <none>           <none>
	 kube-system                                               kube-apiserver-k8s1                1/1     Running   0          5m4s    192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               kube-controller-manager-k8s1       1/1     Running   0          5m13s   192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               kube-proxy-qkkv4                   1/1     Running   0          2m59s   192.168.56.12   k8s2   <none>           <none>
	 kube-system                                               kube-proxy-qp6lc                   1/1     Running   0          5m55s   192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               kube-scheduler-k8s1                1/1     Running   0          5m20s   192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               log-gatherer-6kbf9                 1/1     Running   0          2m14s   192.168.56.12   k8s2   <none>           <none>
	 kube-system                                               log-gatherer-tfbsd                 1/1     Running   0          2m14s   192.168.56.11   k8s1   <none>           <none>
	 kube-system                                               registry-adder-j2dl8               1/1     Running   0          2m57s   192.168.56.12   k8s2   <none>           <none>
	 kube-system                                               registry-adder-nn8sf               1/1     Running   0          2m57s   192.168.56.11   k8s1   <none>           <none>
	 
Stderr:
 	 

Fetching command output from pods [cilium-255gq cilium-bhrs6]
cmd: kubectl exec -n kube-system cilium-255gq -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                       IPv6        IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                             
	 119        Disabled           Disabled          41884      k8s:io.cilium.k8s.policy.cluster=default          fd02::16d   10.0.1.240   ready   
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                    
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                        
	                                                            k8s:k8s-app=kube-dns                                                               
	 769        Disabled           Disabled          4          reserved:health                                   fd02::159   10.0.1.99    ready   
	 2251       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s2                                                 ready   
	                                                            reserved:host                                                                      
	 
Stderr:
 	 

cmd: kubectl exec -n kube-system cilium-bhrs6 -c cilium-agent -- cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                               IPv6       IPv4         STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                                                                    
	 495        Disabled           Disabled          1085       k8s:id=app1                                                                               fd02::f    10.0.0.191   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app1-account                                                                      
	                                                            k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow                                   
	                                                            k8s:zgroup=testapp                                                                                                        
	 564        Disabled           Disabled          1085       k8s:id=app1                                                                               fd02::53   10.0.0.196   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app1-account                                                                      
	                                                            k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow                                   
	                                                            k8s:zgroup=testapp                                                                                                        
	 707        Disabled           Disabled          24767      k8s:app=prometheus                                                                        fd02::76   10.0.0.54    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s                                                                    
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                         
	 887        Disabled           Disabled          35361      k8s:id=app3                                                                               fd02::bf   10.0.0.50    ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                           
	                                                            k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow                                   
	                                                            k8s:zgroup=testapp                                                                                                        
	 1358       Disabled           Disabled          53085      k8s:app=grafana                                                                           fd02::69   10.0.0.137   ready   
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                                           
	                                                            k8s:io.kubernetes.pod.namespace=cilium-monitoring                                                                         
	 1516       Disabled           Disabled          22807      k8s:appSecond=true                                                                        fd02::9e   10.0.0.252   ready   
	                                                            k8s:id=app2                                                                                                               
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=app2-account                                                                      
	                                                            k8s:io.kubernetes.pod.namespace=202304241205k8sagenthubbletesthubbleobservetestl3l4flow                                   
	                                                            k8s:zgroup=testapp                                                                                                        
	 1717       Disabled           Disabled          43495      k8s:app.kubernetes.io/name=hubble-relay                                                   fd02::8f   10.0.0.138   ready   
	                                                            k8s:app.kubernetes.io/part-of=cilium                                                                                      
	                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                                  
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=hubble-relay                                                                      
	                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                                               
	                                                            k8s:k8s-app=hubble-relay                                                                                                  
	 1841       Disabled           Disabled          4          reserved:health                                                                           fd02::d3   10.0.0.141   ready   
	 3099       Disabled           Disabled          1          k8s:cilium.io/ci-node=k8s1                                                                                        ready   
	                                                            k8s:node-role.kubernetes.io/master                                                                                        
	                                                            reserved:host                                                                                                             
	 
Stderr:
 	 

===================== Exiting AfterFailed =====================
12:05:31 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|2cc57028_K8sAgentHubbleTest_Hubble_Observe_Test_L3-L4_Flow.zip]]

ZIP Links:

Click to show.

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19//1081/artifact/2cc57028_K8sAgentHubbleTest_Hubble_Observe_Test_L3-L4_Flow.zip
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19//1081/artifact/test_results_Cilium-PR-K8s-1.16-kernel-4.19_1081_BDD-Test-PR.zip

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-kernel-4.19/1081/

If this is a duplicate of an existing flake, comment 'Duplicate of #<issue-number>' and close this issue.

Merged
@gandro
Copy link
Member

gandro commented Apr 25, 2023 •
edited

The JSON hubble flow in question is
Details

{"flow":{"time":"2023-04-20T20:21:15.653707777Z","verdict":"FORWARDED","ethernet":{"source":"ca:47:d0:56:4d:b3","destination":"36:52:b0:81:06:a6"},"IP":{"source":"10.0.1.49","destination":"10.0.1.173","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":60870,"destination_port":80,"flags":{"SYN":true}}},"source":{"ID":174,"identity":7471,"namespace":"202304202021k8sagenthubbletesthubbleobservetestl3l4flow","labels":["k8s:appSecond=true","k8s:id=app2","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=app2-account","k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow","k8s:zgroup=testapp"],"pod_name":"app2-6dfff7bfbd-djx8s","workloads":[{"name":"app2","kind":"Deployment"}]},"destination":{"ID":3994,"identity":19661,"namespace":"202304202021k8sagenthubbletesthubbleobservetestl3l4flow","labels":["k8s:id=app1","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=app1-account","k8s:io.kubernetes.pod.namespace=202304202021k8sagenthubbletesthubbleobservetestl3l4flow","k8s:zgroup=testapp"],"pod_name":"app1-755788dd65-lxq77","workloads":[{"name":"app1","kind":"Deployment"}]},"Type":"L3_L4","node_name":"k8s1","event_type":{"type":4},"traffic_direction":"INGRESS","trace_observation_point":"TO_ENDPOINT","is_reply":false,"interface":{"index":26},"Summary":"TCP Flags: SYN"},"node_name":"k8s1","time":"2023-04-20T20:21:15.653707777Z"}

as far as I can tell, it is valid JSON and both jq and hubble observe parse it correctly, so not sure what went wrong in this CI failure.

I think the error in question is the following:

could not parse '{\"flow\":...\"Type\":\"L3_L4\",\"node_n' as JSON (line 4 of "kubectl exec -n kube-system ...

It looks like the kubectl output is cut off, notice the node_n at the end (which likely should be node_name, followed by av value). It's not clear yet who is truncating the output, is it Hubble or is it kubectl?

@github-actions
Copy link

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Jun 25, 2023
@github-actions
Copy link

This issue has not seen any activity since it was marked stale.
Closing.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jul 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ci/flake This is a known failure that occurs in the tree. Please investigate me! stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gandro @christarazi