Metadata missing when getting L7 flow in hubble

[mrproliu]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

I have a simple Java program trying to get the L7 flow data within the example application, but I found some flow missing the pod name and namespace name. Here are the steps:

1. Create a pure GKE project.
2. Installing the cilium with following command(Using none TLS for Hubble can help me easy to debugging the cilium data):

NATIVE_CIDR="$(gcloud container clusters describe "xxx" --zone "xxx" --format 'value(clusterIpv4Cidr)')"
echo $NATIVE_CIDR
helm install cilium cilium/cilium --version 1.15.4 \
  --namespace kube-system \
  --set nodeinit.enabled=true \
  --set nodeinit.reconfigureKubelet=true \
  --set nodeinit.removeCbrBridge=true \
  --set cni.binPath=/home/kubernetes/bin \
  --set gke.enabled=true \
  --set ipam.mode=kubernetes \
  --set ipv4NativeRoutingCIDR=$NATIVE_CIDR \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set hubble.tls.enabled=false \
  --set operator.replicas=1 

3. Deploy the demo application with the following command:

k create ns starwar
kubectl create -n starwar -f https://raw.githubusercontent.com/cilium/cilium/1.15.4/examples/minikube/http-sw-app.yaml

4. Adding the Cilium Network Policy for getting L7 data in the Hubble flow(looks like we must define CNP otherwise I cannot getting L7 flow data, Do we have a simple way? In a huge cluster, defined CNP it's a big work.)

k apply -n starwar -f - <<EOF
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "rule1"
spec:
  description: "L7 policy to restrict access to specific HTTP call"
  endpointSelector:
    matchLabels:
      org: empire
      class: deathstar
  ingress:
  - fromEndpoints:
    - matchLabels:
        org: empire
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
      rules:
        http:
        - method: "POST"
          path: "/v1/request-landing"
EOF

k apply -n starwar -f - <<EOF
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "rule2"
spec:
  description: "L7 policy to restrict access to specific HTTP call"
  endpointSelector:
    matchLabels:
      org: empire
      class: tiefighter
  egress:
  - toEndpoints:
    - matchLabels:
        org: empire
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
      rules:
        http:
        - method: "POST"
          path: "/v1/request-landing"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": kube-system
        "k8s:k8s-app": kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
EOF

5. Port forward the cilium proxy for getting flow data locally: k port-forward -n kube-system kube-proxy-gke-han-cilium-demo-default-pool-41a91221-gc5f 4244
6. Write a simple Java program for get the flows data through API with the following mode(I just copy the related protocol files and using protoc to compile with the Java code):

GRPCClient client = new GRPCClient("127.0.0.1", 4244);
client.connect();
ManagedChannel channel = client.getChannel();
ObserverGrpc.ObserverBlockingStub stub = ObserverGrpc.newBlockingStub(channel);
final Iterator<GetFlowsResponse> flows = stub.getFlows(GetFlowsRequest.newBuilder()
        .setFollow(true)
.build());
while (flows.hasNext()) {
        final GetFlowsResponse next = flows.next();
        System.out.println(next);
}

7. Send a request from tiefighter application: kubectl exec -n starwar tiefighter -- curl -s -XPOST deathstar.starwar.svc.cluster.local/v1/request-landing
8. Finally, I can see a flow missing the pod name and namespace data in this screenshot:
   

Here in other screenshots to help you get more context information:

1. Getting the flow data by the original command(hubble observe -f), you can see this endpoint only has the IP address in some flow(not all).
   

2. Here is the pod IP address list through kubectl get pods -A -o wide:
   

3. Here is the endpoint information through the command: kubectl -n kube-system exec cilium-vmlpq -- cilium-dbg endpoint list:
   

Cilium Version

cilium-cli: v0.15.0 compiled with go1.20.4 on darwin/arm64
cilium image (default): v1.13.4
cilium image (stable): v1.15.3
cilium image (running): 1.15.4

Kernel Version

Linux gke-han-cilium-demo-default-pool-41a91221-gc5f 6.1.58+ #1 SMP PREEMPT_DYNAMIC Mon Jan 29 15:19:25 UTC 2024 x86_64 Intel(R) Xeon(R) CPU @ 2.20GHz GenuineIntel GNU/Linux

Kubernetes Version

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.7-gke.1026000

Regression

No response

Sysdump

cilium-sysdump-20240425-104918.zip

Relevant log output

No response

Anything else?

If you want to know more details from Slack, you can check this out.

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct