Hubble 0.7.3 is missing http and dns metrics

[MaciejGawel]

Description

I have deployed Cilium 1.9.0 with Hubble 0.7.3. To verify network connectivity and generate some network traffic, I am using a simple example service.

Querying Hubble metrics endpoint returns the following content:

# HELP hubble_drop_total Number of drops
# TYPE hubble_drop_total counter
hubble_drop_total{protocol="ICMPv4",reason="Stale or unroutable IP"} 1
hubble_drop_total{protocol="ICMPv4",reason="Unsupported protocol for NAT masquerade"} 2
hubble_drop_total{protocol="TCP",reason="Stale or unroutable IP"} 7
# HELP hubble_flows_processed_total Total number of flows processed
# TYPE hubble_flows_processed_total counter
hubble_flows_processed_total{protocol="ICMPv4",subtype="",type="Drop",verdict="DROPPED"} 3
hubble_flows_processed_total{protocol="ICMPv4",subtype="to-endpoint",type="Trace",verdict="FORWARDED"} 39
hubble_flows_processed_total{protocol="ICMPv4",subtype="to-overlay",type="Trace",verdict="FORWARDED"} 43
hubble_flows_processed_total{protocol="ICMPv4",subtype="to-stack",type="Trace",verdict="FORWARDED"} 19
hubble_flows_processed_total{protocol="TCP",subtype="",type="Drop",verdict="DROPPED"} 7
hubble_flows_processed_total{protocol="TCP",subtype="to-endpoint",type="Trace",verdict="FORWARDED"} 17144
hubble_flows_processed_total{protocol="TCP",subtype="to-overlay",type="Trace",verdict="FORWARDED"} 12558
hubble_flows_processed_total{protocol="TCP",subtype="to-stack",type="Trace",verdict="FORWARDED"} 6985
hubble_flows_processed_total{protocol="UDP",subtype="to-endpoint",type="Trace",verdict="FORWARDED"} 10901
hubble_flows_processed_total{protocol="UDP",subtype="to-overlay",type="Trace",verdict="FORWARDED"} 6471
# HELP hubble_icmp_total Number of ICMP messages
# TYPE hubble_icmp_total counter
hubble_icmp_total{family="IPv4",type="EchoReply"} 39
hubble_icmp_total{family="IPv4",type="EchoRequest"} 61
hubble_icmp_total{family="IPv4",type="TimeExceeded(TTLExceeded)"} 4
# HELP hubble_port_distribution_total Numbers of packets distributed by destination port
# TYPE hubble_port_distribution_total counter
hubble_port_distribution_total{port="0",protocol="ICMPv4"} 39
hubble_port_distribution_total{port="2379",protocol="TCP"} 178
hubble_port_distribution_total{port="2380",protocol="TCP"} 3970
hubble_port_distribution_total{port="4240",protocol="TCP"} 180
hubble_port_distribution_total{port="4244",protocol="TCP"} 1264
hubble_port_distribution_total{port="4245",protocol="TCP"} 919
hubble_port_distribution_total{port="53",protocol="UDP"} 6398
hubble_port_distribution_total{port="6443",protocol="TCP"} 430
hubble_port_distribution_total{port="80",protocol="TCP"} 6109
hubble_port_distribution_total{port="8080",protocol="TCP"} 519
hubble_port_distribution_total{port="8181",protocol="TCP"} 493
# HELP hubble_tcp_flags_total TCP flag occurrences
# TYPE hubble_tcp_flags_total counter
hubble_tcp_flags_total{family="IPv4",flag="FIN"} 5707
hubble_tcp_flags_total{family="IPv4",flag="RST"} 2634
hubble_tcp_flags_total{family="IPv4",flag="SYN"} 3415
hubble_tcp_flags_total{family="IPv4",flag="SYN-ACK"} 3379

According to the documentation, http and dns metrics should also be returned.

Am I missing something in the configuration?

Expected behavior

The following metrics should be returned:

• dns
  • dns_queries_total
  • dns_responses_total
  • dns_response_types_total
• http
  • http_requests_total
  • http_responses_total
  • http_request_duration_seconds

Additional context

• Self-hosted Kubernetes v1.18.2 (Centos 8)
• Cilium 1.9.0
• Hubble 0.7.3

To reproduce

helm install cilium cilium/cilium --version 1.9.0 \
   --namespace kube-system \
   --set etcd.enabled=true \
   --set etcd.managed=true \
   --set identityAllocationMode=kvstore \
   --set hubble.enabled=true \
   --set hubble.listenAddress=":4244" \
   --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}" \
   --set hubble.relay.enabled=true \
   --set hubble.ui.enabled=true \
   --set prometheus.enabled=true \
   --set operator.prometheus.enabled=true

kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.9/examples/kubernetes/clustermesh/global-service-example/cluster1.yaml

kubectl -n kube-system port-forward svc/hubble-metrics 9091

[gandro]

Thanks for the report!

Given the steps to reproduce, I believe this is an issue of missing documentation, as there is an additional step required:

For HTTP and DNS metrics to work, there is an additional step required to enable L7 visibility on the services you want to collect HTTP/DNS metrics for. To enable L7 visibility for specific pods, please follow this guide: https://docs.cilium.io/en/v1.9/policy/visibility/

This unfortunately is badly documented right now. I see we do have a note about this in the latest metrics guide, but it is missing in the v1.9 versions of the docs (which is what we expect users to read): https://docs.cilium.io/en/latest/operations/metrics/#hubble-metrics

I have marked the above note for backport to the v1.9 branch, but I believe we should improve the docs in that space overall cilium/cilium#14211

[MaciejGawel]

Thanks for the reply!

I will enable L7 visibility according to the guide. I think that resolves my issue.