hubble: Hubble metrics server conflicts with cilium metrics

[gandro]

It seems like it is currently not possible to enable both the Cilium and the embeded Hubble metrics server. When using the helm option global.prometheus.enabled=true together with global.hubble.metricsServer=":19999", cilium-agent panics due to some conflict with the /metrics endpoint:

level=info msg="Starting Hubble Metrics server" address=":19999" metrics="[dns:query;ignoreAAAA drop tcp flow port-distribution icmp http]" subsys=hubble
time="2020-04-20T13:55:36Z" level=info msg="Configured metrics plugin" name=icmp status=
time="2020-04-20T13:55:36Z" level=info msg="Configured metrics plugin" name=http status=
time="2020-04-20T13:55:36Z" level=info msg="Configured metrics plugin" name=dns status="query,ignoreAAAA,"
time="2020-04-20T13:55:36Z" level=info msg="Configured metrics plugin" name=drop status=
time="2020-04-20T13:55:36Z" level=info msg="Configured metrics plugin" name=tcp status=
time="2020-04-20T13:55:36Z" level=info msg="Configured metrics plugin" name=flow status=
time="2020-04-20T13:55:36Z" level=info msg="Configured metrics plugin" name=port-distribution status=
level=info msg="Beginning to read perf buffer" startTime="2020-04-20 13:55:36.509912077 +0000 UTC m=+9.873313357" subsys=monitor-agent
level=info msg="Starting gRPC server on listener" listener="unix:///var/run/cilium/hubble.sock" subsys=hubble
level=info msg="Starting gRPC server on listener" listener="0.0.0.0:50051" subsys=hubble
panic: http: multiple registrations for /metrics

goroutine 569 [running]:
net/http.(*ServeMux).Handle(0x39e71a0, 0x22c2ba7, 0x8, 0x26c7b20, 0xc0001fa3f0)
	/usr/local/go/src/net/http/server.go:2403 +0x2b6
net/http.Handle(...)
	/usr/local/go/src/net/http/server.go:2446
github.com/cilium/cilium/pkg/hubble/metrics.Init.func1(0xc00049c720, 0xc0004f4620, 0x6)
	/go/src/github.com/cilium/cilium/pkg/hubble/metrics/metrics.go:59 +0xa6
created by github.com/cilium/cilium/pkg/hubble/metrics.Init
	/go/src/github.com/cilium/cilium/pkg/hubble/metrics/metrics.go:58 +0xd6

Not entirely sure why it conflicts, since I would expect the two metrics servers to be served on distinct ports.

[soumynathan]

Thanks for creating this issue. Will monitor it.