proxy metrics endpoint returns only one metric

[siggy]

Background

Hitting the proxy's /metrics endpoint sometimes gives complete data, and sometimes only returns process_start_time_seconds:

$ curl -s localhost:4191/metrics
# HELP process_start_time_seconds Time that the process started (in seconds since the UNIX epoch)
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1525739904

$ curl -s localhost:4191/metrics|wc -l
99
$ curl -s localhost:4191/metrics|wc -l
3
$ curl -s localhost:4191/metrics|wc -l
3
$ curl -s localhost:4191/metrics|wc -l
3
$ curl -s localhost:4191/metrics|wc -l
99

Repro steps

From master, (https://github.com/runconduit/conduit/tree/02e6d018d0790a69b95ef0c5630120668fc5e280 as of this issue):

1. Build Conduit:

DOCKER_TRACE=1 bin/docker-build && bin/conduit install | kubectl apply -f - && curl https://raw.githubusercontent.com/runconduit/conduit-examples/master/emojivoto/emojivoto.yml | bin/conduit inject - | kubectl apply -f -

2. SSH into proxy:

kubectl -n emojivoto exec -it $(kubectl -n emojivoto get po --selector=app=web-svc -o jsonpath='{.items[*].metadata.name}') /bin/bash -c conduit-proxy

3. curl metrics:

curl localhost:4191/metrics

Source

While testing Grafana, @rmars observed gaps in proxy metrics: #904 (comment)

[olix0r]

I can't replicate this exact behavior -- my proxy alternates between 174 and 203 lines of output -- but I think i have a lead to pursue on this

[olix0r]

Using this script:

:; CURL="kubectl -n emojivoto exec -it web-8d775d974-k2mwq -c conduit-proxy -- curl" ./debug.sh
*      154	a9be5182
*      212	787d2bda
*      183	91ced213
       154	a9be5182
       154	a9be5182
       154	a9be5182
       183	91ced213
       183	91ced213
       183	91ced213
...

I notice differences such as

-response_latency_ms_bucket{authority="voting-svc.emojivoto:8080",direction="outbound",dst_namespace="emojivoto",dst_service="voting-svc",dst_pod="voting-59fdf85876-6dhvc",dst_pod_template_hash="1598941432",dst_deployment="voting",classification="failure",status_code="200",grpc_status_code="2",le="+Inf"}
+response_latency_ms_bucket{authority="voting-svc.emojivoto:8080",direction="outbound",dst_namespace="emojivoto",dst_service="voting-svc",dst_pod="voting-59fdf85876-6dhvc",dst_pod_template_hash="1598941432",dst_deployment="voting",classification="success",status_code="200",grpc_status_code="0",le="+Inf"}