Add prometheus proxy related metrics

[dimitar-kostadinov]

If registry is configured as pull-through cache the following proxy related prometheus metrics are exposed:

• registry_proxy_requests_total - total incoming proxy request received for blob/manifest
• registry_proxy_hits_total - total proxy request hits for blob/manifest
• registry_proxy_misses_total - total proxy request misses for blob/manifest
• registry_proxy_pulled_bytes_total - total bytes pulled from the upstream for blob/manifest
• registry_proxy_pushed_bytes_total - total bytes pushed to the client for blob/manifest

[ialidzhikov]

@thaJeztah @milosgajdos , a friendly ping. We would like to proceed with the monitoring story on our side.

[milosgajdos]

Sorry about the delay with this. I've had other priorities to look at. Hopefully this week.

[Jamstah]

There is no test code here, which is a shame. Looking at the code we're missing lots of tests for the proxy though.

I assume there has been a manual test? Can you provide some test results for completeness?

[dimitar-kostadinov]

I assume there has been a manual test? Can you provide some test results for completeness?

Yes, here are steps for a simple manual test:

• Checkout this PR gh pr checkout 4047
• Start registry as pull through cache

# build
% make binaries
# prepare configuration
% mkdir -p ~/registry-test
% export REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=~/registry-test
% export REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io
% export REGISTRY_HTTP_ADDR=":5500"
% export REGISTRY_HTTP_DEBUG_ADDR=":5501"
# start the registry as proxy cache to docker.io
% ./bin/registry serve cmd/registry/config-dev.yml

• Configure docker to use registry mirror

{
  "registry-mirrors": [
    "http://test.registry.mirror:5500"
  ]
}

• Update /etc/hosts file with following record

echo '127.0.0.1 test.registry.mirror' >> /etc/host

• Pull a image % docker pull nginx
• Get the prometeus metrics

 % curl http://test.registry.mirror:5501/metrics
...
# HELP registry_proxy_hits_total The number of total proxy request hits
# TYPE registry_proxy_hits_total counter
registry_proxy_hits_total{type="blob"} 8
registry_proxy_hits_total{type="manifest"} 3
# HELP registry_proxy_misses_total The number of total proxy request misses
# TYPE registry_proxy_misses_total counter
registry_proxy_misses_total{type="manifest"} 2
# HELP registry_proxy_pulled_bytes_total The size of total bytes pulled from the upstream
# TYPE registry_proxy_pulled_bytes_total counter
registry_proxy_pulled_bytes_total{type="manifest"} 3640
# HELP registry_proxy_pushed_bytes_total The size of total bytes pushed to the client
# TYPE registry_proxy_pushed_bytes_total counter
registry_proxy_pushed_bytes_total{type="blob"} 6.7238076e+07
registry_proxy_pushed_bytes_total{type="manifest"} 5502
# HELP registry_proxy_requests_total The number of total incoming proxy request received
# TYPE registry_proxy_requests_total counter
registry_proxy_requests_total{type="blob"} 8
registry_proxy_requests_total{type="manifest"} 3
...

Note: registry_proxy_pulled_bytes_total{type="blob"} metric is missing due to #4044

[Jamstah]

Thanks, having some proof in the PR is better than nothing :)