New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add extra prometheus metrics #32792

Merged
merged 1 commit into from May 9, 2017

Conversation

@rogaha
Contributor

rogaha commented Apr 24, 2017

Signed-off-by: Roberto Gandolfo Hashioka roberto.hashioka@docker.com

- What I did

Added/updated prometheus metrics:

  • buildsTriggered: type Counter
  • buildsFailed: type LabeledCounter
    • dockerfile_syntax
    • dockerfile_empty
    • command_not_supported
      - error_processing_commands
      - build_target_not_reachable
      - missing_onbuild_arguments
      - unknown_instruction
      - build_canceled
  • engineInfo: type LabeledGauge

- How I did it

Extended the existing prometheus metrics using the docker/go-metrics package

- How to verify it

$ make binary

replace docker binaries with the ones located in bundles/17.06.0-dev/binary-daemon

- Description for the changelog

  • Added build & engine info prometheus metrics

- A picture of a cute animal (not mandatory but encouraged)
boo-cute-dog-pomeranian-favim com-452643

@rogaha

This comment has been minimized.

Show comment
Hide comment
@rogaha
Contributor

rogaha commented Apr 25, 2017

@aluzzardi

This comment has been minimized.

Show comment
Hide comment
@aluzzardi
Member

aluzzardi commented Apr 25, 2017

Show outdated Hide outdated daemon/cluster/swarm.go Outdated
Show outdated Hide outdated daemon/cluster/metrics.go Outdated
Show outdated Hide outdated daemon/cluster/metrics.go Outdated
Show outdated Hide outdated daemon/cluster/metrics.go Outdated
Show outdated Hide outdated daemon/metrics.go Outdated
Show outdated Hide outdated builder/dockerfile/builder.go Outdated
Show outdated Hide outdated daemon/metrics.go Outdated
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Apr 30, 2017

Member

Looks like this needs a rebase

Member

thaJeztah commented Apr 30, 2017

Looks like this needs a rebase

@aluzzardi

This comment has been minimized.

Show comment
Hide comment
@aluzzardi

aluzzardi Apr 30, 2017

Member

My 2cts: I think this should be a controller in swarmkit (swarmkit/manager/metrics, instrumentation, telemetry?) that uses the internal store API to export metrics.

@dongluochen @aaronlehmann Thoughts on that?

Member

aluzzardi commented Apr 30, 2017

My 2cts: I think this should be a controller in swarmkit (swarmkit/manager/metrics, instrumentation, telemetry?) that uses the internal store API to export metrics.

@dongluochen @aaronlehmann Thoughts on that?

@rogaha

This comment has been minimized.

Show comment
Hide comment
@rogaha

rogaha Apr 30, 2017

Contributor

@aluzzardi I agree that those metrics should be in swarmkit, but they are not there yet! So nothing prevents us to expose the swarm info data at scrape time using the approach @aaronlehmann suggested (prometheus Collector). I'll remove the swarm related metrics from this PR as @alfred-landrum offered his help to implement the prometheus Collector to expose the engine/swarm info (https://godoc.org/github.com/prometheus/client_golang/prometheus#ex-Collector) -- he has already worked on that in the past.

Contributor

rogaha commented Apr 30, 2017

@aluzzardi I agree that those metrics should be in swarmkit, but they are not there yet! So nothing prevents us to expose the swarm info data at scrape time using the approach @aaronlehmann suggested (prometheus Collector). I'll remove the swarm related metrics from this PR as @alfred-landrum offered his help to implement the prometheus Collector to expose the engine/swarm info (https://godoc.org/github.com/prometheus/client_golang/prometheus#ex-Collector) -- he has already worked on that in the past.

@rogaha

This comment has been minimized.

Show comment
Hide comment
@rogaha

rogaha May 1, 2017

Contributor

@thaJeztah PR rebased

Contributor

rogaha commented May 1, 2017

@thaJeztah PR rebased

@aluzzardi

This comment has been minimized.

Show comment
Hide comment
@aluzzardi

aluzzardi May 1, 2017

Member

The metrics are being exposed under the engine namespace. Once they are exposed by swarmkit, the labels will be different

Member

aluzzardi commented May 1, 2017

The metrics are being exposed under the engine namespace. Once they are exposed by swarmkit, the labels will be different

@aluzzardi

This comment has been minimized.

Show comment
Hide comment
@aluzzardi

aluzzardi May 1, 2017

Member

Also, once exposed by swarmkit, nodes will be exposed as a gauge labeled by state (in the same way the engine exposes containers by state).

Basically, we should have {nodes,services,tasks}_by_state in swarmkit itself.

Member

aluzzardi commented May 1, 2017

Also, once exposed by swarmkit, nodes will be exposed as a gauge labeled by state (in the same way the engine exposes containers by state).

Basically, we should have {nodes,services,tasks}_by_state in swarmkit itself.

@rogaha

This comment has been minimized.

Show comment
Hide comment
@rogaha

rogaha May 1, 2017

Contributor

@aluzzardi it sounds great! Are you ok if for the time being (while the swarmkit metrics are developed) we expose some of the swarm metrics via the engine/info? We are already doing that via the Docker API / CLI.

Contributor

rogaha commented May 1, 2017

@aluzzardi it sounds great! Are you ok if for the time being (while the swarmkit metrics are developed) we expose some of the swarm metrics via the engine/info? We are already doing that via the Docker API / CLI.

@aluzzardi

This comment has been minimized.

Show comment
Hide comment
@aluzzardi

aluzzardi May 1, 2017

Member

@rogaha The thing is, the names, namespace and labels will be different in swarmkit which will break whoever is scraping those.

Are we okay with this? If so, then it's fine.

Otherwise, I've put together a proof of concept here (it's not much more complicated than what has been done here):
docker/swarmkit#2157

Thoughts, @aaronlehmann @rogaha?

Member

aluzzardi commented May 1, 2017

@rogaha The thing is, the names, namespace and labels will be different in swarmkit which will break whoever is scraping those.

Are we okay with this? If so, then it's fine.

Otherwise, I've put together a proof of concept here (it's not much more complicated than what has been done here):
docker/swarmkit#2157

Thoughts, @aaronlehmann @rogaha?

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 May 1, 2017

Contributor

I think it would be best to add to swarmkit.
There's no reason the swarmkit changes can't make the next release other than settling on namespace... which is a problem here as well.

Contributor

cpuguy83 commented May 1, 2017

I think it would be best to add to swarmkit.
There's no reason the swarmkit changes can't make the next release other than settling on namespace... which is a problem here as well.

@rogaha

This comment has been minimized.

Show comment
Hide comment
@rogaha

rogaha May 2, 2017

Contributor

@cpuguy83 agreed. I've removed the swarm related metrics from the PR.

Contributor

rogaha commented May 2, 2017

@cpuguy83 agreed. I've removed the swarm related metrics from the PR.

@rogaha

This comment has been minimized.

Show comment
Hide comment
@rogaha

rogaha May 2, 2017

Contributor

The work done by @aluzzardi here seems to be very promising!

Contributor

rogaha commented May 2, 2017

The work done by @aluzzardi here seems to be very promising!

@rogaha

This comment has been minimized.

Show comment
Hide comment
@rogaha

rogaha May 2, 2017

Contributor

@thaJeztah rebased again.

Contributor

rogaha commented May 2, 2017

@thaJeztah rebased again.

Show outdated Hide outdated builder/dockerfile/builder.go Outdated
Show outdated Hide outdated builder/dockerfile/metrics.go Outdated
Show outdated Hide outdated builder/dockerfile/metrics.go Outdated
Show outdated Hide outdated builder/dockerfile/metrics.go Outdated
Show outdated Hide outdated daemon/metrics.go Outdated
@rogaha

This comment has been minimized.

Show comment
Hide comment
@rogaha
Contributor

rogaha commented May 3, 2017

@rogaha

This comment has been minimized.

Show comment
Hide comment
@rogaha

rogaha May 3, 2017

Contributor

@stevvooe here are the output examples:

engine info

screen shot 2017-05-03 at 2 26 00 pm

total number of successful triggered builds

screen shot 2017-05-03 at 2 25 39 pm

total number of triggered builds

screen shot 2017-05-03 at 2 25 29 pm

total number of failed triggered builds

screen shot 2017-05-03 at 2 25 16 pm

raw data

# HELP builder_dockerfile_builds_triggered_failed_total Number of failed image builds
# TYPE builder_dockerfile_builds_triggered_failed_total counter
builder_dockerfile_builds_triggered_failed_total{reason="build_canceled"} 0
builder_dockerfile_builds_triggered_failed_total{reason="build_target_not_reachable"} 0
builder_dockerfile_builds_triggered_failed_total{reason="command_not_supported"} 0
builder_dockerfile_builds_triggered_failed_total{reason="dockerfile_empty"} 0
builder_dockerfile_builds_triggered_failed_total{reason="dockerfile_syntax"} 0
builder_dockerfile_builds_triggered_failed_total{reason="error_processing_commands"} 0
builder_dockerfile_builds_triggered_failed_total{reason="missing_onbuild_arguments"} 0
builder_dockerfile_builds_triggered_failed_total{reason="unknown_instruction"} 0
# HELP builder_dockerfile_builds_triggered_success_total Number of successful image builds
# TYPE builder_dockerfile_builds_triggered_success_total counter
builder_dockerfile_builds_triggered_success_total 1
# HELP builder_dockerfile_builds_triggered_total Number of triggered image builds
# TYPE builder_dockerfile_builds_triggered_total counter
builder_dockerfile_builds_triggered_total 1
Contributor

rogaha commented May 3, 2017

@stevvooe here are the output examples:

engine info

screen shot 2017-05-03 at 2 26 00 pm

total number of successful triggered builds

screen shot 2017-05-03 at 2 25 39 pm

total number of triggered builds

screen shot 2017-05-03 at 2 25 29 pm

total number of failed triggered builds

screen shot 2017-05-03 at 2 25 16 pm

raw data

# HELP builder_dockerfile_builds_triggered_failed_total Number of failed image builds
# TYPE builder_dockerfile_builds_triggered_failed_total counter
builder_dockerfile_builds_triggered_failed_total{reason="build_canceled"} 0
builder_dockerfile_builds_triggered_failed_total{reason="build_target_not_reachable"} 0
builder_dockerfile_builds_triggered_failed_total{reason="command_not_supported"} 0
builder_dockerfile_builds_triggered_failed_total{reason="dockerfile_empty"} 0
builder_dockerfile_builds_triggered_failed_total{reason="dockerfile_syntax"} 0
builder_dockerfile_builds_triggered_failed_total{reason="error_processing_commands"} 0
builder_dockerfile_builds_triggered_failed_total{reason="missing_onbuild_arguments"} 0
builder_dockerfile_builds_triggered_failed_total{reason="unknown_instruction"} 0
# HELP builder_dockerfile_builds_triggered_success_total Number of successful image builds
# TYPE builder_dockerfile_builds_triggered_success_total counter
builder_dockerfile_builds_triggered_success_total 1
# HELP builder_dockerfile_builds_triggered_total Number of triggered image builds
# TYPE builder_dockerfile_builds_triggered_total counter
builder_dockerfile_builds_triggered_total 1
@brian-brazil

This comment has been minimized.

Show comment
Hide comment
@brian-brazil

brian-brazil May 5, 2017

We have a million different errors returned by the engine - I see no point of making these random ones special and try to have fake stabilization for them.

Having a high or practically unbounded number of label values on a metric is unwise. I'd suggest squashing the errors down to a few general classes (e.g. syntax error) and if the user wants more detail they can look at logs.

brian-brazil commented May 5, 2017

We have a million different errors returned by the engine - I see no point of making these random ones special and try to have fake stabilization for them.

Having a high or practically unbounded number of label values on a metric is unwise. I'd suggest squashing the errors down to a few general classes (e.g. syntax error) and if the user wants more detail they can look at logs.

@rogaha

This comment has been minimized.

Show comment
Hide comment
@rogaha
Contributor

rogaha commented May 5, 2017

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah May 5, 2017

Member

@rogaha needs a rebase already 😅

Member

thaJeztah commented May 5, 2017

@rogaha needs a rebase already 😅

@rogaha

This comment has been minimized.

Show comment
Hide comment
@rogaha

rogaha May 5, 2017

Contributor

@thaJeztah rebased, again.

Contributor

rogaha commented May 5, 2017

@thaJeztah rebased, again.

Show outdated Hide outdated builder/dockerfile/metrics.go Outdated
@rogaha

This comment has been minimized.

Show comment
Hide comment
@rogaha

rogaha May 5, 2017

Contributor

@stevvooe current output:

# HELP builder_builds_failed_total Number of failed image builds
# TYPE builder_builds_failed_total counter
builder_builds_failed_total{reason="build_canceled"} 0
builder_builds_failed_total{reason="build_target_not_reachable"} 0
builder_builds_failed_total{reason="command_not_supported"} 0
builder_builds_failed_total{reason="dockerfile_empty"} 0
builder_builds_failed_total{reason="dockerfile_syntax"} 0
builder_builds_failed_total{reason="error_processing_commands"} 0
builder_builds_failed_total{reason="missing_onbuild_arguments"} 0
builder_builds_failed_total{reason="unknown_instruction"} 0
# HELP builder_builds_triggered_total Number of triggered image builds
# TYPE builder_builds_triggered_total counter
builder_builds_triggered_total 60
Contributor

rogaha commented May 5, 2017

@stevvooe current output:

# HELP builder_builds_failed_total Number of failed image builds
# TYPE builder_builds_failed_total counter
builder_builds_failed_total{reason="build_canceled"} 0
builder_builds_failed_total{reason="build_target_not_reachable"} 0
builder_builds_failed_total{reason="command_not_supported"} 0
builder_builds_failed_total{reason="dockerfile_empty"} 0
builder_builds_failed_total{reason="dockerfile_syntax"} 0
builder_builds_failed_total{reason="error_processing_commands"} 0
builder_builds_failed_total{reason="missing_onbuild_arguments"} 0
builder_builds_failed_total{reason="unknown_instruction"} 0
# HELP builder_builds_triggered_total Number of triggered image builds
# TYPE builder_builds_triggered_total counter
builder_builds_triggered_total 60
@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe May 5, 2017

Contributor

LGTM

Contributor

stevvooe commented May 5, 2017

LGTM

Show outdated Hide outdated daemon/metrics.go Outdated
@rogaha

This comment has been minimized.

Show comment
Hide comment
@rogaha

rogaha May 8, 2017

Contributor

The unit-tests failures don't seem to be related to the changes in this PR:

00:36:12 ----------------------------------------------------------------------
00:36:12 FAIL: check_test.go:355: DockerSwarmSuite.TearDownTest
00:36:12 
00:36:12 unmount of /tmp/docker-execroot/d373fc5c29f3c/netns failed: invalid argument
00:36:12 unmount of /tmp/docker-execroot/d373fc5c29f3c/netns failed: no such file or directory
00:36:12 unmount of /tmp/docker-execroot/d5f595cc1336f/netns failed: no such file or directory
00:36:12 check_test.go:360:
00:36:12     d.Stop(c)
00:36:12 daemon/daemon.go:392:
00:36:12     t.Fatalf("Error while stopping the daemon %s : %v", d.id, err)
00:36:12 ... Error: Error while stopping the daemon df643cfe854d0 : exit status 2
00:43:44 ----------------------------------------------------------------------
00:43:44 FAIL: docker_cli_swarm_test.go:1371: DockerSwarmSuite.TestSwarmClusterRotateUnlockKey
00:43:44 
00:43:44 [d728ab7ffcd95] waiting for daemon to start
00:43:44 [d728ab7ffcd95] daemon started
00:43:44 
00:43:44 [d8a9593b13a08] waiting for daemon to start
00:43:44 [d8a9593b13a08] daemon started
00:43:44 
00:43:44 [d1573de56bd46] waiting for daemon to start
00:43:44 [d1573de56bd46] daemon started
00:43:44 
00:43:44 [d8a9593b13a08] exiting daemon
00:43:44 [d8a9593b13a08] waiting for daemon to start
00:43:44 [d8a9593b13a08] daemon started
00:43:44 
00:43:44 [d1573de56bd46] exiting daemon
00:43:44 [d1573de56bd46] waiting for daemon to start
00:43:44 [d1573de56bd46] daemon started
00:43:44 
00:43:44 docker_cli_swarm_test.go:1448:
00:43:44     c.Assert(err, checker.IsNil)
00:43:44 ... value *exec.ExitError = &exec.ExitError{ProcessState:(*os.ProcessState)(0xc421923ec0), Stderr:[]uint8(nil)} ("exit status 1")
00:43:44 
00:43:44 [d728ab7ffcd95] exiting daemon
00:43:44 [d8a9593b13a08] exiting daemon
00:43:44 [d1573de56bd46] exiting daemon
00:37:51 ----------------------------------------------------------------------
00:37:51 FAIL: check_test.go:355: DockerSwarmSuite.TearDownTest
00:37:51 
00:37:51 unmount of /tmp/docker-execroot/d711731fa58b2/netns failed: invalid argument
00:37:51 unmount of /tmp/docker-execroot/d711731fa58b2/netns failed: no such file or directory
00:37:51 check_test.go:360:
00:37:51     d.Stop(c)
00:37:51 daemon/daemon.go:392:
00:37:51     t.Fatalf("Error while stopping the daemon %s : %v", d.id, err)
00:37:51 ... Error: Error while stopping the daemon def4990e6fdf2 : exit status 2

@crosbymichael PTAL

Contributor

rogaha commented May 8, 2017

The unit-tests failures don't seem to be related to the changes in this PR:

00:36:12 ----------------------------------------------------------------------
00:36:12 FAIL: check_test.go:355: DockerSwarmSuite.TearDownTest
00:36:12 
00:36:12 unmount of /tmp/docker-execroot/d373fc5c29f3c/netns failed: invalid argument
00:36:12 unmount of /tmp/docker-execroot/d373fc5c29f3c/netns failed: no such file or directory
00:36:12 unmount of /tmp/docker-execroot/d5f595cc1336f/netns failed: no such file or directory
00:36:12 check_test.go:360:
00:36:12     d.Stop(c)
00:36:12 daemon/daemon.go:392:
00:36:12     t.Fatalf("Error while stopping the daemon %s : %v", d.id, err)
00:36:12 ... Error: Error while stopping the daemon df643cfe854d0 : exit status 2
00:43:44 ----------------------------------------------------------------------
00:43:44 FAIL: docker_cli_swarm_test.go:1371: DockerSwarmSuite.TestSwarmClusterRotateUnlockKey
00:43:44 
00:43:44 [d728ab7ffcd95] waiting for daemon to start
00:43:44 [d728ab7ffcd95] daemon started
00:43:44 
00:43:44 [d8a9593b13a08] waiting for daemon to start
00:43:44 [d8a9593b13a08] daemon started
00:43:44 
00:43:44 [d1573de56bd46] waiting for daemon to start
00:43:44 [d1573de56bd46] daemon started
00:43:44 
00:43:44 [d8a9593b13a08] exiting daemon
00:43:44 [d8a9593b13a08] waiting for daemon to start
00:43:44 [d8a9593b13a08] daemon started
00:43:44 
00:43:44 [d1573de56bd46] exiting daemon
00:43:44 [d1573de56bd46] waiting for daemon to start
00:43:44 [d1573de56bd46] daemon started
00:43:44 
00:43:44 docker_cli_swarm_test.go:1448:
00:43:44     c.Assert(err, checker.IsNil)
00:43:44 ... value *exec.ExitError = &exec.ExitError{ProcessState:(*os.ProcessState)(0xc421923ec0), Stderr:[]uint8(nil)} ("exit status 1")
00:43:44 
00:43:44 [d728ab7ffcd95] exiting daemon
00:43:44 [d8a9593b13a08] exiting daemon
00:43:44 [d1573de56bd46] exiting daemon
00:37:51 ----------------------------------------------------------------------
00:37:51 FAIL: check_test.go:355: DockerSwarmSuite.TearDownTest
00:37:51 
00:37:51 unmount of /tmp/docker-execroot/d711731fa58b2/netns failed: invalid argument
00:37:51 unmount of /tmp/docker-execroot/d711731fa58b2/netns failed: no such file or directory
00:37:51 check_test.go:360:
00:37:51     d.Stop(c)
00:37:51 daemon/daemon.go:392:
00:37:51     t.Fatalf("Error while stopping the daemon %s : %v", d.id, err)
00:37:51 ... Error: Error while stopping the daemon def4990e6fdf2 : exit status 2

@crosbymichael PTAL

Show outdated Hide outdated builder/dockerfile/metrics.go Outdated
@rogaha

This comment has been minimized.

Show comment
Hide comment
@rogaha

rogaha May 8, 2017

Contributor

@cpuguy83 PTAL

Contributor

rogaha commented May 8, 2017

@cpuguy83 PTAL

@crosbymichael

This comment has been minimized.

Show comment
Hide comment
@crosbymichael

crosbymichael May 8, 2017

Contributor

LGTM

Contributor

crosbymichael commented May 8, 2017

LGTM

Add extra prometheus metrics
 - buildsTriggered
 - buildsFailed
    - valid options:
        metricsDockerfileSyntaxError,
        metricsDockerfileEmptyError,
        metricsCommandNotSupportedError,
        metricsErrorProcessingCommandsError,
        metricsBuildTargetNotReachableError,
        metricsMissingOnbuildArgumentsError,
        metricsUnknownInstructionError,
        metricsBuildCanceled,
- engineInfo

Signed-off-by: Roberto Gandolfo Hashioka <roberto_hashioka@hotmail.com>
@cpuguy83

LGTM code+design

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah May 9, 2017

Member

PowerPC failure is a flaky test;

09:13:59 ----------------------------------------------------------------------
09:13:59 FAIL: check_test.go:355: DockerSwarmSuite.TearDownTest
09:13:59 
09:13:59 unmount of /tmp/docker-execroot/df71d386c61af/netns failed: invalid argument
09:13:59 unmount of /tmp/docker-execroot/df71d386c61af/netns failed: no such file or directory
09:13:59 unmount of /tmp/docker-execroot/dfb4d9d9b9d35/netns failed: no such file or directory
09:13:59 check_test.go:360:
09:13:59     d.Stop(c)
09:13:59 daemon/daemon.go:392:
09:13:59     t.Fatalf("Error while stopping the daemon %s : %v", d.id, err)
09:13:59 ... Error: Error while stopping the daemon d821ee3d36cc5 : exit status 2

Let me trigger once

Member

thaJeztah commented May 9, 2017

PowerPC failure is a flaky test;

09:13:59 ----------------------------------------------------------------------
09:13:59 FAIL: check_test.go:355: DockerSwarmSuite.TearDownTest
09:13:59 
09:13:59 unmount of /tmp/docker-execroot/df71d386c61af/netns failed: invalid argument
09:13:59 unmount of /tmp/docker-execroot/df71d386c61af/netns failed: no such file or directory
09:13:59 unmount of /tmp/docker-execroot/dfb4d9d9b9d35/netns failed: no such file or directory
09:13:59 check_test.go:360:
09:13:59     d.Stop(c)
09:13:59 daemon/daemon.go:392:
09:13:59     t.Fatalf("Error while stopping the daemon %s : %v", d.id, err)
09:13:59 ... Error: Error while stopping the daemon d821ee3d36cc5 : exit status 2

Let me trigger once

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah May 9, 2017

Member

All 💚

Member

thaJeztah commented May 9, 2017

All 💚

@thaJeztah thaJeztah merged commit 1a6f8a9 into moby:master May 9, 2017

6 checks passed

dco-signed All commits are signed
experimental Jenkins build Docker-PRs-experimental 33854 has succeeded
Details
janky Jenkins build Docker-PRs 42454 has succeeded
Details
powerpc Jenkins build Docker-PRs-powerpc 2823 has succeeded
Details
windowsRS1 Jenkins build Docker-PRs-WoW-RS1 13682 has succeeded
Details
z Jenkins build Docker-PRs-s390x 2551 has succeeded
Details
@rogaha

This comment has been minimized.

Show comment
Hide comment
@rogaha

rogaha May 9, 2017

Contributor

thanks @thaJeztah!

Contributor

rogaha commented May 9, 2017

thanks @thaJeztah!

@rogaha rogaha deleted the rogaha:extra_prometheus_metrics branch May 9, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment