feat: Prometheus metrics support #2204

yoonsio · 2022-04-17T16:26:44Z

This PR adds support for Prometheus metrics and endpoint.

add configurable Prometheus metrics endpoint
dynamic subscope-based metric names
expose tally.Reporter to server struct for Prometheus handler (tally.Scope does not expose underlying Reporter)

Due to the limitation of tally, it is not possible to support both Statsd and Prometheus metrics at the same time.

Related Issue: #258

yoonsio · 2022-05-02T16:48:15Z

@jamengual Would appreciate if I could get some eyes on this. I think prometheus metrics would bring a lot of value to this project.

jamengual · 2022-05-02T22:01:49Z

Hi @yoonsio thank you for the contribution!!

I will need @nishkrishnan or @msarvar to take a look at this when they have some time.

ribejara-te · 2022-05-12T15:32:57Z

+1 on this, we'd love to have this feature!

jamengual · 2022-06-10T03:38:42Z

@yoonsio could you fix the conflicts please and pull from master ( we fixed some tests)

…ntis-master

Runatlantis master

yoonsio · 2022-06-22T21:35:58Z

@jamengual I rebased off the latest master. Tested working.

mustafa89 · 2022-06-27T21:25:18Z

Would be great to have this.

jamengual

LGTM

jamengual · 2022-06-29T05:46:38Z

@yoonsio could it be possible to add some docs and examples on how to use it?

yoonsio · 2022-06-29T05:48:39Z

@jamengual okay I will have a PR open in coming days

mustafa89 · 2022-08-04T09:32:13Z

this is breaking for us.

We noticed yesterday after adding this endpoint that the projects locks were not being released after an MR was being merged in our Gitlab.

Upon investigation we saw this:

2022/08/04 09:28:23 PANIC: descriptor Desc{fqName: "atlantis_pullclosed.cleanup_execution_success", help: "atlantis_pullclosed.cleanup_execution_success counter", constLabels: {}, variableLabels: []} is invalid: "atlantis_pullclosed.cleanup_execution_success" is not a valid metric name
goroutine 376 [running]:
github.com/urfave/negroni.(*Recovery).ServeHTTP.func1()
	github.com/urfave/negroni@v1.0.0/recovery.go:159 +0xc7
panic({0x10d5fa0, 0xc000935a10})
	runtime/panic.go:1038 +0x215
github.com/uber-go/tally/prometheus.NewReporter.func1({0x1535c60, 0xc000935a10})
	github.com/uber-go/tally@v3.4.3+incompatible/prometheus/reporter.go:306 +0x68
github.com/uber-go/tally/prometheus.(*reporter).AllocateCounter(0xc0007499e0, {0xc0001aa330, 0x2d}, 0x1517f08)
	github.com/uber-go/tally@v3.4.3+incompatible/prometheus/reporter.go:366 +0x11f
github.com/uber-go/tally.(*scope).Counter(0xc000ba0000, {0x13100d3, 0xc000909720})
	github.com/uber-go/tally@v3.4.3+incompatible/scope.go:280 +0x173
github.com/runatlantis/atlantis/server/events.(*InstrumentedPullClosedExecutor).CleanUpPull(_, {{0xc00040ac18, 0x13}, {0xc00040ac18, 0x8}, {0xc00040ac21, 0xa}, {0xc00011f200, 0x51}, {0xc0008fb040, ...}, ...}, ...)
	github.com/runatlantis/atlantis/server/events/instrumented_pull_closed_executor.go:35 +0x1a3
github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handlePullRequestEvent(_, {_, _}, {{0xc00040ac18, 0x13}, {0xc00040ac18, 0x8}, {0xc00040ac21, 0xa}, {0xc00011f200, ...}, ...}, ...)
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:452 +0x5f5
github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).HandleGitlabMergeRequestEvent(_, {_, _}, {{0xc000c9c890, 0xd}, 0xc0004224b0, {0x24d, {0xc000c9c8c0, 0xa}, {0xc000883c20, ...}, ...}, ...})
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:589 +0x530
github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handleGitlabPost(0xc00035c000, {0x7f72619377b8, 0xc00000e548}, 0x0)
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:488 +0x22c
github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).Post(0xc00035c000, {0x7f72619377b8, 0xc00000e548}, 0xc000203500)
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:107 +0x18f
net/http.HandlerFunc.ServeHTTP(0xc000203300, {0x7f72619377b8, 0xc00000e548}, 0x416fe6)
	net/http/server.go:2047 +0x2f
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000b8a300, {0x7f72619377b8, 0xc00000e548}, 0xc0001b3d00)
	github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1cf
github.com/urfave/negroni.Wrap.func1({0x7f72619377b8, 0xc00000e548}, 0x1183160, 0xc000a660c0)
	github.com/urfave/negroni@v1.0.0/negroni.go:46 +0x4b
github.com/urfave/negroni.HandlerFunc.ServeHTTP(0x40, {0x7f72619377b8, 0xc00000e548}, 0x7f72617b9cc8, 0x8)
	github.com/urfave/negroni@v1.0.0/negroni.go:29 +0x33
github.com/urfave/negroni.middleware.ServeHTTP({{0x1539b20, 0xc000a59560}, 0xc000a595a8}, {0x7f72619377b8, 0xc00000e548}, 0x3)
	github.com/urfave/negroni@v1.0.0/negroni.go:38 +0xb6
github.com/runatlantis/atlantis/server.(*RequestLogger).ServeHTTP(0xc000a65440, {0x7f72619377b8, 0xc00000e548}, 0xc0001b3d00, 0xc000a660a0)
	github.com/runatlantis/atlantis/server/middleware.go:68 +0x448
github.com/urfave/negroni.middleware.ServeHTTP({{0x15366e0, 0xc000a65440}, 0xc000a59590}, {0x7f72619377b8, 0xc00000e548}, 0x20)
	github.com/urfave/negroni@v1.0.0/negroni.go:38 +0xb6
github.com/urfave/negroni.(*Recovery).ServeHTTP(0xc000054400, {0x7f72619377b8, 0xc00000e548}, 0xc0001dba48, 0x40ac8a)
	github.com/urfave/negroni@v1.0.0/recovery.go:193 +0x8f
github.com/urfave/negroni.middleware.ServeHTTP({{0x1537140, 0xc000ab64b0}, 0xc000a59578}, {0x7f72619377b8, 0xc00000e548}, 0x0)
	github.com/urfave/negroni@v1.0.0/negroni.go:38 +0xb6
github.com/urfave/negroni.(*Negroni).ServeHTTP(0xc0004842d0, {0x154a298, 0xc000b07960}, 0x10)
	github.com/urfave/negroni@v1.0.0/negroni.go:96 +0x110
net/http.serverHandler.ServeHTTP({0x1547b50}, {0x154a298, 0xc000b07960}, 0xc0001b3d00)
	net/http/server.go:2879 +0x43b
net/http.(*conn).serve(0xc000762000, {0x15557f0, 0xc0002b6d50})
	net/http/server.go:1930 +0xb08
created by net/http.(*Server).Serve
	net/http/server.go:3034 +0x4e8
{"level":"info","ts":"2022-08-04T09:28:23.863Z","caller":"events/events_controller.go:588","msg":"identified event as type \"closed\"","json":{}}
2022/08/04 09:28:23 http: panic serving 10.254.14.97:2828: runtime error: invalid memory address or nil pointer dereference
goroutine 376 [running]:
net/http.(*conn).serve.func1()
	net/http/server.go:1802 +0xb9
panic({0x110f440, 0x1d9c420})
	runtime/panic.go:1047 +0x266
github.com/urfave/negroni.(*Recovery).ServeHTTP.func1()
	github.com/urfave/negroni@v1.0.0/recovery.go:166 +0x23c
panic({0x10d5fa0, 0xc000935a10})
	runtime/panic.go:1038 +0x215
github.com/uber-go/tally/prometheus.NewReporter.func1({0x1535c60, 0xc000935a10})
	github.com/uber-go/tally@v3.4.3+incompatible/prometheus/reporter.go:306 +0x68
github.com/uber-go/tally/prometheus.(*reporter).AllocateCounter(0xc0007499e0, {0xc0001aa330, 0x2d}, 0x1517f08)
	github.com/uber-go/tally@v3.4.3+incompatible/prometheus/reporter.go:366 +0x11f
github.com/uber-go/tally.(*scope).Counter(0xc000ba0000, {0x13100d3, 0xc000909720})
	github.com/uber-go/tally@v3.4.3+incompatible/scope.go:280 +0x173
github.com/runatlantis/atlantis/server/events.(*InstrumentedPullClosedExecutor).CleanUpPull(_, {{0xc00040ac18, 0x13}, {0xc00040ac18, 0x8}, {0xc00040ac21, 0xa}, {0xc00011f200, 0x51}, {0xc0008fb040, ...}, ...}, ...)
	github.com/runatlantis/atlantis/server/events/instrumented_pull_closed_executor.go:35 +0x1a3
github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handlePullRequestEvent(_, {_, _}, {{0xc00040ac18, 0x13}, {0xc00040ac18, 0x8}, {0xc00040ac21, 0xa}, {0xc00011f200, ...}, ...}, ...)
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:452 +0x5f5
github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).HandleGitlabMergeRequestEvent(_, {_, _}, {{0xc000c9c890, 0xd}, 0xc0004224b0, {0x24d, {0xc000c9c8c0, 0xa}, {0xc000883c20, ...}, ...}, ...})
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:589 +0x530
github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handleGitlabPost(0xc00035c000, {0x7f72619377b8, 0xc00000e548}, 0x0)
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:488 +0x22c
github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).Post(0xc00035c000, {0x7f72619377b8, 0xc00000e548}, 0xc000203500)
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:107 +0x18f
net/http.HandlerFunc.ServeHTTP(0xc000203300, {0x7f72619377b8, 0xc00000e548}, 0x416fe6)
	net/http/server.go:2047 +0x2f
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000b8a300, {0x7f72619377b8, 0xc00000e548}, 0xc0001b3d00)
	github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1cf
github.com/urfave/negroni.Wrap.func1({0x7f72619377b8, 0xc00000e548}, 0x1183160, 0xc000a660c0)
	github.com/urfave/negroni@v1.0.0/negroni.go:46 +0x4b
github.com/urfave/negroni.HandlerFunc.ServeHTTP(0x40, {0x7f72619377b8, 0xc00000e548}, 0x7f72617b9cc8, 0x8)
	github.com/urfave/negroni@v1.0.0/negroni.go:29 +0x33
github.com/urfave/negroni.middleware.ServeHTTP({{0x1539b20, 0xc000a59560}, 0xc000a595a8}, {0x7f72619377b8, 0xc00000e548}, 0x3)
	github.com/urfave/negroni@v1.0.0/negroni.go:38 +0xb6
github.com/runatlantis/atlantis/server.(*RequestLogger).ServeHTTP(0xc000a65440, {0x7f72619377b8, 0xc00000e548}, 0xc0001b3d00, 0xc000a660a0)
	github.com/runatlantis/atlantis/server/middleware.go:68 +0x448
github.com/urfave/negroni.middleware.ServeHTTP({{0x15366e0, 0xc000a65440}, 0xc000a59590}, {0x7f72619377b8, 0xc00000e548}, 0x20)
	github.com/urfave/negroni@v1.0.0/negroni.go:38 +0xb6
github.com/urfave/negroni.(*Recovery).ServeHTTP(0xc000054400, {0x7f72619377b8, 0xc00000e548}, 0xc0001dba48, 0x40ac8a)
	github.com/urfave/negroni@v1.0.0/recovery.go:193 +0x8f
github.com/urfave/negroni.middleware.ServeHTTP({{0x1537140, 0xc000ab64b0}, 0xc000a59578}, {0x7f72619377b8, 0xc00000e548}, 0x0)
	github.com/urfave/negroni@v1.0.0/negroni.go:38 +0xb6
github.com/urfave/negroni.(*Negroni).ServeHTTP(0xc0004842d0, {0x154a298, 0xc000b07960}, 0x10)
	github.com/urfave/negroni@v1.0.0/negroni.go:96 +0x110
net/http.serverHandler.ServeHTTP({0x1547b50}, {0x154a298, 0xc000b07960}, 0xc0001b3d00)
	net/http/server.go:2879 +0x43b
net/http.(*conn).serve(0xc000762000, {0x15557f0, 0xc0002b6d50})
	net/http/server.go:1930 +0xb08
created by net/http.(*Server).Serve
	net/http/server.go:3034 +0x4e8

Seems there is an issue with the metric name here:

 PANIC: descriptor Desc{fqName: "atlantis_pullclosed.cleanup_execution_success", help: "atlantis_pullclosed.cleanup_execution_success counter", constLabels: {}, variableLabels: []} is invalid: "atlantis_pullclosed.cleanup_execution_success" is not a valid metric name

nitrocode · 2022-10-06T19:47:00Z

@mustafa89 looks like the issue you're referring to may have been closed #2379 by #2528. The version 0.19.9 should fix your issue.

* support prometheus metrics * remove redundant error checking * load prometheus metrics config * dynamic separator for subscopes * removing unnecessary config file * adding a few more metrics test cases * remove testing handler * fix upstream merge error

yoonsio added 4 commits April 6, 2022 13:50

support prometheus metrics

0041e43

remove redundant error checking

e7f80ff

load prometheus metrics config

685be2c

dynamic separator for subscopes

55c8dc2

yoonsio requested a review from a team as a code owner April 17, 2022 16:26

yoonsio mentioned this pull request Apr 17, 2022

Expose metrics #258

Closed

yoonsio added 2 commits April 17, 2022 12:27

removing unnecessary config file

6e8405b

adding a few more metrics test cases

85cbf50

yoonsio changed the title ~~Feature: Prometheus metrics support~~ feat: Prometheus metrics support Apr 17, 2022

remove testing handler

e5ef0e3

jamengual added feature New functionality/enhancement waiting-on-review Waiting for a review from a maintainer labels May 13, 2022

jamengual added the waiting-on-response Waiting for a response from the user label Jun 10, 2022

yoonsio added 3 commits June 22, 2022 17:30

Merge branch 'master' of github.com:runatlantis/atlantis into runatla…

4e82a43

…ntis-master

Merge pull request #9 from yoonsio/runatlantis-master

6c0c60a

Runatlantis master

fix upstream merge error

59aab1c

jamengual merged commit e90aa57 into runatlantis:master Jun 28, 2022

jamengual reviewed Jun 28, 2022

View reviewed changes

albertollamaso mentioned this pull request Sep 17, 2022

Fix: Error when enabling prometheus metrics #2528

Merged

nikolaik mentioned this pull request Oct 10, 2022

Docs: Missing prometheus metrics #2576

Closed

nitrocode added this to the v0.19.5 milestone Mar 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Prometheus metrics support #2204

feat: Prometheus metrics support #2204

yoonsio commented Apr 17, 2022 •

edited

Loading

yoonsio commented May 2, 2022

jamengual commented May 2, 2022

ribejara-te commented May 12, 2022

jamengual commented Jun 10, 2022

yoonsio commented Jun 22, 2022 •

edited

Loading

mustafa89 commented Jun 27, 2022

jamengual left a comment

jamengual commented Jun 29, 2022

yoonsio commented Jun 29, 2022

mustafa89 commented Aug 4, 2022 •

edited

Loading

nitrocode commented Oct 6, 2022

feat: Prometheus metrics support #2204

feat: Prometheus metrics support #2204

Conversation

yoonsio commented Apr 17, 2022 • edited Loading

yoonsio commented May 2, 2022

jamengual commented May 2, 2022

ribejara-te commented May 12, 2022

jamengual commented Jun 10, 2022

yoonsio commented Jun 22, 2022 • edited Loading

mustafa89 commented Jun 27, 2022

jamengual left a comment

Choose a reason for hiding this comment

jamengual commented Jun 29, 2022

yoonsio commented Jun 29, 2022

mustafa89 commented Aug 4, 2022 • edited Loading

nitrocode commented Oct 6, 2022

yoonsio commented Apr 17, 2022 •

edited

Loading

yoonsio commented Jun 22, 2022 •

edited

Loading

mustafa89 commented Aug 4, 2022 •

edited

Loading