Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add prometheus metrics #148

Merged
merged 21 commits into from Apr 29, 2020
Merged

Add prometheus metrics #148

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
f6f7931
Run gofmt
Mar 2, 2020
fdae529
Add integration test for prometheus metrics
Feb 28, 2020
2ad68f4
Add default prometheus metrics to router
Feb 28, 2020
dff3eaa
Add high level router metrics
Feb 28, 2020
cb56d93
Add metric for redirect handler redirect count
Feb 28, 2020
af5a21e
Add metric for currently count of loaded routes
Feb 28, 2020
4bea3bd
Add metric for triemux lookup not found
Feb 28, 2020
32e5de0
Add backend handler metric req count / resp secs
Mar 2, 2020
5ad3181
Keep track of backend ID within backend transport
Mar 2, 2020
ee1b938
Instrument backend handler req count / resp secs
Mar 2, 2020
b443d3d
Instrument backend handler response count
Mar 2, 2020
b563e9a
Use a real histogram for backend handler resp secs
Mar 2, 2020
91feca2
Run go mod tidy
Mar 2, 2020
eb014f6
Run go mod vendor
Mar 2, 2020
0c29314
Add a make task to pretty print metric names
Mar 2, 2020
6b1e0e8
Make metrics pass promtool
Mar 3, 2020
b8f84f0
Add comment clarifying use of stripQuery
tlwr Mar 20, 2020
c866a6b
Use pkg http status codes in handlers
tlwr Mar 20, 2020
4e1bad2
Move the reload route count metric inside the defer
tlwr Mar 20, 2020
de41b23
Add make task for checking metrics using promtool
tlwr Mar 20, 2020
84aeed9
Add metric information in README
tlwr Mar 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
26 changes: 22 additions & 4 deletions handlers/backend_handler.go
View file
Open in desktop
Expand Up @@ -16,9 +16,20 @@ import (

var TLSSkipVerify bool

func NewBackendHandler(backendURL *url.URL, connectTimeout, headerTimeout time.Duration, logger logger.Logger) http.Handler {
func NewBackendHandler(
backendID string,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about the value of a backendID if backend URLs are expected to be unique. Could we use the URL as the ID here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't sure if using the URL or the ID was better, and didn't check in the database how they looked.

One side-effect of the backend ID, is that they are easier to type:

router_backend_handler_response_duration_seconds_count{backend_id="search-api"}

Also another "feature" is that if the backend URL is changed in the database, the metric will stay the same. Whether in practice this would ever happen is not something I know.

Presumably the IDs are more consistent across environments (staging/prod) than URLs? This is a question to which I do not know the answer.

What do you think is more ergonomic/intuitive?

backendURL *url.URL,
connectTimeout, headerTimeout time.Duration,
logger logger.Logger,
) http.Handler {

proxy := httputil.NewSingleHostReverseProxy(backendURL)
proxy.Transport = newBackendTransport(connectTimeout, headerTimeout, logger)

proxy.Transport = newBackendTransport(
backendID,
connectTimeout, headerTimeout,
logger,
)

defaultDirector := proxy.Director
proxy.Director = func(req *http.Request) {
Expand Down Expand Up @@ -49,14 +60,21 @@ func populateViaHeader(header http.Header, httpVersion string) {
}

type backendTransport struct {
backendID string

wrapped *http.Transport
logger logger.Logger
}

// Construct a backendTransport that wraps an http.Transport and implements http.RoundTripper.
// This allows us to intercept the response from the backend and modify it before it's copied
// back to the client.
func newBackendTransport(connectTimeout, headerTimeout time.Duration, logger logger.Logger) *backendTransport {
func newBackendTransport(
backendID string,
connectTimeout, headerTimeout time.Duration,
logger logger.Logger,
) *backendTransport {

transport := http.Transport{}

transport.DialContext = (&net.Dialer{
Expand Down Expand Up @@ -98,7 +116,7 @@ func newBackendTransport(connectTimeout, headerTimeout time.Duration, logger log
transport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
}

return &backendTransport{&transport, logger}
return &backendTransport{backendID, &transport, logger}
}

func closeBody(resp *http.Response) {
Expand Down
2 changes: 2 additions & 0 deletions handlers/backend_handler_test.go
View file
Open in desktop
Expand Up @@ -48,6 +48,7 @@ var _ = Describe("Backend handler", func() {
Context("when the backend times out", func() {
BeforeEach(func() {
router = handlers.NewBackendHandler(
"backend-timeout",
backendURL,
timeout, timeout,
logger,
Expand Down Expand Up @@ -76,6 +77,7 @@ var _ = Describe("Backend handler", func() {
Context("when the backend handles the connection", func() {
BeforeEach(func() {
router = handlers.NewBackendHandler(
"backend-handle",
backendURL,
timeout, timeout,
logger,
Expand Down
7 changes: 6 additions & 1 deletion router.go
View file
Open in desktop
Expand Up @@ -159,7 +159,12 @@ func (rt *Router) loadBackends(c *mgo.Collection) (backends map[string]http.Hand
continue
}

backends[backend.BackendID] = handlers.NewBackendHandler(backendURL, rt.backendConnectTimeout, rt.backendHeaderTimeout, rt.logger)
backends[backend.BackendID] = handlers.NewBackendHandler(
backend.BackendID,
backendURL,
rt.backendConnectTimeout, rt.backendHeaderTimeout,
rt.logger,
)
}

if err := iter.Err(); err != nil {
Expand Down