Add Prometheus metrics for number of log messages

[marccarre]

Promrus is a Logrus hook to expose the number of log messages as Prometheus metrics:

log_messages{level="debug"}
log_messages{level="info"}
log_messages{level="warning"}
log_messages{level="error"}

Added dependency using:

$ dep ensure github.com/weaveworks/promrus@v1.0.0 && dep ensure && dep prune

Fixed casing in logrus imports, causing the following issue:

middleware/grpc_logging.go:10:2: case-insensitive import collision:
"github.com/weaveworks/common/vendor/github.com/Sirupsen/logrus" and
"github.com/weaveworks/common/vendor/github.com/sirupsen/logrus"

by:

• changing import statements from Sirupsen/logrus to sirupsen/logrus, as per the official documentation, and
• running dep ensure && dep prune.

[jml]

I'm not 100% sure about the Sirupsen case thing. I got bit by it a while ago. I suggest trying to use this branch of common in a project with executables before merging it.

[marccarre]

What do you mean by not sure?
Do you think it could cause issues to change it?
Is there some sort of a convention I missed regarding this?

I initially did not realise this would lead to such changes when I added sirupsen/logrus to my dependencies in promrus by simply following their documentation. I then realised there was this casing difference here, but also just saw, for example, that Cortex's dependencies are also using Sirupsen/logrus, so if I keep promrus as is, I believe it might lead to more build issues, case changes (I don't mind making these changes if they are painless), or even preventing metrics from being exposed since Go might treat them as different packages altogether (?).
Alternatively, I could just change promrus to use Sirupsen/logrus instead of sirupsen/logrus.

[marccarre]

In any case, I'll try it against Cortex and will report back.

[jml]

Not sure as in "I did something like this a few months ago and regretted it but can't recall the details". It might well break things.

[marccarre]

FYI, looks like we aren't the only ones facing this issue, see here and here, and that the way forward is to use lowercase. I'll bite the bullet and make the necessary changes where we decide to use promrus.

[marccarre]

I haven't tried on Cortex yet, but trying on a reduction of the problem (i.e. a smaller project mimicking its dependencies) shows that:

1. having mixed casing in vendor definitely breaks the build,
2. even if you only have sirupsen/logrus in vendor (e.g.: dep ensure and then rm -fr vendor/github.com/Sirupsen), instrumenting on sirupsen/logrus does not instrument Sirupsen/logrus:

foo.go in project foo:

package foo

import (
	log "github.com/Sirupsen/logrus"
)

// Sirupsen sirupsens.
func Sirupsen() {
	log.Infof("Sirupsen/logrus")
}

main.go in a different project:

package main

import (
	"net/http"
	"time"

	"github.com/marccarre/promrus"
	"github.com/marccarre/foo"
	"github.com/prometheus/client_golang/prometheus/promhttp"
	log "github.com/sirupsen/logrus"
)

func main() {
	hook, err := promrus.NewPrometheusHook()
	if err != nil {
		return
	}
	log.AddHook(hook)
	go http.ListenAndServe(":8080", promhttp.Handler())
	for {
		foo.Sirupsen()
		time.Sleep(10 * time.Second)
	}
}

yields metrics which do not increase over time:

$ curl -fsS localhost:8080 | grep log_messages
# HELP log_messages Total number of log messages.
# TYPE log_messages counter
log_messages{level="debug"} 0
log_messages{level="error"} 0
log_messages{level="info"} 0
log_messages{level="warning"} 0

[bboreham]

Why do we even want this?

[marccarre]

What do you mean by "this"?

• Adding metrics was suggested on this issue: https://github.com/weaveworks/service-conf/issues/977#issuecomment-333528506
• Lower-casing was suggested on this issue: Rename back to Sirupsen/logrus sirupsen/logrus#570 (comment) and some of our dependencies also have moved to this convention.

Feel free to revert if causing issues though.