Add observability

[buro9]

Add as an observer middleware and observe all HTTP attributes, this will later be hooked to https://www.outlyer.com/ in production instances, and dev instances could always spin up prometheus and grafana containers to view it.

[yemble]

Sniffing around this one. I've done prometheus from Go on another project so just learning a bit about how chi and middleware work.

[buro9]

I imagine we'd make a simple middleware that calls much of the existing middleware that Prometheus gives us: https://github.com/prometheus/client_golang/blob/master/prometheus/promhttp/http.go

We'd need to configure a gatherer that has the opts we want, specifically Prometheus labels that surface the HTTP Host, HTTP Method, HTTP Status and a normalised version of http.Req.URL.

This way the resulting metrics can be analysed using other systems so that it's possible to plot reports where data is grouped by HTTP Status (and if it's slow we can see that it's for the POSTs rather than the PUTs), and additionally we would know which templated paths were slow, i.e. /microcosms/{id} would be the normalised path for all forums. And of course, those who host multiple forums would need the host to be able to group data by that.

URL normalisation would simply be "find integers within the path part of the URL and replace them with {id}, strip the scheme, host and query so we just have the path".

[buro9]

Now I work at Grafana I want to change this ticket.

• Add Prometheus histograms
• Add logfmt logging
• Add full tracing
• Modify metrics, logging and tracing to enable exemplars (navigating between all three)
• Add support for on-demand profiling