add local monitoring package and a grafana dashboard

[Choraden]

[mmatczuk]

Can we add max latency

[mmatczuk]

Can Go routines and threads be a diagram to see how things change over time

[mmatczuk]

Would be nice to see

• how many request/s we process
• how many connect tunnels we have (possibly by type) i.e. connect +ws
• network level info rx and tx (maybe by client connection - it depends on the use case really)
• nr of client connection

[mmatczuk]

Not exactly sure what we do now but latency should be measured for connect and upgrades separately from normal http requests.

[mmatczuk]

For things that require code changes please add separate issue for 1.2

[Choraden]

I addressed things that did not require code change:

• Response Code Rates diagram has been changed to plot each codes rates separately.
• Latency has been extended with P100 and division into CONNECT and all other request.
• Goroutines diagram has been changed to be a plot now.
• Gothreads remain a number since it rarely changes (having diverse diagrams is good). However, to be able to observe spike a subplot was added. It's visible on the same diagram under the number.

[mmatczuk]

• System bytes does not have Mean and Max unlike others.
• I guess all diagrams would benefit from info especially if other ppl are to use the dashboard
• Should Go Therads be Threads?
• The threads also could have mean max
• In Response Codes you have Mean XXX req/s I guess req can be remove; also, I'm not sure what's the benefit of this information
• I think we could split Response code Rates for 2XX and everything else, this could improve resolution for the latter diagram
• Latencies seems not to go down

• It would be super cool if after make up it printer Grafana URL I could click to open the page

[Choraden]

Should Go Therads be Threads?

I chose to stay with Go Threads to indicate where the metric comes from.

The threads also could have mean max

I doubt if it's possible in this number form. We can always switch to standard plot, but I would stay with the one we have.

I addressed all other comments. Errors diagrams will be rendered even though there is no data. Moreover, round trip errors diagram was changed to show increment instead of rate. It looks much better IMO as you can see exact number of errors that occurred instead of some random 0.1388 errors/s. I'm thinking of doing the same for HTTP requests diagrams.

Take a look at the updated pictures or run dashboard yourself and share what you think @mmatczuk

[mmatczuk]

Nice I agree that adding increments for error codes is a good idea.

Overall LGTM.

[mmatczuk]

" your forward proxy "?

[mmatczuk]

See local/monitoring for more details.