slo - capture plugin duration for http based plugins

[scottlepp]

What this PR does / why we need it:
We want to capture execution time of plugin requests for SLO duration.

Changes here based on our last slo discussion. The original plan was to capture 2 separate histogram metrics, then try to subtract them. The recommendation by the prometheus team was not to do that, but to instead calculate before storing the metric.

How it works:

1. Middleware will capture the overall duration of a request that is spent making external calls and store it in the context.
2. The MetricsWrapper will initialize the Duration in the context or a request and calculate the plugin duration (overall duration - downstream duration)

This requires minimal changes in the plugin:

1. Add the Metrics Middleware (or @marefr should we just add this to the existing middeware chain?)
2. Add the MetricsWrapper during plugin instantiation (though again, it seems we could just always do it from the sdk when instantiating a plugin? )

[marefr]

Quick feedback. Will take a closer look tomorrow.

Yes, I think it make sense to make the middleware default. Would be good with tests of the wrapper and/or example how you would use it, plugin have to create it or SDK automatically provides it?

The middleware/duration doesn't look to be concurrency safe - that's something we probably have to have before making Middleware default. Tests for middleware would be good as well.

[scottlepp]

@marefr some updates from your pr review. addressed concurrency and tests

[marefr]

LGTM since experimental and want to unblock you. Some nits, feel free to address those or skip :)

[marefr]

Nit. Couldn't these live in the package slo_test as well?

[marefr]

Nit. Wonder if status_code is really needed, leave that up to you. But in general, feels like logs could be more suitable for details following our discussion from last week.