Fixing Websocket Connections when Prometheus Metrics Enabled

[onelapahead]

• Uses my fork https://gitlab.com/hfuss/mux-prometheus (opened a PR against the original: https://gitlab.com/msvechla/mux-prometheus/-/merge_requests/11) so that the necessary Hijack method is implemented in the wrapper around ResponseWriter in order to fix Websocket: response does not implement http.Hijacker - when metrics enabled #370
• Enables Prometheus metrics when running E2E tests so that any bugs its middleware is introducing are noticed
• Adds the ff_ prefix to the middleware metrics
• Ran go mod tidy

[nguyer]

Thanks @hfuss! Looks good to me after tests and coverage passes.

[onelapahead]

Thanks @nguyer! @eberger727 and I will ensure E2E is fixed w/ metrics enabled and then I can cleanup the coverage / unit tests.

[peterbroadhurst]

Would like to understand why can't just put our own middleware in, that wraps the prometheus middleware, rather than the code copy

[codecov-commenter]

Codecov Report

Merging #371 (49b6e8d) into main (a3b5bd3) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##              main      #371   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          239       239           
  Lines        12886     12886           
=========================================
  Hits         12886     12886           

Impacted Files	Coverage Δ
internal/apiserver/server.go	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a3b5bd3...49b6e8d. Read the comment docs.

[onelapahead]

@peterbroadhurst this didn't work as expected because the inherited ResponseWriter for websocketResponseWriter ended up being a statusResponseWriter which does not implement Hijacker.

I tried climbing the "inheritance stack" with something like the following since statusResponseWriter is private:

type wrapperResponseWriter struct {
   http.ResponseWriter
}
// ...
func (w *websocketResponseWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) {
	h, ok := w.ResponseWriter.(wrapperResponseWriter).ResponseWriter.(http.Hijacker)
        // ...
}

but that didn't work either so decided it was easier to fork the library with the fix and have FireFly use that instead.

[peterbroadhurst]

Sorry this turned into more work, but this seems like a great solution 👍