Record Jetty bytes in/out metrics through network listeners

[marcingrzejszczak]

before this change we were dumping in / out bytes from the connection upon the connection close which could result in big traffic whenever the connection got closed after this change we're measuring the bytes through NetworkTrafficListener that hooks in whenever actual in / out bytes are being processed

fixes gh-3873

[shakuzen]

I'm not sure about the difference in the number of bytes counted. It would be good to understand that. The description of the metrics should be updated as well.

[shakuzen]

This looks like new setup that isn't documented in the class JavaDoc or done in the static addToAllConnectors method. It will be a behavior change that the bytes in/out will only be available if this is set, right?

[marcingrzejszczak]

Yes, that's correct plus I'm also showing this in the tests that bytes are not equal to what they were used to. So not only do you need to set it up differently but also the result is different

[shakuzen]

@joakime do you have some time to help us understand this? When changing from counting the bytes in/out on the connection to a network listener and we're getting different numbers.

[joakime]

Depending on where the integration point is in the handler tree you will either see the bytes as initiated by the webapp, or the bytes as sent on the network.

The most obvious and largest difference would be if a GzipHandler is in place.
You could be getting the bytes as seen before the compression or after the compression.

The techniques that micrometer were using in Jetty 9/10/11 were only seeing the bytes passing through the HttpOutput / HttpInput classes.
Which is not network bytes.

Using the org.eclipse.jetty.io.NetworkTrafficListener you will get everything on the network.

• The Headers
• The Body
• The Transfer-Encoding differences (like chunking overhead)
• The HTTP Trailers
• The TLS negotiation
• The TLS overhead
• The HTTP protocol overhead (like in HTTP/2)

While in the techniques you used in Jetty 9/10/11 on micrometer you saw only the Body before Transfer-Encoding.

[shakuzen]

Thanks for the explanation. One effect of changing from counting the bytes via Connection.Listener to NetworkTrafficListener seems to be the loss of the ability to count the bytes in/out for HttpClient, because there's no way to register a NetworkTrafficListener on an HttpClient as far as I can tell; it can only be registered on the server-side. Is that right?

[joakime]

I agree, it would be nice to have HttpClient also participate in NetworkTrafficListener ...

So I opened jetty/jetty.project#11491 to do just that!

[shakuzen]

Thank you for that. I've subscribed to the issue. We can update our tests/documentation when that is available.