A80: gRPC Metrics for TCP connection #428

nanahpang · 2024-04-22T18:15:24Z

Add the proposal of gRPC TCP per-connection metrics.

yousukseung

LGTM to me as discussed offline, I'll leave others to approve.

ctiller · 2024-04-23T17:24:01Z

A80-grpc-metrics-for-tcp-connection

+
+| Name        | Disposition | Description |
+| ----------- | ----------- | ----------- |
+| grpc.tcp.remote_peer_address | optional | Store the peer address info in the format as `ip:port`. |


@markdroth @yashykt I feel like there's an equivalent label already in use?

This could be a relatively expensive fan-out for a busy server

Agreed, we discussed about this offline. Is there a way to support an opaque user-defined peer info string? In prod, by default we only record the cell information due to the fan-out concern.

We need to figure out the right format here

I propose URI form, such as ipv4:1.2.3.4:567. @ejona86 @dfawley, would that be a problem for Java and Go if we ever added these metrics in your languages?

I think this should be clarified more. For instance, is ipv4:///1.2.3.4:567 also OK? What is the format for ipv6? What other schemes do we support? @ejona86 is concerned that ipv4:1.2.3.4:567 is not a URI we should support in the first place. So maybe this should just be specified fully as strictly:

- ipv4:<ipv4-address>:<port-number> OR - ipv6:[<ipv6-address>]:<port-number>

ctiller · 2024-04-23T17:24:54Z

A80-grpc-metrics-for-tcp-connection

+
+| Name          | Type  | Unit  | Labels  | Description |
+| ------------- | ----- | ----- | ------- | ----------- |
+| grpc.tcp.min_rtt | Distribution | s | grpc.tcp.remote_peer_string | Reports TCP's current estimate of minimum round trip time (RTT), typically used as an indication of the network health between two endpoints. |


@yashykt how well does seconds here align with other metrics we have?

We are following OpenTelemetry's conventions mostly which advocates for seconds 's' as the unit for duration

ctiller · 2024-04-23T17:25:42Z

A80-grpc-metrics-for-tcp-connection

+| grpc.tcp.delivery_rate | Distribution | bit/s | grpc.tcp.remote_peer_string | Records the most recent non-app-limited throughput at the time that Fathom samples the connection statistics. |
+| grpc.tcp.packets_sent | Counter | {packet} | grpc.tcp.remote_peer_string | Records total packets TCP sends in the calculation period. |
+| grpc.tcp.packets_retransmitted | Counter | {packet} | grpc.tcp.remote_peer_string | Records total packets lost in the calculation period, including lost or spuriously retransmitted packets. |
+| grpc.tcp.packets_spurious_retransmitted | Counter | {packet} | grpc.tcp.remote_peer_string | Records total packets spuriously retransmitted packets in the calculation period. |


I'm not sure what spurious means in this context... maybe we could link to some write-up, or include some more text defining these things?

I could not find an authoritative source we can link, but I think we can include what's on go/fathom-metrics i.e. "These are retransmissions that TCP later discovered unnecessary.". The definition is public.

Sounds good, I will add that in the description.

atollena · 2024-04-24T08:22:07Z

A80-grpc-metrics-for-tcp-connection

+
+| Name          | Type  | Unit  | Labels  | Description |
+| ------------- | ----- | ----- | ------- | ----------- |
+| grpc.tcp.min_rtt | Distribution | s | grpc.tcp.remote_peer_string | Reports TCP's current estimate of minimum round trip time (RTT), typically used as an indication of the network health between two endpoints. |


A couple of remarks on this:

Do you know how portable this is going to be? It would help to explain how you plan to collect that data. Do you plan to use getsockopt + TCP_INFO that yields a min_rtt in some cases? Even on linux, I'm not sure all TCP algorithms measure the minimum RTT.

What is a Distribution type? Do you mean a histogram?

Since this is a property of the transport, this probably belongs under grpc.transport.tcp.

I think min_rtt is general enough that we can include, I presume most congestion control tracks RTT or equivalent to keep track of the estimated BDP.

And I think this gRFC is supposed to cover the definition of the metric and not about the collection method. @yashykt Yes, in Linux we can get it via either getsockopt(TCP_INFO) or TCP_NLA_MIN_RTT from cmsg when timestamping is enabled.

I think min_rtt is general enough that we can include, I presume most congestion control tracks RTT or equivalent to keep track of the estimated BDP.

OK, and even if it weren't the case, there is probably no problem with omitting it.

And I think this gRFC is supposed to cover the definition of the metric and not about the collection method.

OK, yes. My point is more that the definition needs to be precise enough so that different implementations measure the same thing.

I think if it's important for the correctness of the metrics, it makes sense to give more information on this metric. Maybe you could say that a valid implementation of this is through getsockopt?

I added a summary of how the metrics are collected and linked some references. Let me know if anything is unclear. Thanks.

atollena · 2024-04-24T08:24:38Z

A80-grpc-metrics-for-tcp-connection

+| Name          | Type  | Unit  | Labels  | Description |
+| ------------- | ----- | ----- | ------- | ----------- |
+| grpc.tcp.min_rtt | Distribution | s | grpc.tcp.remote_peer_string | Reports TCP's current estimate of minimum round trip time (RTT), typically used as an indication of the network health between two endpoints. |
+| grpc.tcp.delivery_rate | Distribution | bit/s | grpc.tcp.remote_peer_string | Records the most recent non-app-limited throughput at the time that Fathom samples the connection statistics. |


What is Fathom? A quick internet search yields https://dl.acm.org/doi/pdf/10.1145/3603269.3604815 which sounds related (but I have not read it)... Do you think anything that uses it is going to useful to anyone outside Google?

No, and app-limitedness is also not a general property either. I think we should update to something like "Latest throughput measured of the TCP connection."

Sounds good, updated.

A80-grpc-metrics-for-tcp-connection

yashykt · 2024-04-27T00:27:12Z

A80-grpc-metrics-for-tcp-connection

+
+| Name        | Disposition | Description |
+| ----------- | ----------- | ----------- |
+| grpc.tcp.remote_peer_address | optional | Store the peer address info in the format as `ip:port`. |


yashykt · 2024-04-27T00:27:40Z

A80-grpc-metrics-for-tcp-connection

+
+| Name          | Type  | Unit  | Labels  | Description |
+| ------------- | ----- | ----- | ------- | ----------- |
+| grpc.tcp.min_rtt | Distribution | s | grpc.tcp.remote_peer_string | Records TCP's current estimate of minimum round trip time (RTT), typically used as an indication of the network health between two endpoints. |


I think maybe we should add the remote_string and maybe the local_string as optional labels on these metrics

Sounds good, updated.

yashykt · 2024-04-27T00:28:47Z

A80-grpc-metrics-for-tcp-connection

+
+| Name          | Type  | Unit  | Labels  | Description |
+| ------------- | ----- | ----- | ------- | ----------- |
+| grpc.tcp.min_rtt | Distribution | s | grpc.tcp.remote_peer_string | Reports TCP's current estimate of minimum round trip time (RTT), typically used as an indication of the network health between two endpoints. |


We are following OpenTelemetry's conventions mostly which advocates for seconds 's' as the unit for duration

yashykt · 2024-04-27T00:30:45Z

A80-grpc-metrics-for-tcp-connection

+
+| Name          | Type  | Unit  | Labels  | Description |
+| ------------- | ----- | ----- | ------- | ----------- |
+| grpc.tcp.min_rtt | Distribution | s | grpc.tcp.remote_peer_string | Reports TCP's current estimate of minimum round trip time (RTT), typically used as an indication of the network health between two endpoints. |


I think if it's important for the correctness of the metrics, it makes sense to give more information on this metric. Maybe you could say that a valid implementation of this is through getsockopt?

yashykt

(approved by mistake)

markdroth · 2024-04-29T17:11:18Z

A80-grpc-metrics-for-tcp-connection

+
+| Name        | Disposition | Description |
+| ----------- | ----------- | ----------- |
+| grpc.tcp.remote_peer_address | optional | Store the peer address info in the format as `ip:port`. |


I propose URI form, such as ipv4:1.2.3.4:567. @ejona86 @dfawley, would that be a problem for Java and Go if we ever added these metrics in your languages?

A80-grpc-metrics-for-tcp-connection

…rics-for-tcp-connection.md

markdroth

Need review from @ejona86 and @dfawley to make sure these definitions are still okay in terms of potential future implementation in other languages.

A80-grpc-metrics-for-tcp-connection.md

markdroth · 2024-05-10T19:54:24Z

A80-grpc-metrics-for-tcp-connection.md

+| ------------- | ----- | ----- | ------- | ----------- |
+| grpc.tcp.min_rtt | Histogram (double) | s | grpc.tcp.peer_address, grpc.tcp.local_address | Records TCP's current estimate of minimum round trip time (RTT), typically used as an indication of the network health between two endpoints.  |
+| grpc.tcp.delivery_rate | Histogram (double) | bit/s | grpc.tcp.peer_address, grpc.tcp.local_address | Records latest throughput measured of the TCP connection. |
+| grpc.tcp.packets_sent | Counter (int64) | {packet} | grpc.tcp.peer_address, grpc.tcp.local_address | Records total packets TCP sends in the calculation period. |


@yashykt What types should we be defining here?

I updated the Counter type to uint64 based on the register methods in metrics.h.
@yashykt Feel free to modify if it will use other types.

int64 and double are very C/C++ specific. I prefer integer and floating-point to keep it generic.

Done, thanks for the suggestion.

A80-grpc-metrics-for-tcp-connection.md

atollena · 2024-05-24T07:25:24Z

A80-grpc-metrics-for-tcp-connection.md

+
+This document proposes changes to the following gRPC components.
+
+#### Per-Connection TCP Metrics


nit: this can be ### instead of ####.

Done, thanks for the suggestion.

atollena · 2024-05-24T07:30:05Z

A80-grpc-metrics-for-tcp-connection.md

+
+| Name          | Type  | Unit  | Labels  | Description |
+| ------------- | ----- | ----- | ------- | ----------- |
+| grpc.tcp.min_rtt | Histogram (double) | s | None | Records TCP's current estimate of minimum round trip time (RTT), typically used as an indication of the network health between two endpoints.<br /> RTT = packet acked timestamp - packet sent timestamp. |


Buckets for histogram are explicitly specified for request latency in https://github.com/grpc/proposal/blob/master/A66-otel-stats.md. Do you plan to reuse the same buckets? It might be worth specifying.

Thanks for the suggestion, but using buckets for the min_rtt metric wouldn't offer much significant insights. The value of min_rtt is bound to the physical length of the path between sender and receiver or the queueing time caused by throttling and load. A step-change in min_rtt values usually means that traffic is being throttled or experiencing congestion, or has been re-routed through a different path. I think it's better to opt out buckets in this situation.

@nanahpang If you do not specify buckets, we will get the default OpenTelemetry buckets which is { 0, 5, 10, 25, 50, 75,100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000}. Are there better defaults that we can suggest or do we leave it to the user to figure it out?

Regarding OpenTelemetry buckets, if you're referring to a plugin used for exporting metrics, that would be a separate concern from the metric types we're defining here. However, I agree that specifying a smaller range for the min_rtt metric buckets is the right approach, especially since we're using seconds as the unit (to align with the open-source standard) while the underlying measurements are usually in microseconds.

atollena · 2024-05-24T07:53:22Z

A80-grpc-metrics-for-tcp-connection.md

+A high-level approach to collecting TCP metrics (on Linux) is as follows:
+1) **Enable Network Timestamps for Metric Calculation:** Enable the `SO_TIMESTAMPING` option in the kernel's TCP stack through the `setsocketopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val))` system call. This enables the kernel to capture packet timestamps during transmission.
+2) **Calculate Metrics from Timestamps:**  Linux kernel calculates TCP connection metrics based on the captured packet timestamps. These metrics can be retrieved using the `getsockopt(TCP_INFO)` system call. For example, the delivery_rate metric estimates the goodput—the rate of useful data transmitted—for the most recent group of outbound data packets within a single flow ([code](https://elixir.bootlin.com/linux/v5.11.1/source/net/ipv4/tcp.c#L391)).
+3) **Periodically Collect Statistics:** At a specified time interval (e.g., every 5 minutes), gRPC aggregates the calculated metrics and updates the corresponding statistics records.


That could be a bit more precise. I have 3 questions:

Do you plan to have this interval be specified by users somehow? Or should it be fixed to 5 minutes? 5 minutes seems long to me for updating metrics, ideally it should be something close to the collection period (which iiuc is more typically in the 1 minute).

How should the aggregation logic look like? I imagine you have something like this in mind:

for counters, iterate over sockets and sum up packets_sent, packets_restransmitted of all open sockets.

for histograms, for each socket, record the value (delivery_rate & min_rtt) in the histogram for the corresponding metric.

That's the only thing that really makes sense given the definition of the metrics above, so perhaps it's fine to leave it implicit.

What happens to sockets that have been closed in between the interval? Do we just not collect statistics for those and loose the data? I'm not sure that there is an alternative that works, such as calling getsockopts and updating statistics just before closing the socket.

I think in C++ the interval is not fixed, and the default value is 5 minutes in Fathom. For other language implementations, the interval can be adjusted as needed. @yousukseung for other questions of the Fathom implementation. Thanks.

For context, this high-level plan aims to provide a general understanding of the existing metric collection process in C++ (implemented through Fathom), while offering flexibility for adaptation in other languages. To maintain clarity and focus, implementation details have been omitted from this proposal and can be found in the Fathom documentation.

yashykt · 2024-05-24T22:06:00Z

A80-grpc-metrics-for-tcp-connection.md

+
+| Name          | Type  | Unit  | Labels  | Description |
+| ------------- | ----- | ----- | ------- | ----------- |
+| grpc.tcp.min_rtt | Histogram (double) | s | None | Records TCP's current estimate of minimum round trip time (RTT), typically used as an indication of the network health between two endpoints.<br /> RTT = packet acked timestamp - packet sent timestamp. |


@nanahpang If you do not specify buckets, we will get the default OpenTelemetry buckets which is { 0, 5, 10, 25, 50, 75,100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000}. Are there better defaults that we can suggest or do we leave it to the user to figure it out?

ejona86 · 2024-05-24T22:20:37Z

A80-grpc-metrics-for-tcp-connection.md

+A high-level approach to collecting TCP metrics (on Linux) is as follows:
+1) **Enable Network Timestamps for Metric Calculation:** Enable the `SO_TIMESTAMPING` option in the kernel's TCP stack through the `setsocketopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val))` system call. This enables the kernel to capture packet timestamps during transmission.
+2) **Calculate Metrics from Timestamps:**  Linux kernel calculates TCP connection metrics based on the captured packet timestamps. These metrics can be retrieved using the `getsockopt(TCP_INFO)` system call. For example, the delivery_rate metric estimates the goodput—the rate of useful data transmitted—for the most recent group of outbound data packets within a single flow ([code](https://elixir.bootlin.com/linux/v5.11.1/source/net/ipv4/tcp.c#L391)).
+3) **Periodically Collect Statistics:** At a specified time interval (e.g., every 5 minutes), gRPC aggregates the calculated metrics and updates the corresponding statistics records.


"At a specified time interval" means the interval will be configurable. How is it configured?

ejona86 · 2024-05-24T22:32:36Z

A80-grpc-metrics-for-tcp-connection.md

+| ------------- | ----- | ----- | ------- | ----------- |
+| grpc.tcp.min_rtt | Histogram (double) | s | None | Records TCP's current estimate of minimum round trip time (RTT), typically used as an indication of the network health between two endpoints.<br /> RTT = packet acked timestamp - packet sent timestamp. |
+| grpc.tcp.delivery_rate | Histogram (double) | bit/s | None | Records latest goodput measured of the TCP connection. <br /> Elapsed time = packet acked timestamp - last packet acked timestamp. <br /> Delivery rate = packet acked bytes / elapsed time. |
+| grpc.tcp.packets_sent | Counter (uint64) | {packet} | None | Records total packets TCP sends in the calculation period. |


What is a "calculation period"?

ejona86 · 2024-05-24T22:33:37Z

A80-grpc-metrics-for-tcp-connection.md

+
+A high-level approach to collecting TCP metrics (on Linux) is as follows:
+1) **Enable Network Timestamps for Metric Calculation:** Enable the `SO_TIMESTAMPING` option in the kernel's TCP stack through the `setsocketopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val))` system call. This enables the kernel to capture packet timestamps during transmission.
+2) **Calculate Metrics from Timestamps:**  Linux kernel calculates TCP connection metrics based on the captured packet timestamps. These metrics can be retrieved using the `getsockopt(TCP_INFO)` system call. For example, the delivery_rate metric estimates the goodput—the rate of useful data transmitted—for the most recent group of outbound data packets within a single flow ([code](https://elixir.bootlin.com/linux/v5.11.1/source/net/ipv4/tcp.c#L391)).


I'd still like to know what the mapping is for each metric. The first few are easy because the names here mirror tcp_info (I assume). But then it gets less obvious, and tcp_info isn't documented very well.

Create A80-grpc-metrics-for-tcp-connection

dedfb16

nanahpang requested review from yashykt, ctiller, markdroth and yousukseung April 22, 2024 18:15

yousukseung reviewed Apr 22, 2024

View reviewed changes

ctiller reviewed Apr 23, 2024

View reviewed changes

Update A80-grpc-metrics-for-tcp-connection

ffaeb22

atollena reviewed Apr 24, 2024

View reviewed changes

nanahpang added 2 commits April 24, 2024 13:36

Update A80-grpc-metrics-for-tcp-connection

d413291

Update A80-grpc-metrics-for-tcp-connection

5b5ba3f

yashykt approved these changes Apr 27, 2024

View reviewed changes

yashykt requested changes Apr 27, 2024

View reviewed changes

markdroth reviewed Apr 29, 2024

View reviewed changes

markdroth changed the title ~~Create A80-grpc-metrics-for-tcp-connection~~ A80: gRPC Metrics for TCP connection Apr 29, 2024

markdroth reviewed Apr 29, 2024

View reviewed changes

A80-grpc-metrics-for-tcp-connection Outdated Show resolved Hide resolved

nanahpang added 5 commits April 29, 2024 14:25

Update and rename A80-grpc-metrics-for-tcp-connection to A80-grpc-met…

583e6b3

…rics-for-tcp-connection.md

Update A80-grpc-metrics-for-tcp-connection.md

8aa21c1

Update A80-grpc-metrics-for-tcp-connection.md

9f8038c

Update A80-grpc-metrics-for-tcp-connection.md

59ab138

Update A80-grpc-metrics-for-tcp-connection.md

ce27a69

nanahpang requested review from atollena, yousukseung, markdroth, ctiller and yashykt May 2, 2024 20:12

markdroth reviewed May 10, 2024

View reviewed changes

Update A80-grpc-metrics-for-tcp-connection.md

d239c39

nanahpang requested review from ejona86 and dfawley and removed request for atollena May 10, 2024 20:32

yashykt approved these changes May 15, 2024

View reviewed changes

Update A80-grpc-metrics-for-tcp-connection.md

0726f6e

markdroth reviewed May 15, 2024

View reviewed changes

A80-grpc-metrics-for-tcp-connection.md Outdated Show resolved Hide resolved

Update A80-grpc-metrics-for-tcp-connection.md

3bfe76b

yousukseung approved these changes May 16, 2024

View reviewed changes

nanahpang added 6 commits May 21, 2024 16:30

Update A80-grpc-metrics-for-tcp-connection.md

2ccf768

Update A80-grpc-metrics-for-tcp-connection.md

83ac908

Update A80-grpc-metrics-for-tcp-connection.md

2a11aea

Update A80-grpc-metrics-for-tcp-connection.md

b6dc6d9

Update A80-grpc-metrics-for-tcp-connection.md

0aceebe

Update A80-grpc-metrics-for-tcp-connection.md

052d5cf

nanahpang requested review from atollena, yashykt and markdroth May 23, 2024 23:08

atollena reviewed May 24, 2024

View reviewed changes

nanahpang added 2 commits May 24, 2024 11:15

Update A80-grpc-metrics-for-tcp-connection.md

7e5bc86

Update A80-grpc-metrics-for-tcp-connection.md

092fbc1

yashykt approved these changes May 24, 2024

View reviewed changes

Update A80-grpc-metrics-for-tcp-connection.md

bd18940

ejona86 reviewed May 24, 2024

View reviewed changes


		This document proposes changes to the following gRPC components.

		#### Per-Connection TCP Metrics

A80: gRPC Metrics for TCP connection #428

Are you sure you want to change the base?

A80: gRPC Metrics for TCP connection #428

Conversation

nanahpang commented Apr 22, 2024 • edited

yousukseung left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nanahpang May 2, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yashykt Apr 27, 2024 • edited

Choose a reason for hiding this comment

nanahpang Apr 29, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yashykt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markdroth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nanahpang May 24, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nanahpang commented Apr 22, 2024 •

edited

nanahpang May 2, 2024 •

edited

yashykt Apr 27, 2024 •

edited

nanahpang Apr 29, 2024 •

edited

nanahpang May 24, 2024 •

edited