Use netip.Addr for FlowExporter #5532

antoninbas · 2023-09-27T20:54:22Z

Replace net.IP with netip.Addr in FlowExporter implementation

The net/netip package was introduced in Go 1.18. Compared to the
existing net.IP type, the netip.Addr type takes less memory (especially
for IPv6 addresses), is immutable, and is comparable so it supports ==
and can be used as a map key.

In our case, this means that the flowexporter.Connection type takes a
little less memory, and that the flowexporter.Tuple can be used directly
as the flowexporter.ConnectionKey (no need to generate a string for use
as the map key). Overall, this leads to higher performance (decreased
CPU and memory usage) for the FlowExporter.

There is potential for further improvement, as several key functions
take as parameters both the flowexporter.Connection and
flowexporter.ConnectionKey objects, which is no longer really required.

antoninbas · 2023-09-27T20:54:56Z

Benchmark results:

CPU (ns/op) 

|                                    | net.IP    | netip.Addr | change |
| ---------------------------------- | --------- | ---------- | ------ |
| BenchmarkPoll-IPv4                 | 14206994  | 11121690   | -21.7% |
| BenchmarkPoll-IPv6                 | 16447742  | 10823494   | -34.2% |
| BenchmarkConnStore-IPv4            | 116750728 | 103335897  | -11.5% |
| BenchmarkConnStore-IPv6            | 121885558 | 109426883  | -10.2% |
| BenchmarkExportConntrackConns-IPv4 | 341007    | 282496     | -17.1% |
| BenchmarkExportConntrackConns-IPv6 | 355618    | 283956     | -20.2% |
| BenchmarkExportDenyConns-IPv4      | 336936    | 290783     | -13.7% |
| BenchmarkExportDenyConns-IPv6.     | 356949    | 290070     | -18.7% |

Memory (B/op) 

|                                    | net.IP    | netip.Addr | change |
| ---------------------------------- | --------- | ---------- | ------ |
| BenchmarkPoll-IPv4                 | 1330151   | 970550     | -27.0% |
| BenchmarkPoll-IPv6                 | 2056737   | 1034764    | -49.7% |
| BenchmarkConnStore-IPv4            | 20491862  | 17934887   | -12.5% |
| BenchmarkConnStore-IPv6            | 22041995  | 18814942   | -14.6% |
| BenchmarkExportConntrackConns-IPv4 | 38377     | 33077      | -13.8% |
| BenchmarkExportConntrackConns-IPv6 | 41242     | 35361      | -14.3% |
| BenchmarkExportDenyConns-IPv4      | 38035     | 32729      | -14.0% |
| BenchmarkExportDenyConns-IPv6.     | 40934     | 34732      | -15.2% |

Memory (allocs/op) 

|                                    | net.IP    | netip.Addr | change |
| ---------------------------------- | --------- | ---------- | ------ |
| BenchmarkPoll-IPv4                 | 58299     | 26144      | -55.0% |
| BenchmarkPoll-IPv6                 | 58294     | 26143      | -55.0% |
| BenchmarkConnStore-IPv4            | 272636    | 240475     | -11.8% |
| BenchmarkConnStore-IPv6            | 272634    | 240475     | -11.8% |
| BenchmarkExportConntrackConns-IPv4 | 407       | 348        | -14.5% |
| BenchmarkExportConntrackConns-IPv6 | 407       | 348        | -14.5% |
| BenchmarkExportDenyConns-IPv4      | 427       | 348        | -18.5% |
| BenchmarkExportDenyConns-IPv6.     | 427       | 348        | -18.5% |

antoninbas · 2023-09-28T01:06:22Z

I'm putting this on hold. While the synthetic benchmarks look good, I think real-life usage won't show too much improvement, given that https://github.com/ti-mo/conntrack uses net.IP. I will open an issue to see if there is interest in switching to netip.Addr, but I am not sure the project is actively maintained.

yuntanghsu · 2023-09-28T17:20:30Z

pkg/agent/flowexporter/connections/conntrack.go

@@ -26,15 +27,24 @@ import (

 // InitializeConnTrackDumper initializes the ConnTrackDumper interface for different OS and datapath types.
 func InitializeConnTrackDumper(nodeConfig *config.NodeConfig, serviceCIDRv4 *net.IPNet, serviceCIDRv6 *net.IPNet, ovsDatapathType ovsconfig.OVSDatapathType, isAntreaProxyEnabled bool) ConnTrackDumper {
+	var svcCIDRv4, svcCIDRv6 netip.Prefix


Do you think it is better to change svcCIDRv4/v6 in agent.go?

There is no need to. While I think that in the long term, we could replace all occurrences of net.IP with netip.Addr (and net.IPNet with netip.Prefix), it's better to do it incrementally with small PRs, starting with places providing the bigger benefits (in terms of performance). The configuration CIDRs for the Agent (podCIDR, svcCIDR) are individual values and don't account for any memory usage (while the FlowExporter connection store can potentially holds 1000s of IP addresses).

antoninbas · 2023-10-16T21:26:56Z

@yuntanghsu I'm re-opening this for review. I worked with the maintainer from the conntrack library and it now uses netip.Addr instead of net.IP in the latest release.

yuntanghsu

LGTM, only one nit

pkg/agent/flowexporter/connections/conntrack_linux_test.go

antoninbas · 2023-10-20T20:26:32Z

/test-all

antoninbas · 2023-10-30T18:32:07Z

@tnqn @heanlan reminder in case you want to take a look, now that v1.14 has been released
I plan to merge this some time this week if no further comments

pkg/agent/flowexporter/connections/conntrack.go

pkg/agent/flowexporter/connections/conntrack_others.go

pkg/agent/flowexporter/connections/conntrack_connections_perf_test.go

Also add BenchmarkConnStore, meant to measure memory usage of the connection store (and all the connections). Signed-off-by: Antonin Bas <abas@vmware.com>

The net/netip package was introduced in Go 1.18. Compared to the existing net.IP type, the netip.Addr type takes less memory (especially for IPv6 addresses), is immutable, and is comparable so it supports == and can be used as a map key. In our case, this means that the flowexporter.Connection type takes a little less memory, and that the flowexporter.Tuple can be used directly as the flowexporter.ConnectionKey (no need to generate a string for use as the map key). Overall, this leads to higher performance (decreased CPU and memory usage) for the FlowExporter. There is potential for further improvement, as several key functions take as parameters both the flowexporter.Connection and flowexporter.ConnectionKey objects, which is no longer really required. See antrea-io#5271 Signed-off-by: Antonin Bas <abas@vmware.com>

This release uses netip.Addr instead of net.IP to represent connections. Signed-off-by: Antonin Bas <abas@vmware.com>

Signed-off-by: Antonin Bas <abas@vmware.com>

tnqn

LGTM

antoninbas · 2023-11-01T17:36:40Z

/test-all

antoninbas · 2023-11-01T19:48:43Z

I added the action/release-note label. I think it may be worth mentioning because this should decrease memory usage in the FlowExporter.

antoninbas requested review from tnqn, yuntanghsu and heanlan September 27, 2023 20:55

antoninbas force-pushed the use-netip-for-flow-exporter branch 2 times, most recently from afb208b to 07a20f7 Compare September 28, 2023 00:42

antoninbas marked this pull request as draft September 28, 2023 01:06

yuntanghsu reviewed Sep 28, 2023

View reviewed changes

antoninbas force-pushed the use-netip-for-flow-exporter branch from 07a20f7 to 9a67f0b Compare October 16, 2023 20:58

antoninbas marked this pull request as ready for review October 16, 2023 21:26

antoninbas mentioned this pull request Oct 18, 2023

Bump github.com/ti-mo/conntrack from 0.4.0 to 0.5.0 #5587

Closed

yuntanghsu previously approved these changes Oct 18, 2023

View reviewed changes

pkg/agent/flowexporter/connections/conntrack_linux_test.go Outdated Show resolved Hide resolved

antoninbas dismissed yuntanghsu’s stale review via ec1239b October 18, 2023 18:38

antoninbas force-pushed the use-netip-for-flow-exporter branch from 9a67f0b to ec1239b Compare October 18, 2023 18:38

antoninbas requested a review from yuntanghsu October 18, 2023 21:25

yuntanghsu previously approved these changes Oct 18, 2023

View reviewed changes

antoninbas added this to the Antrea v1.15 release milestone Oct 24, 2023

tnqn reviewed Oct 31, 2023

View reviewed changes

pkg/agent/flowexporter/connections/conntrack.go Outdated Show resolved Hide resolved

pkg/agent/flowexporter/connections/conntrack_others.go Outdated Show resolved Hide resolved

heanlan reviewed Oct 31, 2023

View reviewed changes

pkg/agent/flowexporter/connections/conntrack_connections_perf_test.go Outdated Show resolved Hide resolved

antoninbas added 5 commits October 31, 2023 17:28

Improvements to FlowExporter benchmarks

9988722

Also add BenchmarkConnStore, meant to measure memory usage of the connection store (and all the connections). Signed-off-by: Antonin Bas <abas@vmware.com>

Bump up github.com/ti-mo/conntrack to v0.5

6fe8ec4

This release uses netip.Addr instead of net.IP to represent connections. Signed-off-by: Antonin Bas <abas@vmware.com>

Use existing svcCIDR in TestConnTrackOvsAppCtl_DumpFlows

e030ce1

Signed-off-by: Antonin Bas <abas@vmware.com>

Address review comments

eb02dae

Signed-off-by: Antonin Bas <abas@vmware.com>

antoninbas dismissed yuntanghsu’s stale review via eb02dae November 1, 2023 00:28

antoninbas force-pushed the use-netip-for-flow-exporter branch from ec1239b to eb02dae Compare November 1, 2023 00:28

antoninbas requested a review from tnqn November 1, 2023 00:31

tnqn approved these changes Nov 1, 2023

View reviewed changes

antoninbas merged commit d8bce58 into antrea-io:main Nov 1, 2023
47 of 56 checks passed

antoninbas deleted the use-netip-for-flow-exporter branch November 1, 2023 19:48

antoninbas added the action/release-note Indicates a PR that should be included in release notes. label Nov 1, 2023

antoninbas mentioned this pull request Nov 1, 2023

Remove redundant log in fillPodInfo/fillServiceInfo and update deny connection store. #5592

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use netip.Addr for FlowExporter #5532

Use netip.Addr for FlowExporter #5532

antoninbas commented Sep 27, 2023

antoninbas commented Sep 27, 2023

antoninbas commented Sep 28, 2023

yuntanghsu Sep 28, 2023

antoninbas Sep 28, 2023

antoninbas commented Oct 16, 2023

yuntanghsu left a comment

antoninbas commented Oct 20, 2023

antoninbas commented Oct 30, 2023

tnqn left a comment

antoninbas commented Nov 1, 2023

antoninbas commented Nov 1, 2023

Use netip.Addr for FlowExporter #5532

Use netip.Addr for FlowExporter #5532

Conversation

antoninbas commented Sep 27, 2023

antoninbas commented Sep 27, 2023

antoninbas commented Sep 28, 2023

yuntanghsu Sep 28, 2023

Choose a reason for hiding this comment

antoninbas Sep 28, 2023

Choose a reason for hiding this comment

antoninbas commented Oct 16, 2023

yuntanghsu left a comment

Choose a reason for hiding this comment

antoninbas commented Oct 20, 2023

antoninbas commented Oct 30, 2023

tnqn left a comment

Choose a reason for hiding this comment

antoninbas commented Nov 1, 2023

antoninbas commented Nov 1, 2023