/
2017-performance-progression-ipv6-route-lookup-linux.html
85 lines (73 loc) · 5.37 KB
/
2017-performance-progression-ipv6-route-lookup-linux.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
---
title: "Performance progression of IPv6 route lookup on Linux"
description: |
The performance of IPv6 route lookup on Linux has been improved over
time. Which kernels bring a notable difference?
uuid: a14ce489-92d6-4481-ae8a-b4799b19f768
attachments:
"https://github.com/vincentbernat/network-lab/tree/master/lab-routes-ipv6": "GitHub repository"
tags:
- network
- linux
---
In a [previous article][IPv6 route lookup on Linux], I explained how
Linux implements an IPv6 routing table. The following graph shows the
performance progression of route lookups through Linux history:
![IPv6 route lookup performance progression][i30]
[i30]: [[!!images/linux/ipv6-radix-perf.svg]] "IPv6 route lookup times for various Linux kernel versions. Lookup is done on a table with 40,000 routes. Notable performance changes are highlighted. The shaded surfaces represent the median absolute deviation."
All kernels are compiled with *GCC* 4.9 (from *Debian Jessie*). This version is
able to compile older kernels as well as current ones. The kernel configuration
is the default one with `CONFIG_SMP`, `CONFIG_IPV6`,
`CONFIG_IPV6_MULTIPLE_TABLES` and `CONFIG_IPV6_SUBTREES` options enabled. Some
other unrelated options are enabled to be able to boot them in a virtual machine
and run the benchmark.
There are three notable performance changes:
- In Linux **3.1**, Eric Dumazet delays a bit the copy of route metrics to fix
the undesirable sharing of route-specific metrics by all cache entries
([commit 21efcfa0ff27][]). Each cache entry now gets its own metrics, which
explains the performance hit for the non-/128 scenarios.
- In Linux **3.9**, Yoshifuji Hideaki removes the reference to the neighbor
entry in `struct rt6_info` ([commit 887c95cc1da5][]). This should have lead to
a performance increase. The small regression may be due to cache-related
issues.
- In Linux **4.2**, Martin KaFai Lau prevents the creation of cache entries for
most route lookups. The most sensible performance improvement comes
with [commit 4b32b5ad31a6][]. The second one is from [commit 45e4fd26683c][],
which effectively removes creation of cache entries, except for PMTU
exceptions.
!!! "Update (2018.08)" Here is a graph for more recent kernels.
Measures were performed on an [Intel Xeon Gold 6148][][^xeon] running
at 2.4 GHz, with GCC 6.3 from Debian Stretch. Performance regressions
in **4.4.147** and **4.9.119** are solely due to the Spectre
mitigations.[^meltdown] The small performance increase in Linux
**4.14** is from my own [commit feca7d8c135b][] which optimizes the
case without custom routing rules. In Linux **4.15**, the most notable
improvement can be traced to [commit 66f5d6ce53e6][]: Wei Wang
replaces the read-write lock by an RCU and a spinlock. In Linux
**4.18**, David Ahern [improved the data structures used by route
lookup][]. While the gain is meager, this should bring more
performance improvements in the future. This also allows a fairer
comparison between IPv4 and IPv6.
[^xeon]: That's the hardware we use to run hypervisors at [Exoscale][].
[^meltdown]: These LTS kernels received a backport of the mitigations
found in more recent kernels. For IPv4, the impact was
[non-existent][].
![IPv6 route lookup performance with recent kernels][i4]
[i4]: [[!!images/linux/ipv6-radix-perf-recent-v2.svg]] "IPv6 route lookup times for various Linux kernel versions. Lookup is done on a table with 40,000 routes. The shaded surfaces represent the median absolute deviation."
*[LTS]: Long Term Support
*[RCU]: Read-Copy-Update
[IPv6 route lookup on Linux]: [[en/blog/2017-ipv6-route-lookup-linux.html]] "IPv6 route lookup on Linux"
[non-existent]: [[en/blog/2017-performance-progression-ipv4-route-lookup-linux.html]] "Performance progression of IPv4 route lookup on Linux"
[commit 21efcfa0ff27]: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=21efcfa0ff27776902a8a15e810147be4d937d69 "ipv6: unshare inetpeers"
[commit 887c95cc1da5]: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=887c95cc1da53f66a5890fdeab13414613010097 "ipv6: Complete neighbour entry removal from dst_entry"
[commit 4b32b5ad31a6]: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=4b32b5ad31a68a661f761c76dfd0d076636d3ae9 "ipv6: Stop rt6_info from using inet_peer's metrics"
[commit 45e4fd26683c]: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=45e4fd26683c9a5f88600d91b08a484f7f09226a "ipv6: Only create RTF_CACHE routes after encountering pmtu exception"
[commit feca7d8c135b]: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=feca7d8c135bc1527b244fe817b8b6498066ccec "net: ipv6: avoid overhead when no custom FIB rules are installed"
[commit 66f5d6ce53e6]: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=66f5d6ce53e665477d2a33e8f539d4fa4ca81c83 "ipv6: replace rwlock with rcu and spinlock in fib6_table"
[improved the data structures used by route lookup]: https://patchwork.ozlabs.org/cover/877605/ "net/ipv6: Separate data structures for FIB and data path"
[Intel Xeon Gold 6148]: https://ark.intel.com/products/120489/Intel-Xeon-Gold-6148-Processor-27_5M-Cache-2_40-GHz "Intel® Xeon® Gold 6148 Processor"
[Exoscale]: https://www.exoscale.com "European Cloud Hosting"
{# Local Variables: #}
{# mode: markdown #}
{# indent-tabs-mode: nil #}
{# End: #}