[v23.3.x] introduce chunked_vector #16565
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
For some reason my PR started picking up on these fields as unused which causes build failures. So remove them. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> (cherry picked from commit 2251ca9)
Currently shrink_to_fit doesn't amend _capacity, so the inconsistency checker breaks when this is used in an expanded test suite. We can't simply adjust that because `maybe_add_capacity` doesn't know how to fix the reservation of the last fragment. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> (cherry picked from commit 6e43bd3)
Expand the test suite as well - this is what caught the previous "bug" although I think that only causes extra allocations and isn't actually breaking anything. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> (cherry picked from commit c203baf)
Add iterator invalidation checks to fragmented_vector, this is similar to absl::btree, whenever there is a mutation, we increment a generation and iterators can only be used in the same generation that they were created. These checks are only enabled in debug mode to prevent the overhead during releases. ``` [ RUN ] Vector.GenerationChecks ERROR 2024-01-24 12:06:40,763 [shard 0:main] assert - Assert failure: (/home/rockwood/Workspace/redpanda2/src/v/container/include/container/fragmented_vector.h:338) '_vec->_generation == _my_generation' Attempting to use an invalidated iterator. The corresponding fragmented_vector container has been mutated since this iterator was constructed. ``` Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> (cherry picked from commit 956ad44)
Introduce a fragmented_vector that doubles in size for the first chunk, then after that allocates full chunks. The chunk size is always fixed to our maximum recommended allocation size (128KB). This implmentation relies on the underlying vector using a 2x (elements) growth rate for it's underlying capacity, as to ensure we never overshoot our max alloc size. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> (cherry picked from commit 13f7da5)
This gives us a nice 2x speed up. I originally had kept the bounds checks in debug mode, but ASAN should be catching bad memory accesses in debug mode, so this seems simpler. Benchmarks show this brings fragmented_vector much closer to std::vector in terms of performance for use cases like sorting which does a lot of memory accesses. | test | iterations | median | mad | min | max | allocs | tasks | inst | | - | - | - | - | - | - | - | - | - | | VectorBenchTest_std_vector_int64_t_64.Sort | 619721 | 858.159ns | 1.013ns | 856.948ns | 862.491ns | 0.000 | 0.000 | 4197.4 | | VectorBenchTest_std_vector_int64_t_64.Fifo | 1885505 | 296.765ns | 0.041ns | 296.658ns | 296.954ns | 7.000 | 0.000 | 2353.0 | | VectorBenchTest_std_vector_int64_t_64.Lifo | 1900511 | 288.742ns | 0.237ns | 288.453ns | 289.003ns | 7.000 | 0.000 | 2094.0 | | VectorBenchTest_std_vector_int64_t_64.Fill | 1921621 | 285.968ns | 0.203ns | 285.764ns | 286.674ns | 7.000 | 0.000 | 2003.0 | | VectorBenchTest_std_vector_int64_t_64.RandomAccess | 186419 | 421.798ns | 0.119ns | 421.678ns | 422.006ns | 0.000 | 0.000 | 5266.0 | | VectorBenchTest_fragmented_vector_int64_t_64.Sort | 495515 | 1.263us | 1.857ns | 1.258us | 1.265us | 0.000 | 0.000 | 12735.3 | | VectorBenchTest_fragmented_vector_int64_t_64.Fifo | 1814534 | 312.678ns | 0.184ns | 312.092ns | 312.862ns | 2.000 | 0.000 | 2088.0 | | VectorBenchTest_fragmented_vector_int64_t_64.Lifo | 1809837 | 316.770ns | 0.253ns | 316.261ns | 317.568ns | 2.000 | 0.000 | 2009.0 | | VectorBenchTest_fragmented_vector_int64_t_64.Fill | 1903485 | 286.634ns | 0.076ns | 286.042ns | 286.765ns | 2.000 | 0.000 | 1378.0 | | VectorBenchTest_fragmented_vector_int64_t_64.RandomAccess | 152349 | 571.928ns | 0.104ns | 571.699ns | 572.054ns | 0.000 | 0.000 | 10271.0 | | VectorBenchTest_chunked_vector_int64_t_64.Sort | 490492 | 1.264us | 0.166ns | 1.263us | 1.266us | 0.000 | 0.000 | 12734.8 | | VectorBenchTest_chunked_vector_int64_t_64.Fifo | 1668653 | 359.417ns | 0.290ns | 358.853ns | 359.707ns | 8.000 | 0.000 | 3210.0 | | VectorBenchTest_chunked_vector_int64_t_64.Lifo | 1658706 | 367.894ns | 0.506ns | 367.374ns | 368.564ns | 8.000 | 0.000 | 3133.0 | | VectorBenchTest_chunked_vector_int64_t_64.Fill | 1709065 | 343.436ns | 0.115ns | 343.321ns | 343.971ns | 8.000 | 0.000 | 2500.0 | | VectorBenchTest_chunked_vector_int64_t_64.RandomAccess | 148856 | 578.044ns | 0.604ns | 575.469ns | 578.648ns | 0.000 | 0.000 | 10271.0 | | VectorBenchTest_std_vector_sstring_64.Sort | 95732 | 3.290us | 3.222ns | 3.285us | 3.294us | 0.000 | 0.000 | 17910.5 | | VectorBenchTest_std_vector_sstring_64.Fifo | 1036811 | 497.699ns | 0.235ns | 497.464ns | 499.942ns | 56.229 | 0.000 | 6476.7 | | VectorBenchTest_std_vector_sstring_64.Lifo | 1045502 | 614.899ns | 0.241ns | 613.723ns | 615.140ns | 56.242 | 0.000 | 9099.2 | | VectorBenchTest_std_vector_sstring_64.Fill | 1042004 | 492.807ns | 0.280ns | 492.325ns | 493.204ns | 56.238 | 0.000 | 6213.3 | | VectorBenchTest_std_vector_sstring_64.RandomAccess | 84856 | 481.779ns | 0.209ns | 481.570ns | 482.213ns | 0.000 | 0.000 | 5262.0 | | VectorBenchTest_fragmented_vector_sstring_64.Sort | 91172 | 3.871us | 1.650ns | 3.864us | 3.877us | 0.000 | 0.000 | 24548.4 | | VectorBenchTest_fragmented_vector_sstring_64.Fifo | 1066499 | 463.996ns | 0.204ns | 463.711ns | 464.439ns | 51.250 | 0.000 | 5587.5 | | VectorBenchTest_fragmented_vector_sstring_64.Lifo | 1056011 | 591.878ns | 0.246ns | 591.633ns | 592.681ns | 51.228 | 0.000 | 8258.0 | | VectorBenchTest_fragmented_vector_sstring_64.Fill | 1096842 | 441.743ns | 0.319ns | 441.182ns | 442.062ns | 51.225 | 0.000 | 4875.8 | | VectorBenchTest_fragmented_vector_sstring_64.RandomAccess | 77476 | 581.115ns | 0.578ns | 580.537ns | 1.008us | 0.000 | 0.000 | 10268.0 | | VectorBenchTest_chunked_vector_sstring_64.Sort | 90034 | 3.860us | 1.299ns | 3.858us | 3.864us | 0.000 | 0.000 | 24544.3 | | VectorBenchTest_chunked_vector_sstring_64.Fifo | 971086 | 553.555ns | 0.391ns | 552.819ns | 554.117ns | 57.215 | 0.000 | 7541.8 | | VectorBenchTest_chunked_vector_sstring_64.Lifo | 957865 | 691.826ns | 0.149ns | 691.193ns | 692.049ns | 57.227 | 0.000 | 10211.4 | | VectorBenchTest_chunked_vector_sstring_64.Fill | 1002811 | 525.656ns | 0.331ns | 525.299ns | 526.156ns | 57.243 | 0.000 | 6715.7 | | VectorBenchTest_chunked_vector_sstring_64.RandomAccess | 76972 | 570.100ns | 3.326ns | 566.773ns | 581.070ns | 0.000 | 0.000 | 10268.0 | | VectorBenchTest_std_vector_large_struct_64.Sort | 29219 | 3.273us | 1.440ns | 3.265us | 3.274us | 0.000 | 0.000 | 24874.6 | | VectorBenchTest_std_vector_large_struct_64.Fifo | 382700 | 1.354us | 0.382ns | 1.353us | 1.354us | 172.371 | 0.000 | 20348.4 | | VectorBenchTest_std_vector_large_struct_64.Lifo | 387417 | 1.811us | 0.572ns | 1.809us | 1.814us | 172.378 | 0.000 | 29969.9 | | VectorBenchTest_std_vector_large_struct_64.Fill | 382464 | 1.338us | 0.325ns | 1.336us | 1.339us | 172.386 | 0.000 | 20163.7 | | VectorBenchTest_std_vector_large_struct_64.RandomAccess | 27106 | 468.947ns | 0.211ns | 462.696ns | 469.159ns | 0.000 | 0.000 | 5266.0 | | VectorBenchTest_fragmented_vector_large_struct_64.Sort | 28921 | 3.902us | 0.474ns | 3.901us | 3.914us | 0.000 | 0.000 | 32531.2 | | VectorBenchTest_fragmented_vector_large_struct_64.Fifo | 426410 | 1.085us | 0.908ns | 1.083us | 1.088us | 167.426 | 0.000 | 17422.4 | | VectorBenchTest_fragmented_vector_large_struct_64.Lifo | 438059 | 1.527us | 1.565ns | 1.523us | 1.529us | 167.343 | 0.000 | 26984.5 | | VectorBenchTest_fragmented_vector_large_struct_64.Fill | 437154 | 1.034us | 1.030ns | 1.033us | 1.037us | 167.401 | 0.000 | 16710.4 | | VectorBenchTest_fragmented_vector_large_struct_64.RandomAccess | 27493 | 672.081ns | 0.348ns | 671.147ns | 672.773ns | 0.000 | 0.000 | 11271.0 | | VectorBenchTest_chunked_vector_large_struct_64.Sort | 28539 | 3.887us | 1.106ns | 3.885us | 3.890us | 0.000 | 0.000 | 32522.5 | | VectorBenchTest_chunked_vector_large_struct_64.Fifo | 366534 | 1.443us | 1.211ns | 1.441us | 1.446us | 173.427 | 0.000 | 21269.1 | | VectorBenchTest_chunked_vector_large_struct_64.Lifo | 383031 | 1.852us | 1.293ns | 1.850us | 1.853us | 173.419 | 0.000 | 30819.1 | | VectorBenchTest_chunked_vector_large_struct_64.Fill | 378747 | 1.370us | 2.722ns | 1.368us | 1.373us | 173.451 | 0.000 | 20561.1 | | VectorBenchTest_chunked_vector_large_struct_64.RandomAccess | 27171 | 683.280ns | 0.134ns | 682.664ns | 683.465ns | 0.000 | 0.000 | 11271.0 | | VectorBenchTest_std_vector_int64_t_10000.Sort | 3130 | 239.962us | 317.335ns | 239.604us | 240.447us | 0.000 | 0.000 | 1140047.5 | | VectorBenchTest_std_vector_int64_t_10000.Fifo | 128187 | 7.531us | 0.366ns | 7.531us | 7.532us | 15.000 | 0.000 | 152371.0 | | VectorBenchTest_std_vector_int64_t_10000.Lifo | 168081 | 5.681us | 0.461ns | 5.680us | 5.682us | 15.000 | 0.000 | 112368.0 | | VectorBenchTest_std_vector_int64_t_10000.Fill | 167748 | 5.690us | 0.736ns | 5.690us | 5.692us | 15.000 | 0.000 | 102309.0 | | VectorBenchTest_std_vector_int64_t_10000.RandomAccess | 11234 | 575.352ns | 0.385ns | 574.280ns | 575.921ns | 0.000 | 0.000 | 5266.0 | | VectorBenchTest_fragmented_vector_int64_t_10000.Sort | 2109 | 394.940us | 202.817ns | 394.671us | 395.142us | 0.000 | 0.000 | 3664438.3 | | VectorBenchTest_fragmented_vector_int64_t_10000.Fifo | 98692 | 9.837us | 2.170ns | 9.835us | 9.850us | 15.000 | 0.000 | 261844.0 | | VectorBenchTest_fragmented_vector_int64_t_10000.Lifo | 91114 | 10.706us | 0.342ns | 10.706us | 10.707us | 15.000 | 0.000 | 242348.1 | | VectorBenchTest_fragmented_vector_int64_t_10000.Fill | 138925 | 6.909us | 0.564ns | 6.906us | 6.911us | 15.000 | 0.000 | 151842.0 | | VectorBenchTest_fragmented_vector_int64_t_10000.RandomAccess | 11242 | 643.697ns | 0.752ns | 642.945ns | 645.730ns | 0.000 | 0.000 | 10271.0 | | VectorBenchTest_chunked_vector_int64_t_10000.Sort | 2106 | 393.988us | 5.436ns | 393.982us | 394.504us | 0.000 | 0.000 | 3664458.7 | | VectorBenchTest_chunked_vector_int64_t_10000.Fifo | 84284 | 11.591us | 1.012ns | 11.589us | 11.592us | 16.000 | 0.000 | 291948.1 | | VectorBenchTest_chunked_vector_int64_t_10000.Lifo | 79283 | 12.342us | 0.802ns | 12.340us | 12.344us | 16.000 | 0.000 | 272106.1 | | VectorBenchTest_chunked_vector_int64_t_10000.Fill | 113322 | 8.566us | 4.252ns | 8.561us | 8.570us | 16.000 | 0.000 | 181942.0 | | VectorBenchTest_chunked_vector_int64_t_10000.RandomAccess | 11072 | 662.164ns | 1.176ns | 660.987ns | 672.008ns | 0.000 | 0.000 | 10271.0 | | VectorBenchTest_std_vector_sstring_10000.Sort | 468 | 1.038ms | 653.985ns | 1.035ms | 1.039ms | 0.000 | 0.000 | 6183699.5 | | VectorBenchTest_std_vector_sstring_10000.Fifo | 12562 | 46.618us | 243.543ns | 46.147us | 46.889us | 7693.554 | 0.000 | 989071.9 | | VectorBenchTest_std_vector_sstring_10000.Lifo | 12527 | 79.748us | 83.624ns | 79.426us | 79.946us | 7697.765 | 0.000 | 1629339.0 | | VectorBenchTest_std_vector_sstring_10000.Fill | 12727 | 46.292us | 84.832ns | 45.903us | 57.945us | 7736.223 | 0.000 | 952464.1 | | VectorBenchTest_std_vector_sstring_10000.RandomAccess | 909 | 752.008ns | 6.272ns | 745.736ns | 761.407ns | 0.000 | 0.000 | 5262.0 | | VectorBenchTest_fragmented_vector_sstring_10000.Sort | 426 | 1.252ms | 491.293ns | 1.249ms | 1.252ms | 0.000 | 0.000 | 8534702.6 | | VectorBenchTest_fragmented_vector_sstring_10000.Fifo | 14012 | 38.489us | 66.605ns | 38.333us | 38.555us | 7716.408 | 0.000 | 873580.7 | | VectorBenchTest_fragmented_vector_sstring_10000.Lifo | 13401 | 74.579us | 90.594ns | 73.654us | 74.854us | 7734.678 | 0.000 | 1518865.6 | | VectorBenchTest_fragmented_vector_sstring_10000.Fill | 14210 | 37.138us | 195.759ns | 36.796us | 37.489us | 7724.522 | 0.000 | 764263.9 | | VectorBenchTest_fragmented_vector_sstring_10000.RandomAccess | 914 | 790.419ns | 0.744ns | 789.675ns | 792.869ns | 0.000 | 0.000 | 10268.0 | | VectorBenchTest_chunked_vector_sstring_10000.Sort | 426 | 1.244ms | 1.474us | 1.240ms | 1.246ms | 0.000 | 0.000 | 8541119.6 | | VectorBenchTest_chunked_vector_sstring_10000.Fifo | 12124 | 49.219us | 143.253ns | 48.958us | 49.363us | 7705.552 | 0.000 | 1016646.8 | | VectorBenchTest_chunked_vector_sstring_10000.Lifo | 11977 | 82.392us | 67.740ns | 82.072us | 82.459us | 7679.186 | 0.000 | 1653990.9 | | VectorBenchTest_chunked_vector_sstring_10000.Fill | 13630 | 41.786us | 521.551ns | 40.338us | 42.308us | 7704.601 | 0.000 | 886772.1 | | VectorBenchTest_chunked_vector_sstring_10000.RandomAccess | 909 | 783.381ns | 3.446ns | 779.935ns | 789.689ns | 0.000 | 0.000 | 10268.0 | | VectorBenchTest_std_vector_large_struct_10000.Sort | 163 | 1.221ms | 558.037ns | 1.220ms | 1.222ms | 0.000 | 0.000 | 9232913.6 | | VectorBenchTest_std_vector_large_struct_10000.Fifo | 3297 | 185.445us | 435.807ns | 185.009us | 186.338us | 25748.091 | 0.000 | 3423035.8 | | VectorBenchTest_std_vector_large_struct_10000.Lifo | 3324 | 300.556us | 1.159us | 298.931us | 304.622us | 25951.823 | 0.000 | 5584114.3 | | VectorBenchTest_std_vector_large_struct_10000.Fill | 3288 | 189.389us | 99.941ns | 187.857us | 189.769us | 25870.839 | 0.000 | 3403211.8 | | VectorBenchTest_std_vector_large_struct_10000.RandomAccess | 204 | 758.971ns | 5.123ns | 753.848ns | 774.863ns | 0.000 | 0.000 | 5266.0 | | VectorBenchTest_fragmented_vector_large_struct_10000.Sort | 159 | 1.468ms | 1.424us | 1.465ms | 1.470ms | 0.000 | 0.000 | 12256841.7 | | VectorBenchTest_fragmented_vector_large_struct_10000.Fifo | 3942 | 131.888us | 317.022ns | 130.926us | 132.409us | 26036.117 | 0.000 | 2837780.9 | | VectorBenchTest_fragmented_vector_large_struct_10000.Lifo | 3990 | 252.455us | 525.213ns | 249.613us | 252.980us | 26051.714 | 0.000 | 4976153.5 | | VectorBenchTest_fragmented_vector_large_struct_10000.Fill | 4048 | 128.603us | 545.802ns | 127.828us | 129.148us | 26114.617 | 0.000 | 2734305.9 | | VectorBenchTest_fragmented_vector_large_struct_10000.RandomAccess | 207 | 1.025us | 13.585ns | 1.010us | 1.043us | 0.000 | 0.000 | 11271.0 | | VectorBenchTest_chunked_vector_large_struct_10000.Sort | 158 | 1.456ms | 498.209ns | 1.454ms | 1.458ms | 0.000 | 0.000 | 12254418.1 | | VectorBenchTest_chunked_vector_large_struct_10000.Fifo | 4025 | 133.432us | 149.170ns | 132.473us | 134.499us | 25875.435 | 0.000 | 2867653.3 | | VectorBenchTest_chunked_vector_large_struct_10000.Lifo | 3982 | 249.761us | 1.624us | 246.874us | 254.130us | 25873.820 | 0.000 | 4990941.0 | | VectorBenchTest_chunked_vector_large_struct_10000.Fill | 3967 | 132.043us | 288.577ns | 131.755us | 133.324us | 25914.085 | 0.000 | 2760626.2 | | VectorBenchTest_chunked_vector_large_struct_10000.RandomAccess | 206 | 970.743ns | 0.791ns | 969.951ns | 1.014us | 0.000 | 0.000 | 11271.0 | | VectorBenchTest_std_vector_int64_t_1048576.Sort | 22 | 38.668ms | 4.663us | 38.621ms | 38.683ms | 0.000 | 0.000 | 169825784.2 | | VectorBenchTest_std_vector_int64_t_1048576.Fifo | 1386 | 721.083us | 107.830ns | 720.975us | 722.157us | 21.000 | 0.000 | 14424265.9 | | VectorBenchTest_std_vector_int64_t_1048576.Lifo | 1902 | 525.095us | 746.150ns | 524.348us | 530.522us | 21.000 | 0.000 | 10229958.4 | | VectorBenchTest_std_vector_int64_t_1048576.Fill | 1899 | 526.696us | 66.866ns | 525.806us | 526.763us | 21.000 | 0.000 | 9181299.4 | | VectorBenchTest_std_vector_int64_t_1048576.RandomAccess | 122 | 1.115us | 14.066ns | 1.101us | 1.164us | 0.000 | 0.000 | 5266.0 | | VectorBenchTest_fragmented_vector_int64_t_1048576.Sort | 14 | 63.517ms | 91.118us | 63.403ms | 63.824ms | 0.000 | 0.000 | 549816788.1 | | VectorBenchTest_fragmented_vector_int64_t_1048576.Fifo | 979 | 1.017ms | 344.919ns | 1.016ms | 1.017ms | 1035.000 | 0.000 | 27409585.6 | | VectorBenchTest_fragmented_vector_int64_t_1048576.Lifo | 903 | 1.110ms | 178.002ns | 1.110ms | 1.111ms | 1035.000 | 0.000 | 2542249.4 | | VectorBenchTest_fragmented_vector_int64_t_1048576.Fill | 1404 | 707.385us | 468.632ns | 706.122us | 707.854us | 1035.000 | 0.000 | 15875252.8 | | VectorBenchTest_fragmented_vector_int64_t_1048576.RandomAccess | 124 | 1.446us | 6.992ns | 1.434us | 1.460us | 0.000 | 0.000 | 10271.0 | | VectorBenchTest_chunked_vector_int64_t_1048576.Sort | 14 | 63.276ms | 25.314us | 63.239ms | 63.320ms | 0.000 | 0.000 | 549948277.3 | | VectorBenchTest_chunked_vector_int64_t_1048576.Fifo | 986 | 1.014ms | 319.595ns | 1.013ms | 1.014ms | 85.000 | 0.000 | 27312181.6 | | VectorBenchTest_chunked_vector_int64_t_1048576.Lifo | 916 | 1.092ms | 210.484ns | 1.090ms | 1.092ms | 85.000 | 0.000 | 25225868.8 | | VectorBenchTest_chunked_vector_int64_t_1048576.Fill | 1436 | 695.407us | 222.157ns | 695.155us | 696.050us | 85.000 | 0.000 | 15777838.8 | | VectorBenchTest_chunked_vector_int64_t_1048576.RandomAccess | 124 | 1.190us | 26.806ns | 1.163us | 1.247us | 0.000 | 0.000 | 10271.0 | Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> (cherry picked from commit e8f2f1b)
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> (cherry picked from commit 2390d2b)
When adjusting the capacity of chunked_vector we no longer assume a full chunk is being removed. Since this is only the case with a single chunk, we can just std::min with the full capacity to not get overflow. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> (cherry picked from commit aa9507b)
The dynamic variant will now remove a couple of extra allocations. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> (cherry picked from commit 9f8e7ec)
Account for chunked_vector Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> (cherry picked from commit d5699e0)
rockwotj
force-pushed
the
backport-chunked-vec
branch
from
812de51
to
1e518d4
Compare
|
Force push: add gtest annotation to new frag vector test |
|
@rockwotj - what needed to change related to the series that went into |
|
travisdowns
approved these changes
Feb 13, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backports Required
Release Notes