Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add string scalar support in AST #13061

Merged

Conversation

karthikeyann
Copy link
Contributor

@karthikeyann karthikeyann commented Apr 4, 2023

Description

Adding string scalar support in AST.
A new generic scalar device view class is added in AST to support numeric, timestamp, duration and string scalars.

Register count did not change, and benchmark results are almost same.
Compile time - There is major increase in join.cu by 15%. Other files are in range of -2% to 7%

Addressed part of #8858

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@karthikeyann karthikeyann added feature request New feature or request 2 - In Progress Currently a work in progress libcudf Affects libcudf (C++/CUDA) code. 4 - Needs Review Waiting for reviewer to review or respond non-breaking Non-breaking change labels Apr 4, 2023
@karthikeyann karthikeyann added this to the Expression evaluation milestone Apr 4, 2023
@karthikeyann
Copy link
Contributor Author

All benchmarks results are almost same.

./_deps/nvbench-src/scripts/nvbench_compare.py JOIN_NV.before.json ../../fea-string_scalar_ast_compare/release/JOIN_NV.after.json

['JOIN_NV.before.json', '../../fea-string_scalar_ast_compare/release/JOIN_NV.after.json'] # inner_join_32bit

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I32 I32 0 100000 100000 259.062 us 4.26% 276.762 us 45.22% 17.700 us 6.83% FAIL
I32 I32 0 100000 400000 725.774 us 21.08% 743.625 us 22.41% 17.852 us 2.46% PASS
I32 I32 0 10000000 10000000 17.822 ms 2.76% 17.789 ms 2.60% -32.671 us -0.18% PASS
I32 I32 0 10000000 40000000 47.758 ms 1.09% 46.829 ms 1.32% -929.068 us -1.95% FAIL
I32 I32 0 10000000 100000000 102.037 ms 0.49% 102.084 ms 0.34% 47.677 us 0.05% PASS
I32 I32 0 80000000 100000000 144.044 ms 0.39% 145.223 ms 0.63% 1.179 ms 0.82% FAIL
I32 I32 0 100000000 100000000 156.077 ms 0.29% 155.823 ms 0.27% -254.788 us -0.16% PASS
I32 I32 0 10000000 240000000 236.961 ms 0.32% 235.956 ms 0.28% -1004.461 us -0.42% FAIL
I32 I32 0 80000000 240000000 269.564 ms 0.23% 270.752 ms 0.22% 1.188 ms 0.44% FAIL
I32 I32 0 100000000 240000000 282.870 ms 0.07% 282.251 ms 0.39% -618.483 us -0.22% FAIL

inner_join_64bit

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I64 I64 0 40000000 50000000 76.408 ms 1.65% 76.398 ms 1.21% -9.267 us -0.01% PASS
I64 I64 0 50000000 50000000 83.026 ms 0.97% 82.899 ms 0.84% -126.140 us -0.15% PASS
I64 I64 0 40000000 120000000 140.004 ms 0.36% 140.594 ms 0.40% 590.421 us 0.42% FAIL
I64 I64 0 50000000 120000000 147.258 ms 0.48% 146.176 ms 0.39% -1081.729 us -0.73% FAIL

inner_join_32bit_nulls

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I32 I32 1 100000 100000 154.781 us 3.52% 156.321 us 4.37% 1.540 us 1.00% PASS
I32 I32 1 100000 400000 473.075 us 24.79% 469.674 us 24.59% -3.401 us -0.72% PASS
I32 I32 1 10000000 10000000 8.539 ms 6.37% 8.428 ms 5.81% -111.236 us -1.30% PASS
I32 I32 1 10000000 40000000 19.796 ms 0.14% 19.826 ms 0.23% 30.043 us 0.15% FAIL
I32 I32 1 10000000 100000000 41.030 ms 19.44% 35.141 ms 2.37% -5889.844 us -14.35% FAIL
I32 I32 1 80000000 100000000 56.136 ms 1.88% 56.298 ms 2.52% 161.466 us 0.29% PASS
I32 I32 1 100000000 100000000 60.123 ms 0.75% 60.190 ms 0.95% 66.722 us 0.11% PASS
I32 I32 1 10000000 240000000 99.456 ms 11.40% 99.462 ms 11.46% 6.308 us 0.01% PASS
I32 I32 1 80000000 240000000 103.508 ms 13.02% 102.898 ms 10.87% -610.402 us -0.59% PASS
I32 I32 1 100000000 240000000 102.849 ms 6.74% 102.108 ms 10.80% -741.125 us -0.72% PASS

inner_join_64bit_nulls

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I64 I64 1 40000000 50000000 31.353 ms 3.75% 31.447 ms 4.03% 93.700 us 0.30% PASS
I64 I64 1 50000000 50000000 33.667 ms 1.80% 33.536 ms 1.67% -130.675 us -0.39% PASS
I64 I64 1 40000000 120000000 51.862 ms 2.44% 53.442 ms 1.18% 1.580 ms 3.05% FAIL
I64 I64 1 50000000 120000000 54.871 ms 1.17% 54.906 ms 1.20% 34.986 us 0.06% PASS

left_join_32bit

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I32 I32 0 100000 100000 311.202 us 104.49% 316.878 us 101.39% 5.676 us 1.82% PASS
I32 I32 0 100000 400000 753.111 us 21.89% 753.446 us 19.64% 0.334 us 0.04% PASS
I32 I32 0 10000000 10000000 18.551 ms 3.42% 18.842 ms 4.04% 291.313 us 1.57% PASS
I32 I32 0 10000000 40000000 48.120 ms 1.14% 46.998 ms 0.87% -1122.513 us -2.33% FAIL
I32 I32 0 10000000 100000000 102.188 ms 0.43% 102.267 ms 0.50% 78.652 us 0.08% PASS
I32 I32 0 80000000 100000000 149.290 ms 0.51% 150.210 ms 0.75% 920.493 us 0.62% FAIL
I32 I32 0 100000000 100000000 163.612 ms 0.29% 163.459 ms 0.50% -152.529 us -0.09% PASS
I32 I32 0 10000000 240000000 237.035 ms 0.26% 236.565 ms 0.37% -469.625 us -0.20% PASS
I32 I32 0 80000000 240000000 274.036 ms 0.20% 274.639 ms 0.34% 603.119 us 0.22% FAIL
I32 I32 0 100000000 240000000 287.867 ms 0.18% 287.999 ms 0.24% 132.116 us 0.05% PASS

left_join_64bit

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I64 I64 0 40000000 50000000 79.084 ms 1.44% 79.179 ms 1.45% 94.796 us 0.12% PASS
I64 I64 0 50000000 50000000 87.036 ms 0.64% 86.077 ms 0.89% -959.717 us -1.10% FAIL
I64 I64 0 40000000 120000000 142.760 ms 0.47% 142.815 ms 0.50% 54.277 us 0.04% PASS
I64 I64 0 50000000 120000000 149.277 ms 0.50% 149.136 ms 0.36% -141.637 us -0.09% PASS

left_join_32bit_nulls

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I32 I32 1 100000 100000 349.770 us 115.03% 337.338 us 110.42% -12.432 us -3.55% PASS
I32 I32 1 100000 400000 689.004 us 20.93% 691.735 us 18.53% 2.732 us 0.40% PASS
I32 I32 1 10000000 10000000 11.185 ms 4.10% 11.206 ms 4.37% 21.519 us 0.19% PASS
I32 I32 1 10000000 40000000 22.437 ms 2.81% 22.059 ms 2.78% -377.620 us -1.68% PASS
I32 I32 1 10000000 100000000 42.837 ms 13.11% 42.825 ms 14.79% -12.165 us -0.03% PASS
I32 I32 1 80000000 100000000 62.337 ms 1.91% 64.651 ms 2.44% 2.314 ms 3.71% FAIL
I32 I32 1 100000000 100000000 70.306 ms 1.09% 69.794 ms 1.42% -511.999 us -0.73% PASS
I32 I32 1 10000000 240000000 100.991 ms 13.09% 101.012 ms 13.11% 20.476 us 0.02% PASS
I32 I32 1 80000000 240000000 118.950 ms 11.62% 118.244 ms 8.01% -706.057 us -0.59% PASS
I32 I32 1 100000000 240000000 122.671 ms 11.95% 122.608 ms 15.40% -62.686 us -0.05% PASS

left_join_64bit_nulls

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I64 I64 1 40000000 50000000 37.196 ms 2.17% 36.359 ms 2.81% -837.478 us -2.25% FAIL
I64 I64 1 50000000 50000000 40.781 ms 1.85% 40.079 ms 1.15% -701.590 us -1.72% FAIL
I64 I64 1 40000000 120000000 57.311 ms 9.38% 57.282 ms 6.63% -29.175 us -0.05% PASS
I64 I64 1 50000000 120000000 61.810 ms 9.46% 61.797 ms 9.70% -12.906 us -0.02% PASS

full_join_32bit

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I32 I32 0 100000 100000 1.018 ms 127.57% 1.027 ms 127.22% 8.923 us 0.88% PASS
I32 I32 0 100000 400000 1.476 ms 15.59% 1.499 ms 14.72% 22.641 us 1.53% PASS
I32 I32 0 10000000 10000000 27.175 ms 2.59% 27.273 ms 2.81% 98.687 us 0.36% PASS
I32 I32 0 10000000 40000000 63.154 ms 1.53% 63.247 ms 2.13% 92.297 us 0.15% PASS
I32 I32 0 10000000 100000000 132.582 ms 0.86% 132.509 ms 0.67% -72.535 us -0.05% PASS
I32 I32 0 80000000 100000000 186.075 ms 0.69% 185.622 ms 0.49% -453.152 us -0.24% PASS
I32 I32 0 100000000 100000000 200.217 ms 0.75% 200.405 ms 0.12% 187.858 us 0.09% PASS
I32 I32 0 10000000 240000000 297.954 ms 1.01% 297.840 ms 0.59% -113.856 us -0.04% PASS
I32 I32 0 80000000 240000000 338.047 ms 0.50% 336.795 ms 0.41% -1252.022 us -0.37% PASS
I32 I32 0 100000000 240000000 353.345 ms 0.49% 354.328 ms 0.86% 983.435 us 0.28% PASS

full_join_64bit

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I64 I64 0 40000000 50000000 100.657 ms 1.90% 100.202 ms 1.60% -455.087 us -0.45% PASS
I64 I64 0 50000000 50000000 108.627 ms 0.75% 109.909 ms 1.08% 1.283 ms 1.18% FAIL
I64 I64 0 40000000 120000000 177.003 ms 0.82% 179.887 ms 0.48% 2.884 ms 1.63% FAIL
I64 I64 0 50000000 120000000 186.733 ms 0.86% 187.414 ms 0.49% 680.581 us 0.36% PASS

full_join_32bit_nulls

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I32 I32 1 100000 100000 754.698 us 124.99% 771.205 us 124.24% 16.507 us 2.19% PASS
I32 I32 1 100000 400000 1.411 ms 6.69% 1.420 ms 5.93% 8.487 us 0.60% PASS
I32 I32 1 10000000 10000000 18.622 ms 0.72% 18.613 ms 0.74% -9.048 us -0.05% PASS
I32 I32 1 10000000 40000000 40.535 ms 9.38% 39.833 ms 2.43% -701.750 us -1.73% PASS
I32 I32 1 10000000 100000000 98.397 ms 40.06% 96.789 ms 15.95% -1608.768 us -1.63% PASS
I32 I32 1 80000000 100000000 129.340 ms 25.99% 129.508 ms 31.69% 168.333 us 0.13% PASS
I32 I32 1 100000000 100000000 137.399 ms 32.46% 137.450 ms 32.44% 50.795 us 0.04% PASS
I32 I32 1 10000000 240000000 235.058 ms 45.15% 235.282 ms 43.72% 224.338 us 0.10% PASS
I32 I32 1 80000000 240000000 260.492 ms 49.25% 260.818 ms 47.69% 326.087 us 0.13% PASS
I32 I32 1 100000000 240000000 269.525 ms 43.44% 270.228 ms 53.49% 703.121 us 0.26% PASS

full_join_64bit_nulls

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I64 I64 1 40000000 50000000 64.483 ms 13.81% 64.246 ms 14.99% -236.952 us -0.37% PASS
I64 I64 1 50000000 50000000 68.519 ms 24.32% 68.476 ms 16.54% -42.345 us -0.06% PASS
I64 I64 1 40000000 120000000 130.151 ms 27.52% 131.293 ms 36.99% 1.142 ms 0.88% PASS
I64 I64 1 50000000 120000000 134.537 ms 21.18% 134.886 ms 18.32% 348.404 us 0.26% PASS

mixed_inner_join_32bit

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I32 I32 0 100000 100000 896.152 us 78.02% 904.646 us 77.03% 8.494 us 0.95% PASS
I32 I32 0 100000 400000 1.425 ms 10.97% 1.436 ms 11.15% 11.674 us 0.82% PASS
I32 I32 0 10000000 10000000 40.067 ms 0.73% 40.026 ms 1.29% -40.801 us -0.10% PASS
I32 I32 0 10000000 40000000 118.238 ms 0.61% 120.059 ms 0.50% 1.821 ms 1.54% FAIL
I32 I32 0 10000000 100000000 276.233 ms 0.32% 279.209 ms 1.72% 2.976 ms 1.08% FAIL
I32 I32 0 80000000 100000000 334.296 ms 0.36% 333.527 ms 0.86% -768.933 us -0.23% PASS
I32 I32 0 100000000 100000000 352.535 ms 0.45% 357.081 ms 1.62% 4.546 ms 1.29% FAIL
I32 I32 0 10000000 240000000 633.606 ms 0.17% 635.617 ms 0.08% 2.010 ms 0.32% FAIL
I32 I32 0 80000000 240000000 695.998 ms 1.35% 694.310 ms 0.93% -1687.877 us -0.24% PASS
I32 I32 0 100000000 240000000 713.518 ms 0.09% 711.754 ms 0.10% -1764.077 us -0.25% FAIL

mixed_inner_join_64bit

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I64 I64 0 40000000 50000000 170.057 ms 0.39% 169.865 ms 0.30% -192.416 us -0.11% PASS
I64 I64 0 50000000 50000000 180.040 ms 0.50% 180.131 ms 0.50% 90.583 us 0.05% PASS
I64 I64 0 40000000 120000000 351.926 ms 0.22% 351.877 ms 0.15% -48.859 us -0.01% PASS
I64 I64 0 50000000 120000000 360.546 ms 0.04% 362.501 ms 0.24% 1.955 ms 0.54% FAIL

mixed_inner_join_32bit_nulls

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I32 I32 1 100000 100000 546.543 us 45.51% 550.088 us 44.94% 3.545 us 0.65% PASS
I32 I32 1 100000 400000 1.407 ms 8.82% 1.416 ms 9.16% 9.235 us 0.66% PASS
I32 I32 1 10000000 10000000 28.895 ms 0.55% 28.896 ms 0.74% 0.670 us 0.00% PASS
I32 I32 1 10000000 40000000 100.447 ms 0.71% 100.647 ms 0.86% 199.994 us 0.20% PASS
I32 I32 1 10000000 100000000 234.060 ms 0.30% 234.142 ms 0.57% 81.905 us 0.03% PASS
I32 I32 1 80000000 100000000 250.123 ms 1.02% 247.551 ms 0.35% -2571.763 us -1.03% FAIL
I32 I32 1 100000000 100000000 256.712 ms 1.06% 252.815 ms 0.37% -3897.174 us -1.52% FAIL
I32 I32 1 10000000 240000000 545.831 ms 0.16% 540.046 ms 0.10% -5784.202 us -1.06% FAIL
I32 I32 1 80000000 240000000 564.909 ms 0.98% 557.731 ms 0.12% -7178.238 us -1.27% FAIL
I32 I32 1 100000000 240000000 570.269 ms 1.32% 563.843 ms 0.17% -6425.147 us -1.13% FAIL

mixed_inner_join_64bit_nulls

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I64 I64 1 40000000 50000000 133.329 ms 1.13% 128.824 ms 0.69% -4504.956 us -3.38% FAIL
I64 I64 1 50000000 50000000 133.632 ms 0.72% 134.302 ms 0.56% 670.214 us 0.50% PASS
I64 I64 1 40000000 120000000 290.781 ms 0.88% 287.186 ms 0.25% -3595.815 us -1.24% FAIL
I64 I64 1 50000000 120000000 291.007 ms 0.25% 291.972 ms 0.89% 965.140 us 0.33% FAIL

mixed_left_join_32bit

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I32 I32 0 100000 100000 540.843 us 44.09% 541.570 us 43.33% 0.727 us 0.13% PASS
I32 I32 0 100000 400000 1.701 ms 15.64% 1.700 ms 10.70% -1.093 us -0.06% PASS
I32 I32 0 10000000 10000000 40.061 ms 0.50% 40.380 ms 0.39% 318.699 us 0.80% FAIL
I32 I32 0 10000000 40000000 120.339 ms 0.50% 120.137 ms 0.55% -202.127 us -0.17% PASS
I32 I32 0 10000000 100000000 276.945 ms 0.25% 276.943 ms 0.25% -1.678 us -0.00% PASS
I32 I32 0 80000000 100000000 338.779 ms 0.19% 337.143 ms 0.21% -1635.792 us -0.48% FAIL
I32 I32 0 100000000 100000000 355.801 ms 0.18% 354.805 ms 0.21% -996.074 us -0.28% FAIL
I32 I32 0 10000000 240000000 638.759 ms 0.14% 638.463 ms 0.18% -296.592 us -0.05% PASS
I32 I32 0 80000000 240000000 703.461 ms 0.13% 698.790 ms 0.13% -4671.004 us -0.66% FAIL
I32 I32 0 100000000 240000000 720.024 ms 0.14% 720.708 ms 0.09% 684.004 us 0.09% FAIL

mixed_left_join_64bit

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I64 I64 0 40000000 50000000 174.119 ms 0.41% 173.885 ms 0.50% -234.125 us -0.13% PASS
I64 I64 0 50000000 50000000 181.411 ms 0.83% 183.455 ms 1.22% 2.044 ms 1.13% FAIL
I64 I64 0 40000000 120000000 360.517 ms 0.36% 360.687 ms 0.19% 169.800 us 0.05% PASS
I64 I64 0 50000000 120000000 366.610 ms 0.49% 370.249 ms 0.39% 3.640 ms 0.99% FAIL

mixed_left_join_32bit_nulls

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I32 I32 1 100000 100000 575.628 us 45.83% 576.270 us 46.54% 0.642 us 0.11% PASS
I32 I32 1 100000 400000 1.754 ms 6.61% 1.767 ms 5.53% 13.015 us 0.74% PASS
I32 I32 1 10000000 10000000 32.989 ms 0.50% 33.619 ms 0.67% 629.350 us 1.91% FAIL
I32 I32 1 10000000 40000000 110.298 ms 1.17% 108.783 ms 1.39% -1515.314 us -1.37% FAIL
I32 I32 1 10000000 100000000 256.018 ms 0.34% 253.758 ms 0.22% -2259.405 us -0.88% FAIL
I32 I32 1 80000000 100000000 274.254 ms 1.22% 272.706 ms 1.48% -1547.228 us -0.56% PASS
I32 I32 1 100000000 100000000 278.098 ms 1.33% 276.994 ms 0.69% -1103.854 us -0.40% PASS
I32 I32 1 10000000 240000000 588.691 ms 0.14% 591.998 ms 0.14% 3.307 ms 0.56% FAIL
I32 I32 1 80000000 240000000 607.632 ms 0.16% 611.825 ms 1.03% 4.193 ms 0.69% FAIL
I32 I32 1 100000000 240000000 622.565 ms 1.46% 610.726 ms 0.28% -11838.910 us -1.90% FAIL

mixed_left_join_64bit_nulls

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I64 I64 1 40000000 50000000 143.107 ms 0.39% 140.230 ms 0.11% -2876.228 us -2.01% FAIL
I64 I64 1 50000000 50000000 143.138 ms 0.50% 144.290 ms 0.64% 1.153 ms 0.81% FAIL
I64 I64 1 40000000 120000000 315.384 ms 0.30% 314.174 ms 1.46% -1209.865 us -0.38% FAIL
I64 I64 1 50000000 120000000 318.414 ms 1.45% 316.597 ms 0.35% -1816.884 us -0.57% FAIL

mixed_full_join_32bit

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I32 I32 0 100000 100000 841.352 us 56.56% 837.606 us 58.68% -3.746 us -0.45% PASS
I32 I32 0 100000 400000 2.088 ms 10.47% 2.114 ms 9.21% 25.854 us 1.24% PASS
I32 I32 0 10000000 10000000 48.097 ms 1.30% 48.168 ms 1.59% 71.175 us 0.15% PASS
I32 I32 0 10000000 40000000 133.855 ms 0.75% 132.102 ms 0.81% -1752.498 us -1.31% FAIL
I32 I32 0 10000000 100000000 296.622 ms 0.23% 296.709 ms 0.36% 86.395 us 0.03% PASS
I32 I32 0 80000000 100000000 369.932 ms 0.22% 368.177 ms 0.34% -1755.501 us -0.47% FAIL
I32 I32 0 100000000 100000000 390.128 ms 0.41% 385.927 ms 0.50% -4200.789 us -1.08% FAIL
I32 I32 0 10000000 240000000 671.886 ms 0.20% 671.114 ms 0.25% -771.806 us -0.11% PASS
I32 I32 0 80000000 240000000 749.794 ms 0.11% 750.318 ms 0.09% 523.998 us 0.07% PASS
I32 I32 0 100000000 240000000 768.935 ms 0.13% 769.221 ms 0.20% 285.711 us 0.04% PASS

mixed_full_join_64bit

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I64 I64 0 40000000 50000000 193.532 ms 0.75% 193.036 ms 0.71% -496.166 us -0.26% PASS
I64 I64 0 50000000 50000000 202.890 ms 0.62% 203.855 ms 0.43% 965.304 us 0.48% FAIL
I64 I64 0 40000000 120000000 386.734 ms 0.25% 387.462 ms 0.14% 728.535 us 0.19% FAIL
I64 I64 0 50000000 120000000 397.169 ms 0.25% 396.266 ms 0.44% -902.618 us -0.23% PASS

mixed_full_join_32bit_nulls

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I32 I32 1 100000 100000 885.579 us 62.24% 880.265 us 63.33% -5.314 us -0.60% PASS
I32 I32 1 100000 400000 2.150 ms 3.93% 2.168 ms 4.16% 17.975 us 0.84% PASS
I32 I32 1 10000000 10000000 40.711 ms 0.81% 40.485 ms 0.49% -225.561 us -0.55% FAIL
I32 I32 1 10000000 40000000 121.413 ms 1.22% 120.128 ms 0.88% -1284.671 us -1.06% FAIL
I32 I32 1 10000000 100000000 269.942 ms 0.44% 270.705 ms 1.27% 762.494 us 0.28% PASS
I32 I32 1 80000000 100000000 293.846 ms 0.37% 296.208 ms 0.89% 2.361 ms 0.80% FAIL
I32 I32 1 100000000 100000000 302.961 ms 0.32% 303.586 ms 1.15% 624.617 us 0.21% PASS
I32 I32 1 10000000 240000000 618.200 ms 0.14% 620.554 ms 1.38% 2.354 ms 0.38% FAIL
I32 I32 1 80000000 240000000 643.941 ms 0.96% 647.805 ms 1.41% 3.864 ms 0.60% PASS
I32 I32 1 100000000 240000000 659.308 ms 1.39% 645.013 ms 0.16% -14295.397 us -2.17% FAIL

mixed_full_join_64bit_nulls

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I64 I64 1 40000000 50000000 159.183 ms 1.09% 158.649 ms 1.20% -534.253 us -0.34% PASS
I64 I64 1 50000000 50000000 160.348 ms 1.14% 160.325 ms 0.50% -23.095 us -0.01% PASS
I64 I64 1 40000000 120000000 335.723 ms 1.37% 335.106 ms 0.75% -616.675 us -0.18% PASS
I64 I64 1 50000000 120000000 341.403 ms 0.35% 335.967 ms 0.29% -5436.027 us -1.59% FAIL

mixed_left_semi_join_32bit

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I32 I32 0 100000 100000 661.678 us 40.27% 663.545 us 40.18% 1.867 us 0.28% PASS
I32 I32 0 100000 400000 1.773 ms 9.20% 1.759 ms 9.24% -13.370 us -0.75% PASS
I32 I32 0 10000000 10000000 50.197 ms 0.15% 49.976 ms 0.15% -220.899 us -0.44% FAIL
I32 I32 0 10000000 40000000 164.426 ms 0.53% 163.810 ms 0.26% -615.917 us -0.37% FAIL
I32 I32 0 10000000 100000000 393.922 ms 0.12% 390.792 ms 0.24% -3129.886 us -0.79% FAIL
I32 I32 0 80000000 100000000 455.066 ms 0.15% 449.401 ms 0.08% -5665.242 us -1.24% FAIL
I32 I32 0 100000000 100000000 471.995 ms 0.16% 468.922 ms 0.27% -3072.931 us -0.65% FAIL
I32 I32 0 10000000 240000000 919.882 ms 0.08% 918.560 ms 0.08% -1321.805 us -0.14% FAIL
I32 I32 0 80000000 240000000 985.893 ms 0.12% 983.167 ms 0.09% -2726.346 us -0.28% FAIL
I32 I32 0 100000000 240000000 1.003 s 0.11% 992.692 ms 0.05% -10647.283 us -1.06% FAIL

mixed_left_semi_join_64bit

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I64 I64 0 40000000 50000000 234.092 ms 0.24% 232.709 ms 0.62% -1382.977 us -0.59% FAIL
I64 I64 0 50000000 50000000 242.748 ms 0.30% 240.782 ms 0.23% -1966.257 us -0.81% FAIL
I64 I64 0 40000000 120000000 503.977 ms 0.15% 500.583 ms 0.05% -3394.201 us -0.67% FAIL
I64 I64 0 50000000 120000000 513.618 ms 0.17% 508.578 ms 0.18% -5040.206 us -0.98% FAIL

mixed_left_semi_join_32bit_nulls

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I32 I32 1 100000 100000 637.386 us 35.41% 641.929 us 35.04% 4.543 us 0.71% PASS
I32 I32 1 100000 400000 2.394 ms 4.79% 2.386 ms 3.94% -7.774 us -0.32% PASS
I32 I32 1 10000000 10000000 65.478 ms 0.22% 65.232 ms 0.24% -245.484 us -0.37% FAIL
I32 I32 1 10000000 40000000 242.358 ms 1.20% 241.445 ms 0.99% -913.410 us -0.38% PASS
I32 I32 1 10000000 100000000 587.214 ms 0.22% 585.087 ms 0.47% -2126.282 us -0.36% FAIL
I32 I32 1 80000000 100000000 621.249 ms 0.49% 626.927 ms 1.51% 5.678 ms 0.91% FAIL
I32 I32 1 100000000 100000000 635.400 ms 0.27% 631.882 ms 0.16% -3517.523 us -0.55% FAIL
I32 I32 1 10000000 240000000 1.389 s 0.26% 1.392 s 1.50% 3.351 ms 0.24% PASS
I32 I32 1 80000000 240000000 1.450 s 0.56% 1.431 s 0.21% -18867.576 us -1.30% FAIL
I32 I32 1 100000000 240000000 1.456 s 1.50% 1.444 s 0.26% -12324.396 us -0.85% FAIL

mixed_left_semi_join_64bit_nulls

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I64 I64 1 40000000 50000000 316.124 ms 0.46% 312.906 ms 1.05% -3217.446 us -1.02% FAIL
I64 I64 1 50000000 50000000 317.465 ms 0.34% 318.866 ms 0.31% 1.401 ms 0.44% FAIL
I64 I64 1 40000000 120000000 718.934 ms 0.43% 716.117 ms 0.19% -2816.673 us -0.39% FAIL
I64 I64 1 50000000 120000000 722.030 ms 0.25% 721.455 ms 0.36% -574.452 us -0.08% PASS

mixed_left_anti_join_32bit

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I32 I32 0 100000 100000 661.516 us 32.02% 659.017 us 32.88% -2.499 us -0.38% PASS
I32 I32 0 100000 400000 1.917 ms 10.12% 1.907 ms 8.81% -9.248 us -0.48% PASS
I32 I32 0 10000000 10000000 50.676 ms 0.91% 51.424 ms 1.49% 747.793 us 1.48% FAIL
I32 I32 0 10000000 40000000 167.298 ms 0.98% 165.547 ms 0.49% -1750.765 us -1.05% FAIL
I32 I32 0 10000000 100000000 395.914 ms 0.49% 391.151 ms 0.05% -4763.070 us -1.20% FAIL
I32 I32 0 80000000 100000000 458.492 ms 0.16% 452.824 ms 0.18% -5668.759 us -1.24% FAIL
I32 I32 0 100000000 100000000 474.459 ms 0.15% 469.529 ms 0.17% -4929.532 us -1.04% FAIL
I32 I32 0 10000000 240000000 918.421 ms 2.14% 918.805 ms 1.03% 384.593 us 0.04% PASS
I32 I32 0 80000000 240000000 992.823 ms 0.07% 982.413 ms 0.05% -10410.362 us -1.05% FAIL
I32 I32 0 100000000 240000000 1.011 s 0.07% 1.003 s 0.05% -8024.336 us -0.79% FAIL

mixed_left_anti_join_64bit

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I64 I64 0 40000000 50000000 235.284 ms 0.46% 233.199 ms 0.28% -2084.876 us -0.89% FAIL
I64 I64 0 50000000 50000000 243.822 ms 0.85% 242.875 ms 0.29% -947.865 us -0.39% FAIL
I64 I64 0 40000000 120000000 508.428 ms 0.23% 504.012 ms 0.20% -4416.407 us -0.87% FAIL
I64 I64 0 50000000 120000000 517.928 ms 0.77% 515.125 ms 0.13% -2803.166 us -0.54% FAIL

mixed_left_anti_join_32bit_nulls

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I32 I32 1 100000 100000 669.366 us 33.43% 663.724 us 34.42% -5.642 us -0.84% PASS
I32 I32 1 100000 400000 2.762 ms 4.54% 2.751 ms 2.36% -10.578 us -0.38% PASS
I32 I32 1 10000000 10000000 76.013 ms 0.29% 76.888 ms 0.32% 875.612 us 1.15% FAIL
I32 I32 1 10000000 40000000 281.725 ms 0.90% 280.011 ms 0.35% -1714.128 us -0.61% FAIL
I32 I32 1 10000000 100000000 690.275 ms 1.69% 688.107 ms 1.19% -2167.947 us -0.31% PASS
I32 I32 1 80000000 100000000 721.780 ms 0.31% 718.294 ms 0.37% -3485.535 us -0.48% FAIL
I32 I32 1 100000000 100000000 725.353 ms 0.49% 730.381 ms 1.10% 5.028 ms 0.69% FAIL
I32 I32 1 10000000 240000000 1.637 s 1.76% 1.617 s 0.40% -19953.284 us -1.22% FAIL
I32 I32 1 80000000 240000000 1.674 s 0.41% 1.667 s 0.41% -6517.063 us -0.39% PASS
I32 I32 1 100000000 240000000 1.686 s 1.25% 1.682 s 1.16% -4133.898 us -0.25% PASS

mixed_left_anti_join_64bit_nulls

[0] Quadro GV100

Key Type Payload Type Nullable Build Table Size Probe Table Size Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I64 I64 1 40000000 50000000 364.919 ms 0.32% 363.612 ms 0.40% -1306.910 us -0.36% FAIL
I64 I64 1 50000000 50000000 369.668 ms 1.06% 369.756 ms 0.88% 87.437 us 0.02% PASS
I64 I64 1 40000000 120000000 835.854 ms 0.19% 835.599 ms 0.90% -254.892 us -0.03% PASS
I64 I64 1 50000000 120000000 844.393 ms 1.04% 841.092 ms 0.40% -3301.181 us -0.39% PASS

Summary

  • Total Matches: 224
    • Pass (diff <= min_noise): 124
    • Unknown (infinite noise): 0
    • Failure (diff > min_noise): 100

./_deps/gbench-src/tools/compare.py benchmarks AST.before.json ../../fea-string_scalar_ast_compare/release/AST.after.json

Comparing AST.before.json to ../../fea-string_scalar_ast_compare/release/AST.after.json

Benchmark                                                                                                                         Time             CPU      Time Old      Time New       CPU Old       CPU New
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
AST<int32_t, TreeType::IMBALANCED_LEFT, false, false>/ast_int32_imbalanced_unique/100000/1/manual_time                         -0.0091         -0.0098             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, false, false>/ast_int32_imbalanced_unique/100000/5/manual_time                         -0.0105         -0.0121             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, false, false>/ast_int32_imbalanced_unique/100000/10/manual_time                        -0.0099         -0.0102             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, false, false>/ast_int32_imbalanced_unique/1000000/1/manual_time                        -0.0056         -0.0067             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, false, false>/ast_int32_imbalanced_unique/1000000/5/manual_time                        -0.0017         -0.0022             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, false, false>/ast_int32_imbalanced_unique/1000000/10/manual_time                       -0.0006         -0.0007             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, false, false>/ast_int32_imbalanced_unique/10000000/1/manual_time                       -0.0004         -0.0007             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, false, false>/ast_int32_imbalanced_unique/10000000/5/manual_time                       +0.0001         +0.0002             1             1             1             1
AST<int32_t, TreeType::IMBALANCED_LEFT, false, false>/ast_int32_imbalanced_unique/10000000/10/manual_time                      -0.0002         -0.0003             2             2             2             2
AST<int32_t, TreeType::IMBALANCED_LEFT, false, false>/ast_int32_imbalanced_unique/100000000/1/manual_time                      -0.0002         -0.0001             3             3             3             3
AST<int32_t, TreeType::IMBALANCED_LEFT, false, false>/ast_int32_imbalanced_unique/100000000/5/manual_time                      -0.0002         -0.0001             9             9             9             9
AST<int32_t, TreeType::IMBALANCED_LEFT, false, false>/ast_int32_imbalanced_unique/100000000/10/manual_time                     +0.0000         +0.0001            16            16            16            16
AST<int32_t, TreeType::IMBALANCED_LEFT, true, false>/ast_int32_imbalanced_reuse/100000/1/manual_time                           -0.0041         -0.0026             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, true, false>/ast_int32_imbalanced_reuse/100000/5/manual_time                           -0.0009         -0.0014             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, true, false>/ast_int32_imbalanced_reuse/100000/10/manual_time                          -0.0003         -0.0010             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, true, false>/ast_int32_imbalanced_reuse/1000000/1/manual_time                          -0.0032         -0.0027             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, true, false>/ast_int32_imbalanced_reuse/1000000/5/manual_time                          -0.0026         -0.0028             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, true, false>/ast_int32_imbalanced_reuse/1000000/10/manual_time                         -0.0014         -0.0011             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, true, false>/ast_int32_imbalanced_reuse/10000000/1/manual_time                         -0.0008         -0.0008             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, true, false>/ast_int32_imbalanced_reuse/10000000/5/manual_time                         -0.0004         -0.0007             1             1             1             1
AST<int32_t, TreeType::IMBALANCED_LEFT, true, false>/ast_int32_imbalanced_reuse/10000000/10/manual_time                        -0.0001         +0.0000             1             1             1             1
AST<int32_t, TreeType::IMBALANCED_LEFT, true, false>/ast_int32_imbalanced_reuse/100000000/1/manual_time                        -0.0002         -0.0001             2             2             2             2
AST<int32_t, TreeType::IMBALANCED_LEFT, true, false>/ast_int32_imbalanced_reuse/100000000/5/manual_time                        +0.0000         +0.0000             6             6             6             6
AST<int32_t, TreeType::IMBALANCED_LEFT, true, false>/ast_int32_imbalanced_reuse/100000000/10/manual_time                       -0.0005         -0.0005            11            11            11            11
AST<double, TreeType::IMBALANCED_LEFT, false, false>/ast_double_imbalanced_unique/100000/1/manual_time                         -0.0106         -0.0067             0             0             0             0
AST<double, TreeType::IMBALANCED_LEFT, false, false>/ast_double_imbalanced_unique/100000/5/manual_time                         -0.0066         -0.0076             0             0             0             0
AST<double, TreeType::IMBALANCED_LEFT, false, false>/ast_double_imbalanced_unique/100000/10/manual_time                        -0.0061         -0.0059             0             0             0             0
AST<double, TreeType::IMBALANCED_LEFT, false, false>/ast_double_imbalanced_unique/1000000/1/manual_time                        -0.0046         -0.0054             0             0             0             0
AST<double, TreeType::IMBALANCED_LEFT, false, false>/ast_double_imbalanced_unique/1000000/5/manual_time                        -0.0012         -0.0018             0             0             0             0
AST<double, TreeType::IMBALANCED_LEFT, false, false>/ast_double_imbalanced_unique/1000000/10/manual_time                       -0.0007         -0.0008             0             0             0             0
AST<double, TreeType::IMBALANCED_LEFT, false, false>/ast_double_imbalanced_unique/10000000/1/manual_time                       -0.0003         -0.0004             0             0             0             0
AST<double, TreeType::IMBALANCED_LEFT, false, false>/ast_double_imbalanced_unique/10000000/5/manual_time                       +0.0007         +0.0007             1             1             1             1
AST<double, TreeType::IMBALANCED_LEFT, false, false>/ast_double_imbalanced_unique/10000000/10/manual_time                      +0.0008         +0.0008             2             2             2             2
AST<double, TreeType::IMBALANCED_LEFT, false, false>/ast_double_imbalanced_unique/100000000/1/manual_time                      +0.0005         +0.0005             4             4             4             4
AST<double, TreeType::IMBALANCED_LEFT, false, false>/ast_double_imbalanced_unique/100000000/5/manual_time                      +0.0010         +0.0011            11            11            11            11
AST<double, TreeType::IMBALANCED_LEFT, false, false>/ast_double_imbalanced_unique/100000000/10/manual_time                     +0.0006         +0.0006            21            21            21            21
AST<int32_t, TreeType::IMBALANCED_LEFT, false, true>/ast_int32_imbalanced_unique_nulls/100000/1/manual_time                    -0.0035         -0.0046             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, false, true>/ast_int32_imbalanced_unique_nulls/100000/5/manual_time                    -0.0035         -0.0037             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, false, true>/ast_int32_imbalanced_unique_nulls/100000/10/manual_time                   -0.0049         -0.0057             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, false, true>/ast_int32_imbalanced_unique_nulls/1000000/1/manual_time                   -0.0020         -0.0020             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, false, true>/ast_int32_imbalanced_unique_nulls/1000000/5/manual_time                   -0.0010         -0.0008             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, false, true>/ast_int32_imbalanced_unique_nulls/1000000/10/manual_time                  -0.0005         -0.0004             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, false, true>/ast_int32_imbalanced_unique_nulls/10000000/1/manual_time                  -0.0004         -0.0004             1             1             1             1
AST<int32_t, TreeType::IMBALANCED_LEFT, false, true>/ast_int32_imbalanced_unique_nulls/10000000/5/manual_time                  +0.0001         +0.0002             2             2             2             2
AST<int32_t, TreeType::IMBALANCED_LEFT, false, true>/ast_int32_imbalanced_unique_nulls/10000000/10/manual_time                 -0.0001         -0.0001             3             3             3             3
AST<int32_t, TreeType::IMBALANCED_LEFT, false, true>/ast_int32_imbalanced_unique_nulls/100000000/1/manual_time                 -0.0005         -0.0004             9             9             9             9
AST<int32_t, TreeType::IMBALANCED_LEFT, false, true>/ast_int32_imbalanced_unique_nulls/100000000/5/manual_time                 +0.0004         +0.0004            17            17            17            17
AST<int32_t, TreeType::IMBALANCED_LEFT, false, true>/ast_int32_imbalanced_unique_nulls/100000000/10/manual_time                -0.0000         -0.0000            27            27            27            27
AST<int32_t, TreeType::IMBALANCED_LEFT, true, true>/ast_int32_imbalanced_reuse_nulls/100000/1/manual_time                      -0.0050         -0.0060             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, true, true>/ast_int32_imbalanced_reuse_nulls/100000/5/manual_time                      -0.0016         -0.0021             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, true, true>/ast_int32_imbalanced_reuse_nulls/100000/10/manual_time                     -0.0027         -0.0023             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, true, true>/ast_int32_imbalanced_reuse_nulls/1000000/1/manual_time                     -0.0026         -0.0026             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, true, true>/ast_int32_imbalanced_reuse_nulls/1000000/5/manual_time                     -0.0016         -0.0023             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, true, true>/ast_int32_imbalanced_reuse_nulls/1000000/10/manual_time                    -0.0013         -0.0020             0             0             0             0
AST<int32_t, TreeType::IMBALANCED_LEFT, true, true>/ast_int32_imbalanced_reuse_nulls/10000000/1/manual_time                    -0.0009         -0.0013             1             1             1             1
AST<int32_t, TreeType::IMBALANCED_LEFT, true, true>/ast_int32_imbalanced_reuse_nulls/10000000/5/manual_time                    -0.0004         -0.0007             1             1             1             1
AST<int32_t, TreeType::IMBALANCED_LEFT, true, true>/ast_int32_imbalanced_reuse_nulls/10000000/10/manual_time                   -0.0003         -0.0003             2             2             2             2
AST<int32_t, TreeType::IMBALANCED_LEFT, true, true>/ast_int32_imbalanced_reuse_nulls/100000000/1/manual_time                   -0.0004         -0.0003             9             9             9             9
AST<int32_t, TreeType::IMBALANCED_LEFT, true, true>/ast_int32_imbalanced_reuse_nulls/100000000/5/manual_time                   -0.0003         -0.0004            12            12            12            12
AST<int32_t, TreeType::IMBALANCED_LEFT, true, true>/ast_int32_imbalanced_reuse_nulls/100000000/10/manual_time                  -0.0002         -0.0001            18            18            18            18
AST<double, TreeType::IMBALANCED_LEFT, false, true>/ast_double_imbalanced_unique_nulls/100000/1/manual_time                    -0.0211         -0.0149             0             0             0             0
AST<double, TreeType::IMBALANCED_LEFT, false, true>/ast_double_imbalanced_unique_nulls/100000/5/manual_time                    -0.0375         -0.0293             0             0             0             0
AST<double, TreeType::IMBALANCED_LEFT, false, true>/ast_double_imbalanced_unique_nulls/100000/10/manual_time                   -0.0459         -0.0387             0             0             0             0
AST<double, TreeType::IMBALANCED_LEFT, false, true>/ast_double_imbalanced_unique_nulls/1000000/1/manual_time                   -0.0343         -0.0312             0             0             0             0
AST<double, TreeType::IMBALANCED_LEFT, false, true>/ast_double_imbalanced_unique_nulls/1000000/5/manual_time                   -0.0514         -0.0485             0             0             0             0
AST<double, TreeType::IMBALANCED_LEFT, false, true>/ast_double_imbalanced_unique_nulls/1000000/10/manual_time                  -0.0554         -0.0534             0             0             0             0
AST<double, TreeType::IMBALANCED_LEFT, false, true>/ast_double_imbalanced_unique_nulls/10000000/1/manual_time                  -0.0344         -0.0337             1             1             1             1
AST<double, TreeType::IMBALANCED_LEFT, false, true>/ast_double_imbalanced_unique_nulls/10000000/5/manual_time                  -0.0581         -0.0577             2             2             2             2
AST<double, TreeType::IMBALANCED_LEFT, false, true>/ast_double_imbalanced_unique_nulls/10000000/10/manual_time                 -0.0555         -0.0552             3             3             3             3
AST<double, TreeType::IMBALANCED_LEFT, false, true>/ast_double_imbalanced_unique_nulls/100000000/1/manual_time                 -0.0373         -0.0372            11            10            11            10
AST<double, TreeType::IMBALANCED_LEFT, false, true>/ast_double_imbalanced_unique_nulls/100000000/5/manual_time                 -0.0589         -0.0589            19            18            19            18
AST<double, TreeType::IMBALANCED_LEFT, false, true>/ast_double_imbalanced_unique_nulls/100000000/10/manual_time                -0.0549         -0.0548            32            30            32            30

./_deps/gbench-src/tools/compare.py benchmarks JOIN.before.json ../../fea-string_scalar_ast_compare/release/JOIN.after.json

Comparing JOIN.before.json to ../../fea-string_scalar_ast_compare/release/JOIN.after.json

Benchmark                                                                                                             Time             CPU      Time Old      Time New       CPU Old       CPU New
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Join<int32_t, int32_t>/left_anti_join_32bit/100000/100000/manual_time                                              +0.0344         +0.0388             0             0             0             0
Join<int32_t, int32_t>/left_anti_join_32bit/100000/400000/manual_time                                              +0.0053         +0.0035             0             0             0             0
Join<int32_t, int32_t>/left_anti_join_32bit/100000/1000000/manual_time                                             +0.0017         -0.0007             0             0             0             0
Join<int32_t, int32_t>/left_anti_join_32bit/10000000/10000000/manual_time                                          +0.0001         +0.0000            11            11            11            11
Join<int32_t, int32_t>/left_anti_join_32bit/10000000/40000000/manual_time                                          +0.0003         +0.0003            26            26            26            26
Join<int32_t, int32_t>/left_anti_join_32bit/10000000/100000000/manual_time                                         +0.0001         -0.0000            54            54            54            54
Join<int32_t, int32_t>/left_anti_join_32bit/100000000/100000000/manual_time                                        +0.0006         +0.0006           122           122           122           122
Join<int32_t, int32_t>/left_anti_join_32bit/80000000/240000000/manual_time                                         -0.0001         -0.0002           178           178           178           178
Join<int64_t, int64_t>/left_anti_join_64bit/50000000/50000000/manual_time                                          -0.0002         -0.0003            61            61            61            61
Join<int64_t, int64_t>/left_anti_join_64bit/40000000/120000000/manual_time                                         -0.0006         -0.0005            90            90            90            90
Join<int32_t, int32_t>/left_anti_join_32bit_nulls/100000/100000/manual_time                                        -0.0025         +0.0035             0             0             0             0
Join<int32_t, int32_t>/left_anti_join_32bit_nulls/100000/400000/manual_time                                        +0.0013         -0.0009             0             0             0             0
Join<int32_t, int32_t>/left_anti_join_32bit_nulls/100000/1000000/manual_time                                       -0.0033         -0.0034             0             0             0             0
Join<int32_t, int32_t>/left_anti_join_32bit_nulls/10000000/10000000/manual_time                                    -0.0007         -0.0007             5             5             5             5
Join<int32_t, int32_t>/left_anti_join_32bit_nulls/10000000/40000000/manual_time                                    -0.0002         -0.0003            11            11            11            11
Join<int32_t, int32_t>/left_anti_join_32bit_nulls/10000000/100000000/manual_time                                   -0.0001         -0.0001            22            22            22            22
Join<int32_t, int32_t>/left_anti_join_32bit_nulls/100000000/100000000/manual_time                                  -0.0002         -0.0003            51            51            51            51
Join<int32_t, int32_t>/left_anti_join_32bit_nulls/80000000/240000000/manual_time                                   -0.0009         -0.0009            72            72            72            72
Join<int64_t, int64_t>/left_anti_join_64bit_nulls/50000000/50000000/manual_time                                    +0.0002         +0.0001            27            27            27            27
Join<int64_t, int64_t>/left_anti_join_64bit_nulls/40000000/120000000/manual_time                                   -0.0005         -0.0005            37            37            37            37
Join<int32_t, int32_t>/left_semi_join_32bit/100000/100000/manual_time                                              +0.0071         +0.0132             0             0             0             0
Join<int32_t, int32_t>/left_semi_join_32bit/100000/400000/manual_time                                              +0.0070         +0.0047             0             0             0             0
Join<int32_t, int32_t>/left_semi_join_32bit/100000/1000000/manual_time                                             +0.0033         +0.0015             0             0             0             0
Join<int32_t, int32_t>/left_semi_join_32bit/10000000/10000000/manual_time                                          +0.0001         +0.0000            11            11            11            11
Join<int32_t, int32_t>/left_semi_join_32bit/10000000/40000000/manual_time                                          -0.0003         -0.0004            25            25            26            25
Join<int32_t, int32_t>/left_semi_join_32bit/10000000/100000000/manual_time                                         +0.0000         +0.0001            54            54            54            54
Join<int32_t, int32_t>/left_semi_join_32bit/100000000/100000000/manual_time                                        +0.0001         +0.0001           121           121           121           121
Join<int32_t, int32_t>/left_semi_join_32bit/80000000/240000000/manual_time                                         -0.0001         -0.0001           178           178           178           178
Join<int64_t, int64_t>/left_semi_join_64bit/50000000/50000000/manual_time                                          -0.0001         -0.0001            61            61            61            61
Join<int64_t, int64_t>/left_semi_join_64bit/40000000/120000000/manual_time                                         +0.0000         +0.0000            90            90            90            90
Join<int32_t, int32_t>/left_semi_join_32bit_nulls/100000/100000/manual_time                                        -0.0024         +0.0052             0             0             0             0
Join<int32_t, int32_t>/left_semi_join_32bit_nulls/100000/400000/manual_time                                        +0.0049         +0.0027             0             0             0             0
Join<int32_t, int32_t>/left_semi_join_32bit_nulls/100000/1000000/manual_time                                       -0.0022         -0.0003             0             0             0             0
Join<int32_t, int32_t>/left_semi_join_32bit_nulls/10000000/10000000/manual_time                                    -0.0004         -0.0005             5             5             5             5
Join<int32_t, int32_t>/left_semi_join_32bit_nulls/10000000/40000000/manual_time                                    -0.0008         -0.0008            11            11            11            11
Join<int32_t, int32_t>/left_semi_join_32bit_nulls/10000000/100000000/manual_time                                   -0.0003         -0.0004            22            22            22            22
Join<int32_t, int32_t>/left_semi_join_32bit_nulls/100000000/100000000/manual_time                                  -0.0000         -0.0000            51            51            51            51
Join<int32_t, int32_t>/left_semi_join_32bit_nulls/80000000/240000000/manual_time                                   +0.0002         +0.0002            71            71            71            71
Join<int64_t, int64_t>/left_semi_join_64bit_nulls/50000000/50000000/manual_time                                    +0.0004         +0.0003            26            26            26            27
Join<int64_t, int64_t>/left_semi_join_64bit_nulls/40000000/120000000/manual_time                                   -0.0002         -0.0004            37            37            37            37
ConditionalJoin<int32_t, int32_t>/conditional_inner_join_32bit/100000/100000/manual_time                           +0.0166         +0.0166           407           413           407           413
ConditionalJoin<int32_t, int32_t>/conditional_inner_join_32bit/100000/400000/manual_time                           +0.0049         +0.0049          1235          1241          1235          1241
ConditionalJoin<int32_t, int32_t>/conditional_inner_join_32bit/400000/100000/manual_time                           +0.0048         +0.0049          1235          1241          1235          1241
ConditionalJoin<int32_t, int32_t>/conditional_inner_join_32bit/100000/1000000/manual_time                          +0.0063         +0.0062          2881          2899          2881          2898
ConditionalJoin<int64_t, int64_t>/conditional_inner_join_64bit/100000/100000/manual_time                           -0.0036         -0.0036           412           410           412           410
ConditionalJoin<int64_t, int64_t>/conditional_inner_join_64bit/100000/400000/manual_time                           +0.0010         +0.0010          1244          1245          1244          1245
ConditionalJoin<int64_t, int64_t>/conditional_inner_join_64bit/400000/100000/manual_time                           +0.0011         +0.0010          1244          1245          1244          1245
ConditionalJoin<int64_t, int64_t>/conditional_inner_join_64bit/100000/1000000/manual_time                          +0.0016         +0.0015          2904          2909          2904          2909
ConditionalJoin<int32_t, int32_t>/conditional_inner_join_32bit_nulls/100000/100000/manual_time                     +0.0043         +0.0043           641           644           641           643
ConditionalJoin<int32_t, int32_t>/conditional_inner_join_32bit_nulls/100000/400000/manual_time                     -0.0083         -0.0090          1959          1943          1959          1941
ConditionalJoin<int32_t, int32_t>/conditional_inner_join_32bit_nulls/400000/100000/manual_time                     -0.0085         -0.0085          2010          1992          2009          1992
ConditionalJoin<int32_t, int32_t>/conditional_inner_join_32bit_nulls/100000/1000000/manual_time                    -0.0001         -0.0001          4699          4699          4699          4699
ConditionalJoin<int64_t, int64_t>/conditional_inner_join_64bit_nulls/100000/100000/manual_time                     +0.0150         +0.0150           657           667           657           667
ConditionalJoin<int64_t, int64_t>/conditional_inner_join_64bit_nulls/100000/400000/manual_time                     +0.0003         +0.0001          2013          2014          2013          2013
ConditionalJoin<int64_t, int64_t>/conditional_inner_join_64bit_nulls/400000/100000/manual_time                     -0.0103         -0.0103          2067          2046          2067          2046
ConditionalJoin<int64_t, int64_t>/conditional_inner_join_64bit_nulls/100000/1000000/manual_time                    -0.0433         -0.0434          5025          4807          5024          4806
ConditionalJoin<int32_t, int32_t>/conditional_left_join_32bit/100000/100000/manual_time                            +0.0145         +0.0145           405           411           405           411
ConditionalJoin<int32_t, int32_t>/conditional_left_join_32bit/100000/400000/manual_time                            +0.0049         +0.0048          1235          1241          1235          1241
ConditionalJoin<int32_t, int32_t>/conditional_left_join_32bit/100000/1000000/manual_time                           +0.0061         +0.0063          2881          2899          2880          2899
ConditionalJoin<int64_t, int64_t>/conditional_left_join_64bit/100000/100000/manual_time                            -0.0095         -0.0095           414           410           414           410
ConditionalJoin<int64_t, int64_t>/conditional_left_join_64bit/100000/400000/manual_time                            -0.0140         -0.0141          1263          1245          1263          1245
ConditionalJoin<int64_t, int64_t>/conditional_left_join_64bit/100000/1000000/manual_time                           -0.0265         -0.0265          2988          2909          2988          2909
ConditionalJoin<int32_t, int32_t>/conditional_left_join_32bit_nulls/100000/100000/manual_time                      -0.0548         -0.0548           681           643           681           643
ConditionalJoin<int32_t, int32_t>/conditional_left_join_32bit_nulls/100000/400000/manual_time                      -0.0776         -0.0776          2106          1942          2106          1942
ConditionalJoin<int32_t, int32_t>/conditional_left_join_32bit_nulls/100000/1000000/manual_time                     -0.0827         -0.0827          5116          4693          5116          4693
ConditionalJoin<int64_t, int64_t>/conditional_left_join_64bit_nulls/100000/100000/manual_time                      -0.0652         -0.0652           703           658           703           658
ConditionalJoin<int64_t, int64_t>/conditional_left_join_64bit_nulls/100000/400000/manual_time                      -0.0902         -0.0902          2196          1998          2196          1997
ConditionalJoin<int64_t, int64_t>/conditional_left_join_64bit_nulls/100000/1000000/manual_time                     -0.0912         -0.0913          5304          4820          5304          4820
ConditionalJoin<int32_t, int32_t>/conditional_full_join_32bit/100000/100000/manual_time                            +0.0071         +0.0071           411           413           411           413
ConditionalJoin<int32_t, int32_t>/conditional_full_join_32bit/100000/400000/manual_time                            -0.0039         -0.0039          1246          1241          1246          1241
ConditionalJoin<int32_t, int32_t>/conditional_full_join_32bit/100000/1000000/manual_time                           -0.0231         -0.0230          2968          2899          2967          2899
ConditionalJoin<int64_t, int64_t>/conditional_full_join_64bit/100000/100000/manual_time                            -0.0439         -0.0440           432           413           432           413
ConditionalJoin<int64_t, int64_t>/conditional_full_join_64bit/100000/400000/manual_time                            -0.0221         -0.0221          1274          1246          1274          1246
ConditionalJoin<int64_t, int64_t>/conditional_full_join_64bit/100000/1000000/manual_time                           -0.0485         -0.0490          3057          2909          3057          2907
ConditionalJoin<int32_t, int32_t>/conditional_full_join_32bit_nulls/100000/100000/manual_time                      -0.0626         -0.0626           695           651           695           651
ConditionalJoin<int32_t, int32_t>/conditional_full_join_32bit_nulls/100000/400000/manual_time                      -0.0748         -0.0748          2119          1960          2119          1960
ConditionalJoin<int32_t, int32_t>/conditional_full_join_32bit_nulls/100000/1000000/manual_time                     -0.0709         -0.0700          5158          4792          5152          4792
ConditionalJoin<int64_t, int64_t>/conditional_full_join_64bit_nulls/100000/100000/manual_time                      -0.0819         -0.0819           724           664           724           664
ConditionalJoin<int64_t, int64_t>/conditional_full_join_64bit_nulls/100000/400000/manual_time                      -0.0584         -0.0607          2210          2081          2210          2076
ConditionalJoin<int64_t, int64_t>/conditional_full_join_64bit_nulls/100000/1000000/manual_time                     -0.0457         -0.0457          5354          5109          5354          5109
ConditionalJoin<int32_t, int32_t>/conditional_left_anti_join_32bit/100000/100000/manual_time                       -0.0480         -0.0480           429           408           429           408
ConditionalJoin<int32_t, int32_t>/conditional_left_anti_join_32bit/100000/400000/manual_time                       -0.0120         -0.0120          1256          1241          1256          1241
ConditionalJoin<int32_t, int32_t>/conditional_left_anti_join_32bit/100000/1000000/manual_time                      -0.0190         -0.0190          3011          2954          3011          2954
ConditionalJoin<int64_t, int64_t>/conditional_left_anti_join_64bit/100000/100000/manual_time                       -0.0281         -0.0282           428           416           428           416
ConditionalJoin<int64_t, int64_t>/conditional_left_anti_join_64bit/100000/400000/manual_time                       -0.0372         -0.0372          1301          1252          1301          1252
ConditionalJoin<int64_t, int64_t>/conditional_left_anti_join_64bit/100000/1000000/manual_time                      -0.0313         -0.0313          3094          2997          3093          2997
ConditionalJoin<int32_t, int32_t>/conditional_left_anti_join_32bit_nulls/100000/100000/manual_time                 -0.0016         -0.0016           694           693           694           693
ConditionalJoin<int32_t, int32_t>/conditional_left_anti_join_32bit_nulls/100000/400000/manual_time                 -0.0260         -0.0256          2129          2074          2128          2074
ConditionalJoin<int32_t, int32_t>/conditional_left_anti_join_32bit_nulls/100000/1000000/manual_time                -0.0410         -0.0410          5254          5039          5254          5038
ConditionalJoin<int64_t, int64_t>/conditional_left_anti_join_64bit_nulls/100000/100000/manual_time                 -0.0286         -0.0286           716           696           716           696
ConditionalJoin<int64_t, int64_t>/conditional_left_anti_join_64bit_nulls/100000/400000/manual_time                 -0.0310         -0.0310          2210          2141          2210          2141
ConditionalJoin<int64_t, int64_t>/conditional_left_anti_join_64bit_nulls/100000/1000000/manual_time                -0.0284         -0.0284          5371          5218          5370          5218
ConditionalJoin<int32_t, int32_t>/conditional_left_semi_join_32bit/100000/100000/manual_time                       -0.0463         -0.0463           431           411           431           411
ConditionalJoin<int32_t, int32_t>/conditional_left_semi_join_32bit/100000/400000/manual_time                       -0.0025         -0.0026          1259          1255          1259          1255
ConditionalJoin<int32_t, int32_t>/conditional_left_semi_join_32bit/100000/1000000/manual_time                      -0.0061         -0.0062          3018          3000          3018          2999
ConditionalJoin<int64_t, int64_t>/conditional_left_semi_join_64bit/100000/100000/manual_time                       -0.0300         -0.0300           430           417           430           417
ConditionalJoin<int64_t, int64_t>/conditional_left_semi_join_64bit/100000/400000/manual_time                       -0.0105         -0.0105          1293          1279          1293          1279
ConditionalJoin<int64_t, int64_t>/conditional_left_semi_join_64bit/100000/1000000/manual_time                      +0.0221         +0.0221          3102          3171          3102          3171
ConditionalJoin<int32_t, int32_t>/conditional_left_semi_join_32bit_nulls/100000/100000/manual_time                 +0.0244         +0.0244           683           700           683           700
ConditionalJoin<int32_t, int32_t>/conditional_left_semi_join_32bit_nulls/100000/400000/manual_time                 -0.0302         -0.0302          2155          2090          2155          2090
ConditionalJoin<int32_t, int32_t>/conditional_left_semi_join_32bit_nulls/100000/1000000/manual_time                -0.0278         -0.0278          5207          5062          5207          5062
ConditionalJoin<int64_t, int64_t>/conditional_left_semi_join_64bit_nulls/100000/100000/manual_time                 -0.0358         -0.0358           708           683           708           683
ConditionalJoin<int64_t, int64_t>/conditional_left_semi_join_64bit_nulls/100000/400000/manual_time                 -0.0279         -0.0279          2222          2160          2222          2160
ConditionalJoin<int64_t, int64_t>/conditional_left_semi_join_64bit_nulls/100000/1000000/manual_time                -0.0342         -0.0345          5380          5196          5380          5194

@karthikeyann
Copy link
Contributor Author

Compile time differences (From .ninja_log)

files branch-23.06 ast compare diff %diff
CMakeFiles/cudf.dir/src/join/conditional_join.cu.o 124817 131282 6465 5%
CMakeFiles/cudf.dir/src/join/cross_join.cu.o 20121 21161 1040 5%
CMakeFiles/cudf.dir/src/join/hash_join.cu.o 51821 55529 3708 7%
CMakeFiles/cudf.dir/src/join/join.cu.o 24475 28093 3618 15%
CMakeFiles/cudf.dir/src/join/join_utils.cu.o 29909 29496 -413 -1%
CMakeFiles/cudf.dir/src/join/mixed_join.cu.o 41330 41197 -133 0%
CMakeFiles/cudf.dir/src/join/mixed_join_kernel.cu.o 134206 141358 7152 5%
CMakeFiles/cudf.dir/src/join/mixed_join_kernel_nulls.cu.o 179307 188245 8938 5%
CMakeFiles/cudf.dir/src/join/mixed_join_kernels_semi.cu.o 101032 106287 5255 5%
CMakeFiles/cudf.dir/src/join/mixed_join_semi.cu.o 53062 52447 -615 -1%
CMakeFiles/cudf.dir/src/join/mixed_join_size_kernel.cu.o 146499 154751 8252 6%
CMakeFiles/cudf.dir/src/join/mixed_join_size_kernel_nulls.cu.o 189890 199687 9797 5%
CMakeFiles/cudf.dir/src/join/mixed_join_size_kernels_semi.cu.o 103155 107622 4467 4%
CMakeFiles/cudf.dir/src/join/semi_join.cu.o 26818 26362 -456 -2%
         
CMakeFiles/cudf.dir/src/transform/bools_to_mask.cu.o 21477 22789 1312 6%
CMakeFiles/cudf.dir/src/transform/compute_column.cu.o 97877 105062 7185 7%
CMakeFiles/cudf.dir/src/transform/encode.cu.o 19075 20085 1010 5%
CMakeFiles/cudf.dir/src/transform/mask_to_bools.cu.o 20944 20582 -362 -2%
CMakeFiles/cudf.dir/src/transform/nans_to_nulls.cu.o 22563 23055 492 2%
CMakeFiles/cudf.dir/src/transform/transform.cpp.o 35600 36611 1011 3%

@karthikeyann karthikeyann self-assigned this Apr 5, 2023
@karthikeyann karthikeyann added 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currently a work in progress labels Apr 5, 2023
@karthikeyann karthikeyann marked this pull request as ready for review April 5, 2023 18:43
@karthikeyann karthikeyann requested a review from a team as a code owner April 5, 2023 18:43
Copy link
Contributor

@vyasr vyasr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks fine to me. Do we have a plan for how this generic scalar can be pulled out of AST code, though? I believe there is some refactoring needed of the scalar device views for that to be possible, could we open an issue documenting that before we merge this?

@karthikeyann
Copy link
Contributor Author

Created #13160 to track this.

@karthikeyann karthikeyann requested a review from vyasr April 18, 2023 09:21
Copy link
Contributor

@vyasr vyasr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@karthikeyann
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 267bc6a into rapidsai:branch-23.06 Apr 20, 2023
rapids-bot bot pushed a commit that referenced this pull request Apr 25, 2023
Depends on #13061
Add Java bindings for string scalar support in AST
Add unit test for string comparison - column vs column, column vs literal.

Authors:
  - Karthikeyan (https://github.com/karthikeyann)

Approvers:
  - Jason Lowe (https://github.com/jlowe)
  - MithunR (https://github.com/mythrocks)

URL: #13072
rapids-bot bot pushed a commit that referenced this pull request May 8, 2023
Depends on #13061
Add Python bindings for string scalar support in AST
Add unit test for string comparison - column vs column, column vs literal.

Authors:
  - Karthikeyan (https://github.com/karthikeyann)
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)
  - Ashwin Srinath (https://github.com/shwina)

URL: #13073
davidwendt pushed a commit to davidwendt/cudf that referenced this pull request May 8, 2023
Depends on rapidsai#13061
Add Python bindings for string scalar support in AST
Add unit test for string comparison - column vs column, column vs literal.

Authors:
  - Karthikeyan (https://github.com/karthikeyann)
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)
  - Ashwin Srinath (https://github.com/shwina)

URL: rapidsai#13073
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3 - Ready for Review Ready for review by team 4 - Needs Review Waiting for reviewer to review or respond feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants