Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf(drop): speed up performance of drop #9440

Merged
merged 2 commits into from
Jun 26, 2024
Merged

perf(drop): speed up performance of drop #9440

merged 2 commits into from
Jun 26, 2024

Conversation

cpcloud
Copy link
Member

@cpcloud cpcloud commented Jun 25, 2024

This PR speeds up drop construction and compilation in cases where the number
of dropped columns is small relative to the total number of parent-table
columns.

There are two ways this is done:

  • drop construction is sped up by reducing the number of iterations
    over the full set of columns when constructing an output schema.

    This is where the bulk of the improvement is.

  • Compilation of the drop operation is also a bit faster for smaller sets of
    dropped columns on some backends due to use of * EXCLUDE syntax.

Since the optimization is done in the schema property, adding a new
DropColumns relation IR seemed like the lightest weight approach given
that that also enables compilers to use EXCLUDE syntax, which will produce a
far smaller query than using project-without-the-dropped-columns approach.

Partially addresses the drop performance seen in #9111.

To address this for all backends, they either need to all support
SELECT * EXCLUDE(col1, ..., colN) syntax or we need to implement column
pruning.

Follow-ups could include applying a similar approach to rename (using REPLACE
syntax for compilation).

It might be possible to reduce the overhead of relocate as well, but
I haven't explored that.

@cpcloud cpcloud added this to the 9.2 milestone Jun 25, 2024
@cpcloud cpcloud added internals Issues or PRs related to ibis's internal APIs ux User experience related issues performance Issues related to ibis's performance labels Jun 25, 2024
@cpcloud
Copy link
Member Author

cpcloud commented Jun 25, 2024

Benchmarks:

ibis/tests/benchmarks/test_benchmarks.py ........                                                                                                                                                       [100%]
Saved benchmark data in: /home/cloud/src/ibis/.benchmarks/Linux-CPython-3.10-64bit/0003_d3f8cb4727858b2b7facdcb8add0f204169032fd_20240625_182530_uncommited-changes.json



---------------------------------------------------------------------------- benchmark 'test_wide_drop_compile[10000]': 2 tests ---------------------------------------------------------------------------
Name (time in s)                                    Min               Max              Mean            StdDev            Median               IQR            Outliers     OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[10000] (0002_d3f8cb4)     2.0755 (2.19)     2.1360 (2.23)     2.1036 (2.21)     0.0224 (4.36)     2.1028 (2.21)     0.0286 (4.44)          2;0  0.4754 (0.45)          5           1
test_wide_drop_compile[10000] (NOW)              0.9459 (1.0)      0.9599 (1.0)      0.9527 (1.0)      0.0051 (1.0)      0.9524 (1.0)      0.0065 (1.0)           2;0  1.0496 (1.0)           5           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------- benchmark 'test_wide_drop_compile[1000]': 2 tests ---------------------------------------------------------------------------------
Name (time in ms)                                    Min                 Max                Mean             StdDev              Median                IQR            Outliers      OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[1000] (0002_d3f8cb4)     165.1536 (2.18)     211.3253 (1.72)     183.2472 (2.10)     24.1177 (1.28)     166.4773 (2.15)     43.4800 (4.18)          2;0   5.4571 (0.48)          5           1
test_wide_drop_compile[1000] (NOW)               75.7718 (1.0)      123.0444 (1.0)       87.1311 (1.0)      18.8298 (1.0)       77.5490 (1.0)      10.4056 (1.0)           3;3  11.4770 (1.0)          13           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------- benchmark 'test_wide_drop_compile[100]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                  Min                Max               Mean            StdDev             Median               IQR            Outliers       OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[100] (0002_d3f8cb4)     17.3570 (2.03)     57.9362 (1.15)     18.3480 (2.03)     5.3407 (1.35)     17.5755 (2.03)     0.3080 (4.81)          1;1   54.5020 (0.49)         57           1
test_wide_drop_compile[100] (NOW)               8.5370 (1.0)      50.5738 (1.0)       9.0313 (1.0)      3.9614 (1.0)       8.6489 (1.0)      0.0640 (1.0)           1;4  110.7262 (1.0)         112           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_wide_drop_compile[10]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[10] (0002_d3f8cb4)     2.4459 (1.52)     2.8055 (1.22)     2.5302 (1.40)     0.0572 (1.0)      2.5095 (1.40)     0.0447 (1.28)        95;60  395.2256 (0.71)        370           1
test_wide_drop_compile[10] (NOW)              1.6116 (1.0)      2.2924 (1.0)      1.8016 (1.0)      0.0580 (1.01)     1.7918 (1.0)      0.0350 (1.0)         66;61  555.0747 (1.0)         487           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_wide_drop_construct[10000]': 2 tests ---------------------------------------------------------------------------
Name (time in s)                                      Min               Max              Mean            StdDev            Median               IQR            Outliers     OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[10000] (0002_d3f8cb4)     1.4352 (4.06)     1.4693 (3.69)     1.4571 (4.01)     0.0131 (1.0)      1.4584 (4.10)     0.0131 (1.0)           1;0  0.6863 (0.25)          5           1
test_wide_drop_construct[10000] (NOW)              0.3533 (1.0)      0.3987 (1.0)      0.3636 (1.0)      0.0196 (1.50)     0.3557 (1.0)      0.0132 (1.01)          1;1  2.7502 (1.0)           5           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------- benchmark 'test_wide_drop_construct[1000]': 2 tests ---------------------------------------------------------------------------------
Name (time in ms)                                      Min                 Max                Mean             StdDev              Median               IQR            Outliers      OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[1000] (0002_d3f8cb4)     132.1098 (4.38)     172.9310 (4.49)     140.0918 (4.57)     13.3853 (9.31)     136.3866 (4.49)     2.6920 (14.48)         1;1   7.1382 (0.22)          8           1
test_wide_drop_construct[1000] (NOW)               30.1833 (1.0)       38.4965 (1.0)       30.6543 (1.0)       1.4381 (1.0)       30.4034 (1.0)      0.1859 (1.0)           1;2  32.6218 (1.0)          32           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------- benchmark 'test_wide_drop_construct[100]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                    Min                Max               Mean            StdDev             Median               IQR            Outliers       OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[100] (0002_d3f8cb4)     13.8188 (4.58)     14.3798 (2.87)     14.0754 (4.33)     0.1001 (1.0)      14.0750 (4.37)     0.1102 (4.04)         20;4   71.0461 (0.23)         69           1
test_wide_drop_construct[100] (NOW)               3.0182 (1.0)       5.0053 (1.0)       3.2527 (1.0)      0.2163 (2.16)      3.2213 (1.0)      0.0273 (1.0)          8;21  307.4391 (1.0)         297           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_wide_drop_construct[10]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                  Min               Max              Mean            StdDev            Median               IQR            Outliers         OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[10] (0002_d3f8cb4)     1.5844 (3.76)     2.0803 (3.91)     1.6544 (3.49)     0.0366 (2.43)     1.6523 (3.50)     0.0372 (1.68)       119;11    604.4585 (0.29)        560           1
test_wide_drop_construct[10] (NOW)              0.4211 (1.0)      0.5317 (1.0)      0.4736 (1.0)      0.0151 (1.0)      0.4725 (1.0)      0.0221 (1.0)        403;37  2,111.5923 (1.0)        1752           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

@cpcloud cpcloud force-pushed the drop-perf branch 3 times, most recently from b9bc7f7 to 1ee4118 Compare June 26, 2024 12:24
@cpcloud cpcloud force-pushed the drop-perf branch 7 times, most recently from 495aa08 to 2e68785 Compare June 26, 2024 13:05
@cpcloud
Copy link
Member Author

cpcloud commented Jun 26, 2024

New benchmark results (note these aren't comparing to main because some of them hit the Python recursion limit on main):

Construction

------------------------------------------------------------------------------------- benchmark 'test_wide_drop_construct wide_table=10': 3 tests -------------------------------------------------------------------------------------
Name (time in ns)                            Min                     Max                    Mean                 StdDev                  Median                    IQR             Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[10-1]           93.1698 (1.0)          469.8901 (1.0)           97.5044 (1.0)           4.7709 (1.0)           96.7801 (1.0)           2.7101 (1.0)      5384;5541   10,255.9426 (1.0)       90245         100
test_wide_drop_construct[10-50]     603,079.0100 (>1000.0)  926,768.9893 (>1000.0)  626,248.7081 (>1000.0)  12,677.9655 (>1000.0)  626,722.9910 (>1000.0)  11,589.5018 (>1000.0)     299;31        1.5968 (0.00)       1477           1
test_wide_drop_construct[10-99]     767,188.9907 (>1000.0)  815,709.9946 (>1000.0)  787,975.6173 (>1000.0)   9,581.1056 (>1000.0)  789,791.0255 (>1000.0)  16,580.9979 (>1000.0)      435;0        1.2691 (0.00)       1171           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------ benchmark 'test_wide_drop_construct wide_table=100': 3 tests -----------------------------------------------------------------
Name (time in ms)                       Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[100-1]      3.2136 (1.0)      3.3814 (1.0)      3.2634 (1.0)      0.0154 (1.0)      3.2630 (1.0)      0.0169 (1.0)          75;7  306.4272 (1.0)         304           1
test_wide_drop_construct[100-50]     5.3193 (1.66)     5.4533 (1.61)     5.3668 (1.64)     0.0216 (1.41)     5.3635 (1.64)     0.0260 (1.54)         46;5  186.3296 (0.61)        177           1
test_wide_drop_construct[100-99]     7.1622 (2.23)     7.7344 (2.29)     7.3242 (2.24)     0.0860 (5.60)     7.3506 (2.25)     0.1379 (8.15)         38;2  136.5345 (0.45)        134           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------- benchmark 'test_wide_drop_construct wide_table=1000': 3 tests --------------------------------------------------------------------
Name (time in ms)                         Min                 Max               Mean             StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[1000-1]      30.8797 (1.0)       75.5230 (1.0)      34.1607 (1.0)      10.7278 (1.0)      31.4653 (1.0)      0.4418 (1.41)          2;2  29.2734 (1.0)          32           1
test_wide_drop_construct[1000-50]     51.5099 (1.67)      98.9728 (1.31)     56.5733 (1.66)     14.0950 (1.31)     52.0354 (1.65)     0.3142 (1.0)           2;2  17.6762 (0.60)         20           1
test_wide_drop_construct[1000-99]     71.7183 (2.32)     116.9485 (1.55)     77.0926 (2.26)     14.9489 (1.39)     72.1563 (2.29)     0.6274 (2.00)          1;1  12.9714 (0.44)          9           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------- benchmark 'test_wide_drop_construct wide_table=10000': 3 tests ---------------------------------------------------------------------
Name (time in ms)                           Min                 Max                Mean             StdDev              Median                IQR            Outliers     OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[10000-1]      362.6293 (1.0)      369.5271 (1.0)      366.2398 (1.0)       2.6692 (1.0)      365.5609 (1.0)       3.7630 (1.0)           2;0  2.7305 (1.0)           5           1
test_wide_drop_construct[10000-50]     576.4179 (1.59)     627.0368 (1.70)     596.8235 (1.63)     23.3386 (8.74)     585.5543 (1.60)     41.1440 (10.93)         1;0  1.6755 (0.61)          5           1
test_wide_drop_construct[10000-99]     803.4628 (2.22)     863.5623 (2.34)     845.3921 (2.31)     24.0425 (9.01)     855.6460 (2.34)     20.5968 (5.47)          1;1  1.1829 (0.43)          5           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Still scaling with the number of columns, but overall doing fewer full scans of all columns.

Compilation

--------------------------------------------------------------------------- benchmark 'test_wide_drop_compile wide_table=10': 3 tests ----------------------------------------------------------------------------
Name (time in us)                        Min                   Max                  Mean             StdDev                Median                IQR            Outliers         OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[10-1]        345.9230 (1.0)      1,363.8150 (1.0)        362.9936 (1.0)      29.7177 (1.13)       359.9900 (1.0)      11.1312 (1.0)         46;48  2,754.8695 (1.0)        1801           1
test_wide_drop_compile[10-50]     1,744.1130 (5.04)     1,896.1190 (1.39)     1,794.3696 (4.94)     31.3941 (1.20)     1,789.0170 (4.97)     25.5180 (2.29)       131;60    557.2988 (0.20)        506           1
test_wide_drop_compile[10-99]     1,455.6380 (4.21)     1,615.3990 (1.18)     1,495.9611 (4.12)     26.1993 (1.0)      1,493.1040 (4.15)     25.5780 (2.30)       118;41    668.4666 (0.24)        652           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------- benchmark 'test_wide_drop_compile wide_table=100': 3 tests -------------------------------------------------------------------
Name (time in ms)                      Min                Max               Mean            StdDev             Median               IQR            Outliers       OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[100-1]       8.4531 (1.21)     51.9655 (5.58)      8.9061 (1.25)     4.0507 (21.57)     8.5271 (1.20)     0.0510 (1.0)           1;2  112.2823 (0.80)        115           1
test_wide_drop_compile[100-50]     10.6639 (1.53)     11.5317 (1.24)     10.7723 (1.51)     0.1878 (1.0)      10.7165 (1.51)     0.0901 (1.77)          1;1   92.8311 (0.66)         20           1
test_wide_drop_compile[100-99]      6.9695 (1.0)       9.3140 (1.0)       7.1432 (1.0)      0.2645 (1.41)      7.0957 (1.0)      0.0840 (1.65)          5;5  139.9940 (1.0)         137           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------- benchmark 'test_wide_drop_compile wide_table=1000': 3 tests ---------------------------------------------------------------------
Name (time in ms)                       Min                 Max                Mean             StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[1000-1]      77.0119 (1.20)     124.5776 (1.15)      84.6448 (1.25)     17.1849 (1.60)     77.5110 (1.19)     1.3668 (1.88)          2;2  11.8141 (0.80)         13           1
test_wide_drop_compile[1000-50]     97.6675 (1.52)     149.0500 (1.38)     105.6067 (1.56)     19.1694 (1.78)     98.1521 (1.51)     1.5220 (2.10)          1;1   9.4691 (0.64)          7           1
test_wide_drop_compile[1000-99]     64.3628 (1.0)      108.1386 (1.0)       67.7730 (1.0)      10.7716 (1.0)      65.1457 (1.0)      0.7255 (1.0)           1;1  14.7551 (1.0)          16           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------- benchmark 'test_wide_drop_compile wide_table=10000': 3 tests -------------------------------------------------------------------------
Name (time in ms)                           Min                   Max                  Mean             StdDev                Median                IQR            Outliers     OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[10000-1]        966.8117 (1.28)     1,011.6055 (1.23)       982.9185 (1.27)     17.2819 (2.27)       979.6515 (1.28)     19.2930 (1.77)          1;0  1.0174 (0.79)          5           1
test_wide_drop_compile[10000-50]     1,239.3541 (1.64)     1,257.4882 (1.53)     1,251.3766 (1.62)      7.6045 (1.0)      1,254.3803 (1.64)     10.9124 (1.0)           1;0  0.7991 (0.62)          5           1
test_wide_drop_compile[10000-99]       755.1665 (1.0)        823.6795 (1.0)        773.4736 (1.0)      28.2772 (3.72)       762.6143 (1.0)      19.0754 (1.75)          1;1  1.2929 (1.0)           5           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

This now scales with min(len(dropped_columns), len(included_columns)). The worst case is when 50% of the columns are dropped. On either side of that number the situation gets better because one of the numbers is smaller than the other.

@cpcloud
Copy link
Member Author

cpcloud commented Jun 26, 2024

Here are the comparison benchmarks for benchmarks that passed:

Compilation

------------------------------------------------------------------------------------ benchmark 'test_wide_drop_compile[10-1]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                    Min                   Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[10-1] (0009_49fa537)     347.0570 (1.0)      1,012.9990 (1.22)     360.3105 (1.0)      22.5341 (1.16)     357.2370 (1.0)      6.3310 (1.08)        55;76        2.7754 (1.0)        2107           1
test_wide_drop_compile[10-1] (NOW)              356.0980 (1.03)       829.0300 (1.0)      367.1064 (1.02)     19.4419 (1.0)      363.8230 (1.02)     5.8610 (1.0)         56;71        2.7240 (0.98)       2194           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_wide_drop_compile[10-50]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                   Min                Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[10-50] (0009_49fa537)     1.6311 (1.0)      67.9432 (1.48)     1.8130 (1.0)      3.0473 (1.62)     1.6626 (1.0)      0.0201 (1.14)         1;72  551.5815 (1.0)         473           1
test_wide_drop_compile[10-50] (NOW)              1.7575 (1.08)     46.0382 (1.0)      1.8698 (1.03)     1.8819 (1.0)      1.7781 (1.07)     0.0176 (1.0)          1;89  534.8092 (0.97)        553           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_wide_drop_compile[10-99]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                   Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[10-99] (0009_49fa537)     1.0194 (1.0)      1.1345 (1.0)      1.0350 (1.0)      0.0215 (1.0)      1.0300 (1.0)      0.0081 (1.0)         56;63  966.1845 (1.0)         942           1
test_wide_drop_compile[10-99] (NOW)              1.4928 (1.46)     1.6372 (1.44)     1.5155 (1.46)     0.0223 (1.04)     1.5094 (1.47)     0.0116 (1.42)        44;49  659.8429 (0.68)        617           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------- benchmark 'test_wide_drop_compile[100-1]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                    Min                Max               Mean            StdDev             Median               IQR            Outliers       OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[100-1] (0009_49fa537)     16.6067 (2.09)     17.1536 (2.09)     16.7943 (2.08)     0.1344 (1.35)     16.7840 (2.08)     0.2136 (1.15)         20;0   59.5441 (0.48)         59           1
test_wide_drop_compile[100-1] (NOW)               7.9509 (1.0)       8.2143 (1.0)       8.0722 (1.0)      0.0995 (1.0)       8.0842 (1.0)      0.1862 (1.0)           8;0  123.8826 (1.0)          17           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_wide_drop_compile[100-50]': 2 tests -------------------------------------------------------------------------------
Name (time in ms)                                     Min                Max               Mean            StdDev             Median               IQR            Outliers       OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[100-50] (0009_49fa537)      9.4396 (1.0)       9.7975 (1.0)       9.5248 (1.0)      0.0949 (1.55)      9.4804 (1.0)      0.0877 (1.08)          2;1  104.9887 (1.0)          16           1
test_wide_drop_compile[100-50] (NOW)              10.3446 (1.10)     10.6122 (1.08)     10.4716 (1.10)     0.0612 (1.0)      10.4731 (1.10)     0.0815 (1.0)          29;0   95.4965 (0.91)         93           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_wide_drop_compile[100-99]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                    Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[100-99] (0009_49fa537)     2.0139 (1.0)      2.1670 (1.0)      2.0643 (1.0)      0.0232 (1.0)      2.0595 (1.0)      0.0169 (1.0)         76;46  484.4182 (1.0)         456           1
test_wide_drop_compile[100-99] (NOW)              6.7885 (3.37)     7.0579 (3.26)     6.9001 (3.34)     0.0568 (2.45)     6.8961 (3.35)     0.0854 (5.06)         55;0  144.9246 (0.30)        139           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------- benchmark 'test_wide_drop_compile[1000-1]': 2 tests ---------------------------------------------------------------------------------
Name (time in ms)                                      Min                 Max                Mean             StdDev              Median                IQR            Outliers      OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[1000-1] (0009_49fa537)     158.7531 (2.07)     286.4965 (2.31)     184.4322 (2.21)     57.0560 (3.48)     158.9027 (2.05)     32.2317 (31.75)         1;1   5.4220 (0.45)          5           1
test_wide_drop_compile[1000-1] (NOW)               76.8130 (1.0)      123.8069 (1.0)       83.2822 (1.0)      16.3823 (1.0)       77.5547 (1.0)       1.0151 (1.0)           1;1  12.0074 (1.0)           8           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------- benchmark 'test_wide_drop_compile[1000-50]': 1 tests ------------------------------------
Name (time in ms)                       Min       Max      Mean   StdDev   Median     IQR  Outliers     OPS  Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[1000-50]     97.3127  147.7012  106.8236  19.6334  98.0965  1.1368       2;2  9.3612      11           1
-------------------------------------------------------------------------------------------------------------------------------

------------------------------------- benchmark 'test_wide_drop_compile[1000-99]': 1 tests ------------------------------------
Name (time in ms)                       Min       Max     Mean   StdDev   Median     IQR  Outliers      OPS  Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[1000-99]     62.9960  108.4716  66.5057  11.1994  63.8443  0.6417       1;1  15.0363      16           1
-------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_wide_drop_compile[10000-1]': 2 tests ---------------------------------------------------------------------------
Name (time in s)                                      Min               Max              Mean            StdDev            Median               IQR            Outliers     OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[10000-1] (0009_49fa537)     1.8831 (1.96)     2.1667 (2.20)     2.0503 (2.12)     0.1501 (17.40)    2.1465 (2.22)     0.2785 (33.87)         2;0  0.4877 (0.47)          5           1
test_wide_drop_compile[10000-1] (NOW)              0.9622 (1.0)      0.9842 (1.0)      0.9694 (1.0)      0.0086 (1.0)      0.9675 (1.0)      0.0082 (1.0)           1;0  1.0316 (1.0)           5           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------- benchmark 'test_wide_drop_compile[10000-50]': 1 tests ---------------------------------
Name (time in s)                        Min     Max    Mean  StdDev  Median     IQR  Outliers     OPS  Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[10000-50]     1.2380  1.2531  1.2453  0.0056  1.2461  0.0069       2;0  0.8030       5           1
-------------------------------------------------------------------------------------------------------------------------

-------------------------------------- benchmark 'test_wide_drop_compile[10000-99]': 1 tests --------------------------------------
Name (time in ms)                         Min       Max      Mean   StdDev    Median      IQR  Outliers     OPS  Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_compile[10000-99]     731.1328  789.5454  748.5010  23.3121  740.3010  15.3716       1;1  1.3360       5           1
-----------------------------------------------------------------------------------------------------------------------------------

image

Construction

------------------------------------------------------------------------------------ benchmark 'test_wide_drop_construct[10-1]': 2 tests -------------------------------------------------------------------------------------
Name (time in ns)                                      Min                   Max                Mean             StdDev              Median               IQR             Outliers  OPS (Mops/s)            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[10-1] (0009_49fa537)     535.4988 (4.76)     1,648.5996 (5.78)     551.2802 (4.76)     17.2943 (4.55)     549.0510 (4.76)     4.5009 (2.37)     1230;3099        1.8140 (0.21)      87398          20
test_wide_drop_construct[10-1] (NOW)              112.5198 (1.0)        285.2399 (1.0)      115.7949 (1.0)       3.7970 (1.0)      115.3201 (1.0)      1.8996 (1.0)      1644;1685        8.6360 (1.0)       85238         100
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_wide_drop_construct[10-50]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                     Min                Max              Mean            StdDev            Median               IQR            Outliers         OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[10-50] (0009_49fa537)     2.0220 (3.33)     56.2875 (82.33)    2.1956 (3.56)     2.6335 (580.29)   2.0603 (3.34)     0.0255 (4.54)         1;10    455.4647 (0.28)        424           1
test_wide_drop_construct[10-50] (NOW)              0.6075 (1.0)       0.6837 (1.0)      0.6159 (1.0)      0.0045 (1.0)      0.6161 (1.0)      0.0056 (1.0)        392;38  1,623.6889 (1.0)        1460           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_wide_drop_construct[10-99]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                     Min                Max              Mean            StdDev            Median               IQR            Outliers         OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[10-99] (0009_49fa537)     2.4375 (3.16)     81.7003 (96.59)    2.7294 (3.49)     4.0894 (717.75)   2.5079 (3.21)     0.0302 (7.20)         1;20    366.3873 (0.29)        375           1
test_wide_drop_construct[10-99] (NOW)              0.7717 (1.0)       0.8459 (1.0)      0.7827 (1.0)      0.0057 (1.0)      0.7816 (1.0)      0.0042 (1.0)       327;159  1,277.6765 (1.0)        1214           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------- benchmark 'test_wide_drop_construct[100-1]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                      Min                Max               Mean            StdDev             Median               IQR            Outliers       OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[100-1] (0009_49fa537)     13.2165 (4.27)     13.4489 (3.85)     13.2704 (4.18)     0.0388 (1.08)     13.2641 (4.19)     0.0358 (1.0)          18;6   75.3559 (0.24)         75           1
test_wide_drop_construct[100-1] (NOW)               3.0947 (1.0)       3.4907 (1.0)       3.1721 (1.0)      0.0359 (1.0)       3.1691 (1.0)      0.0360 (1.01)         76;9  315.2472 (1.0)         307           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------- benchmark 'test_wide_drop_construct[100-50]': 2 tests --------------------------------------------------------------------------------
Name (time in ms)                                       Min                 Max               Mean             StdDev             Median               IQR            Outliers       OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[100-50] (0009_49fa537)     69.5039 (13.86)    118.7244 (22.65)    73.0373 (14.23)    12.6398 (247.67)   69.8009 (13.57)    0.1857 (5.95)          1;1   13.6916 (0.07)         15           1
test_wide_drop_construct[100-50] (NOW)               5.0138 (1.0)        5.2412 (1.0)       5.1321 (1.0)       0.0510 (1.0)       5.1434 (1.0)      0.0312 (1.0)         57;46  194.8509 (1.0)         189           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------- benchmark 'test_wide_drop_construct[100-99]': 2 tests ---------------------------------------------------------------------------------
Name (time in ms)                                        Min                 Max                Mean             StdDev              Median               IQR            Outliers       OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[100-99] (0009_49fa537)     104.8313 (14.89)    202.2731 (27.98)    115.3051 (16.20)    30.5606 (812.05)   105.7192 (14.87)    0.5761 (13.02)         1;1    8.6726 (0.06)         10           1
test_wide_drop_construct[100-99] (NOW)                7.0398 (1.0)        7.2294 (1.0)        7.1166 (1.0)       0.0376 (1.0)        7.1109 (1.0)      0.0442 (1.0)          37;5  140.5167 (1.0)         135           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------- benchmark 'test_wide_drop_construct[1000-1]': 2 tests ---------------------------------------------------------------------------------
Name (time in ms)                                        Min                 Max                Mean             StdDev              Median                IQR            Outliers      OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[1000-1] (0009_49fa537)     255.4666 (8.17)     381.5130 (11.97)    281.1369 (8.90)     56.1137 (355.36)   256.3884 (8.12)     32.0986 (123.25)        1;1   3.5570 (0.11)          5           1
test_wide_drop_construct[1000-1] (NOW)               31.2874 (1.0)       31.8718 (1.0)       31.5994 (1.0)       0.1579 (1.0)       31.5938 (1.0)       0.2604 (1.0)          12;0  31.6461 (1.0)          32           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------- benchmark 'test_wide_drop_construct[1000-50]': 1 tests ------------------------------------
Name (time in ms)                         Min       Max     Mean   StdDev   Median     IQR  Outliers      OPS  Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[1000-50]     51.5820  108.7799  54.9660  13.0334  51.9721  0.2425       1;1  18.1931      19           1
---------------------------------------------------------------------------------------------------------------------------------

------------------------------------- benchmark 'test_wide_drop_construct[1000-99]': 1 tests ------------------------------------
Name (time in ms)                         Min       Max     Mean   StdDev   Median     IQR  Outliers      OPS  Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[1000-99]     72.3577  119.3239  77.8347  15.5597  72.7420  0.4079       1;1  12.8477       9           1
---------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_wide_drop_construct[10000-1]': 2 tests -----------------------------------------------------------------------------
Name (time in s)                                         Min                Max               Mean            StdDev             Median               IQR            Outliers     OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[10000-1] (0009_49fa537)     13.8786 (39.57)    14.4703 (40.17)    14.3036 (40.13)    0.2441 (66.90)    14.4242 (40.43)    0.2298 (47.98)         1;0  0.0699 (0.02)          5           1
test_wide_drop_construct[10000-1] (NOW)               0.3507 (1.0)       0.3602 (1.0)       0.3564 (1.0)      0.0036 (1.0)       0.3567 (1.0)      0.0048 (1.0)           2;0  2.8058 (1.0)           5           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------- benchmark 'test_wide_drop_construct[10000-50]': 1 tests --------------------------------------
Name (time in ms)                           Min       Max      Mean   StdDev    Median      IQR  Outliers     OPS  Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[10000-50]     572.4034  622.5245  599.2539  23.8446  612.8374  42.1003       2;0  1.6687       5           1
-------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------- benchmark 'test_wide_drop_construct[10000-99]': 1 tests --------------------------------------
Name (time in ms)                           Min       Max      Mean   StdDev    Median      IQR  Outliers     OPS  Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------
test_wide_drop_construct[10000-99]     765.4037  827.2840  807.1245  25.3184  813.3571  32.5968       1;0  1.2390       5           1
-------------------------------------------------------------------------------------------------------------------------------------

image

@cpcloud
Copy link
Member Author

cpcloud commented Jun 26, 2024

For construction, things are better across the board.

For compilation, it's kind of a wash except for the important fact that some cases of drop simply didn't work because construction the expression overflowed the Python stack. I think then this is overall a net improvement.

@cpcloud cpcloud requested review from gforsyth and jcrist June 26, 2024 13:47
@cpcloud
Copy link
Member Author

cpcloud commented Jun 26, 2024

Clouds are passing:

cloud in 🌐 falcon in …/ibis on  drop-perf is 📦 v9.1.0 via 🐍 v3.10.14 via ❄️   impure (ibis-3.10.14-env) took 10s
❯ pytest -m 'bigquery or snowflake' --dist loadgroup -q -n 8 --snapshot-update
bringing up nodes...
x............................x.s.........................x......x.......x.........x................x........x.......................x.......................................x....x.x.... [  5%]
......xx......x.x...x.......................xxx.........x..............x..............x........x.x.......x.....................................x...x...........s........................ [ 10%]
.x......x................x.......x................................x..........x........................................x........x..xx......x...x...............x..xxx.........xx..x...x.. [ 15%]
..x......x........x.............x......................x...............x............x................................................xx............x.......x......x...x................. [ 20%]
.....x........x.....x...x.x....x.................................ssssssssssssssssss.ssss..x....x.x.x............x.......xxx.xx.....x.x...xx.xx.x...xx.x...x....x..x..xx..x.x.xx.x.x.xx.. [ 25%]
.xx.xxxx.....xxxxx.x.xxx..xxxxx.xx..x.xx.xxxxxxxxxx.xxxxxxxxxxx.......................................x..x..x.x..xxx....s.....xx..xx...xxxx..x.xxx.x....................xxx.x.........xx [ 30%]
.x....x............................xx.....x...x.x......x......x...............xx.x.....x..........x....xx....x.........x.xxx....x..........x..s.x....................................... [ 35%]
....s.........................s..................x.....x........................x...................x..x.......xx........x...............x...........x....x.......x.................x... [ 40%]
..x...............x....x...................x.x..........x.......................x.....s.x.....x....................................................xx..x.x..xx..x..x...x...x..x......x.. [ 45%]
.x.......x.x...........x..........xx..............x.x.x.....x...xx...............x.....xxx...xxx..xxx.....x.xx.xxx.xx..xxxx....x.x.xx.x....s.x....x........x.........x.sx.......x......x [ 50%]
......x....s.xxx.x..s.....................x.xx........x.....x..x..x..........x............x...x..x.x...x......x..x..xx...x......x.x.x......................x....x.....xx.x...........x.. [ 55%]
............x............x....x........x............x.....x.....x..............................................xx..x...........x.....x...............................x............x..... [ 60%]
...............................................x....................x.......................x......................x.......................x.x.x.xxxxx.xxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxx [ 65%]
xxxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.x.....x.........x...........x.......xx...x..x....xx..x.......xx.x.xx.xx...xxx.x.xxxx.x.x [ 70%]
.x...x.......x..x.....................x........x.............x....xxxxxx...x.....xx.......x.......x....x...x...x.............x.....x..x......xx.x........x.x.........x..........x.xx..x. [ 75%]
.........................s........................s...s.............................x.....x......x.......x.........xx............xx....xx........x.......x........x....x..x............x [ 80%]
............x....x.x....x...x........x.x...x...........x....................x...........................x............................................................................... [ 85%]
........................................................................................................................................................................................ [ 90%]
...........................x........................................................................................................................................ss.................. [ 95%]
...............................x...........x..........x..................................s................................................................s....                          [100%]
3063 passed, 40 skipped, 552 xfailed in 623.14s (0:10:23)

gforsyth
gforsyth approved these changes Jun 26, 2024
View reviewed changes
Copy link
Member

@gforsyth gforsyth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lovely, looks like a win to me

@gforsyth gforsyth merged commit 1c6eb5c into ibis-project:main Jun 26, 2024
76 checks passed
@cpcloud cpcloud deleted the drop-perf branch June 26, 2024 16:09
@cpcloud cpcloud mentioned this pull request Jul 21, 2024
Closed
1 task
@deepyaman deepyaman mentioned this pull request Jul 23, 2024
Merged
@ncclementi ncclementi mentioned this pull request Aug 13, 2024
Open
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gforsyth gforsyth gforsyth approved these changes

@jcrist jcrist Awaiting requested review from jcrist

Assignees
No one assigned
Labels
internals Issues or PRs related to ibis's internal APIs performance Issues related to ibis's performance ux User experience related issues
Projects
None yet
Milestone
9.2
Development

Successfully merging this pull request may close these issues.
2 participants
@cpcloud @gforsyth