New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up incremental sequential writes and batch creation from unsorted data #199

Merged
merged 5 commits into from Dec 23, 2017

Conversation

Projects
None yet
3 participants
@richardstartin
Copy link
Contributor

richardstartin commented Dec 17, 2017

This pull request makes it possible to write sequentially and incrementally to a RoaringBitmap efficiently, without collecting values in an int[] beforehand.

Usage

OrderedWriter wraps a RoaringBitmap and requires the caller to flush it to ensure all added data is saved to the bitmap.

    OrderedWriter writer = new OrderedWriter(rb);
    // add data incrementally, the most signficant 16 bits must be in ascending unsigned order
    for (int i : values) {
      writer.add(i);
    }
    // flush all buffered data down to the underlying bitmap
    writer.flush();

Testing

The basic behaviour of OrderedWriter is tested in TestOrderedWriter.

That a RoaringBitmap built by an OrderedWriter is equivalent to one built by RoaringBitmap.bitmapOf is tested for random bitmaps of various size in OrderedWriterRandomisedTest

Benchmarks

WriteSequential compares RoaringBitmap.bitmapOf (i.e. what you could do if you had the entire array in one go), sequential calls to RoaringBitmap.add and OrderedWriter. Data is generated for a range of sizes and a range of "runniness" (i.e. how likely it is that the data can be run length encoded.) The results are good: OrderedWriter is faster than RoaringBitmap.bitmapOf for small data sets. For larger datasets the performance is of similar performance, and a great deal better than sequential calls to RoaringBitmap.add.

Benchmark                                    (randomness)    (size)  Mode  Cnt       Score       Error  Units
WriteSequential.buildRoaringBitmap                    0.1     10000  avgt    5      53.628 ▒     2.544  us/op
WriteSequential.buildRoaringBitmap                    0.1    100000  avgt    5     354.691 ▒     9.786  us/op
WriteSequential.buildRoaringBitmap                    0.1   1000000  avgt    5    3530.126 ▒   157.438  us/op
WriteSequential.buildRoaringBitmap                    0.1  10000000  avgt    5   31400.821 ▒   529.543  us/op
WriteSequential.buildRoaringBitmap                    0.5     10000  avgt    5      52.083 ▒     2.663  us/op
WriteSequential.buildRoaringBitmap                    0.5    100000  avgt    5     353.520 ▒     7.453  us/op
WriteSequential.buildRoaringBitmap                    0.5   1000000  avgt    5    3492.611 ▒   111.276  us/op
WriteSequential.buildRoaringBitmap                    0.5  10000000  avgt    5   37178.051 ▒  9069.530  us/op
WriteSequential.buildRoaringBitmap                    0.9     10000  avgt    5      59.458 ▒     8.530  us/op
WriteSequential.buildRoaringBitmap                    0.9    100000  avgt    5     360.718 ▒    52.790  us/op
WriteSequential.buildRoaringBitmap                    0.9   1000000  avgt    5    3766.560 ▒  1198.259  us/op
WriteSequential.buildRoaringBitmap                    0.9  10000000  avgt    5   33221.955 ▒  2151.770  us/op
WriteSequential.incrementalNativeAdd                  0.1     10000  avgt    5      95.012 ▒     8.962  us/op
WriteSequential.incrementalNativeAdd                  0.1    100000  avgt    5     780.217 ▒   106.684  us/op
WriteSequential.incrementalNativeAdd                  0.1   1000000  avgt    5    9265.335 ▒   846.230  us/op
WriteSequential.incrementalNativeAdd                  0.1  10000000  avgt    5  101037.210 ▒  6457.910  us/op
WriteSequential.incrementalNativeAdd                  0.5     10000  avgt    5     111.847 ▒    14.197  us/op
WriteSequential.incrementalNativeAdd                  0.5    100000  avgt    5     887.937 ▒    69.129  us/op
WriteSequential.incrementalNativeAdd                  0.5   1000000  avgt    5    7620.004 ▒   500.564  us/op
WriteSequential.incrementalNativeAdd                  0.5  10000000  avgt    5  101455.646 ▒ 14751.135  us/op
WriteSequential.incrementalNativeAdd                  0.9     10000  avgt    5      93.574 ▒     8.004  us/op
WriteSequential.incrementalNativeAdd                  0.9    100000  avgt    5     882.819 ▒    46.454  us/op
WriteSequential.incrementalNativeAdd                  0.9   1000000  avgt    5    7507.996 ▒   426.764  us/op
WriteSequential.incrementalNativeAdd                  0.9  10000000  avgt    5   98675.524 ▒  1715.163  us/op
WriteSequential.incrementalUseOrderedWriter           0.1     10000  avgt    5      26.338 ▒     1.190  us/op
WriteSequential.incrementalUseOrderedWriter           0.1    100000  avgt    5     367.215 ▒    26.818  us/op
WriteSequential.incrementalUseOrderedWriter           0.1   1000000  avgt    5    3832.171 ▒   347.830  us/op
WriteSequential.incrementalUseOrderedWriter           0.1  10000000  avgt    5   43193.867 ▒ 20770.625  us/op
WriteSequential.incrementalUseOrderedWriter           0.5     10000  avgt    5      32.928 ▒    10.230  us/op
WriteSequential.incrementalUseOrderedWriter           0.5    100000  avgt    5     394.717 ▒    93.505  us/op
WriteSequential.incrementalUseOrderedWriter           0.5   1000000  avgt    5    3728.619 ▒   177.213  us/op
WriteSequential.incrementalUseOrderedWriter           0.5  10000000  avgt    5   36694.794 ▒  1931.691  us/op
WriteSequential.incrementalUseOrderedWriter           0.9     10000  avgt    5      26.152 ▒     1.391  us/op
WriteSequential.incrementalUseOrderedWriter           0.9    100000  avgt    5     366.515 ▒    14.643  us/op
WriteSequential.incrementalUseOrderedWriter           0.9   1000000  avgt    5    3696.070 ▒   107.797  us/op
WriteSequential.incrementalUseOrderedWriter           0.9  10000000  avgt    5   36587.744 ▒  1793.303  us/op

@coveralls

This comment has been minimized.

Copy link

coveralls commented Dec 17, 2017

Coverage Status

Coverage increased (+0.03%) to 90.475% when pulling f88ea0a on richardstartin:speed-up-incremental-writes into b9a2107 on RoaringBitmap:master.

combine partial radix sort with ordered writer for order of magnitude…
… faster creation of bitmaps from unsorted data
@richardstartin

This comment has been minimized.

Copy link
Contributor

richardstartin commented Dec 18, 2017

OrderedWriter combined with a partial unsigned radix sort on the most significant 16 bits enables an order of magnitude improvement on RoaringBitmap.bitmapOf with unsorted data. See WriteUnordered for details.

Benchmark                               (randomness)    (size)  Mode  Cnt        Score        Error  Units
WriteUnordered.bitmapOf                          0.1     10000  avgt    5     1037.003 ▒     26.395  us/op
WriteUnordered.bitmapOf                          0.1    100000  avgt    5    12299.824 ▒    294.524  us/op
WriteUnordered.bitmapOf                          0.1   1000000  avgt    5   170323.092 ▒   5505.515  us/op
WriteUnordered.bitmapOf                          0.1  10000000  avgt    5  3435560.427 ▒ 682797.878  us/op
WriteUnordered.bitmapOf                          0.5     10000  avgt    5     1073.153 ▒     22.407  us/op
WriteUnordered.bitmapOf                          0.5    100000  avgt    5    12419.177 ▒    169.826  us/op
WriteUnordered.bitmapOf                          0.5   1000000  avgt    5   185998.036 ▒  13497.803  us/op
WriteUnordered.bitmapOf                          0.5  10000000  avgt    5  3583094.295 ▒ 244335.621  us/op
WriteUnordered.bitmapOf                          0.9     10000  avgt    5     1171.972 ▒    274.890  us/op
WriteUnordered.bitmapOf                          0.9    100000  avgt    5    15975.240 ▒   4470.437  us/op
WriteUnordered.bitmapOf                          0.9   1000000  avgt    5   200199.930 ▒   9340.904  us/op
WriteUnordered.bitmapOf                          0.9  10000000  avgt    5  3745673.287 ▒ 446143.469  us/op
WriteUnordered.bitmapOfUnordered                 0.1     10000  avgt    5      178.555 ▒      3.534  us/op
WriteUnordered.bitmapOfUnordered                 0.1    100000  avgt    5     2128.869 ▒     49.906  us/op
WriteUnordered.bitmapOfUnordered                 0.1   1000000  avgt    5    22400.536 ▒   4287.448  us/op
WriteUnordered.bitmapOfUnordered                 0.1  10000000  avgt    5   256592.935 ▒  48725.402  us/op
WriteUnordered.bitmapOfUnordered                 0.5     10000  avgt    5      167.748 ▒      4.057  us/op
WriteUnordered.bitmapOfUnordered                 0.5    100000  avgt    5     1760.488 ▒     52.177  us/op
WriteUnordered.bitmapOfUnordered                 0.5   1000000  avgt    5    38268.539 ▒  34597.447  us/op
WriteUnordered.bitmapOfUnordered                 0.5  10000000  avgt    5   197447.337 ▒  10637.582  us/op
WriteUnordered.bitmapOfUnordered                 0.9     10000  avgt    5      157.787 ▒      4.045  us/op
WriteUnordered.bitmapOfUnordered                 0.9    100000  avgt    5     1626.672 ▒     79.114  us/op
WriteUnordered.bitmapOfUnordered                 0.9   1000000  avgt    5    16971.516 ▒    482.369  us/op
WriteUnordered.bitmapOfUnordered                 0.9  10000000  avgt    5   187644.489 ▒  63699.982  us/op
WriteUnordered.partialSortThenBitmapOf           0.1     10000  avgt    5     1008.949 ▒    130.725  us/op
WriteUnordered.partialSortThenBitmapOf           0.1    100000  avgt    5     9523.936 ▒    118.937  us/op
WriteUnordered.partialSortThenBitmapOf           0.1   1000000  avgt    5   215872.621 ▒ 438194.652  us/op
WriteUnordered.partialSortThenBitmapOf           0.1  10000000  avgt    5   975361.402 ▒  30206.634  us/op
WriteUnordered.partialSortThenBitmapOf           0.5     10000  avgt    5      992.042 ▒     31.578  us/op
WriteUnordered.partialSortThenBitmapOf           0.5    100000  avgt    5     9750.091 ▒    466.963  us/op
WriteUnordered.partialSortThenBitmapOf           0.5   1000000  avgt    5    98795.285 ▒   1870.356  us/op
WriteUnordered.partialSortThenBitmapOf           0.5  10000000  avgt    5   963903.476 ▒  34484.348  us/op
WriteUnordered.partialSortThenBitmapOf           0.9     10000  avgt    5     1048.502 ▒     95.204  us/op
WriteUnordered.partialSortThenBitmapOf           0.9    100000  avgt    5    11216.359 ▒   7694.827  us/op
WriteUnordered.partialSortThenBitmapOf           0.9   1000000  avgt    5   105528.071 ▒   8956.799  us/op
WriteUnordered.partialSortThenBitmapOf           0.9  10000000  avgt    5  1071744.101 ▒  50138.981  us/op
WriteUnordered.setupCost                         0.1     10000  avgt    5        6.648 ▒      0.221  us/op
WriteUnordered.setupCost                         0.1    100000  avgt    5       60.891 ▒     22.480  us/op
WriteUnordered.setupCost                         0.1   1000000  avgt    5      815.620 ▒     54.018  us/op
WriteUnordered.setupCost                         0.1  10000000  avgt    5    13378.423 ▒    102.168  us/op
WriteUnordered.setupCost                         0.5     10000  avgt    5        6.605 ▒      0.081  us/op
WriteUnordered.setupCost                         0.5    100000  avgt    5       61.444 ▒     23.614  us/op
WriteUnordered.setupCost                         0.5   1000000  avgt    5      808.808 ▒     20.525  us/op
WriteUnordered.setupCost                         0.5  10000000  avgt    5    13370.367 ▒    266.791  us/op
WriteUnordered.setupCost                         0.9     10000  avgt    5        6.623 ▒      0.117  us/op
WriteUnordered.setupCost                         0.9    100000  avgt    5       61.346 ▒     22.610  us/op
WriteUnordered.setupCost                         0.9   1000000  avgt    5      797.639 ▒     31.887  us/op
WriteUnordered.setupCost                         0.9  10000000  avgt    5    13395.827 ▒    293.565  us/op
WriteUnordered.sortThenBitmapOf                  0.1     10000  avgt    5      606.933 ▒     11.337  us/op
WriteUnordered.sortThenBitmapOf                  0.1    100000  avgt    5     8744.447 ▒    909.596  us/op
WriteUnordered.sortThenBitmapOf                  0.1   1000000  avgt    5   101774.330 ▒  17236.479  us/op
WriteUnordered.sortThenBitmapOf                  0.1  10000000  avgt    5  1121935.627 ▒ 102644.996  us/op
WriteUnordered.sortThenBitmapOf                  0.5     10000  avgt    5      619.765 ▒     31.629  us/op
WriteUnordered.sortThenBitmapOf                  0.5    100000  avgt    5     8102.926 ▒     64.866  us/op
WriteUnordered.sortThenBitmapOf                  0.5   1000000  avgt    5    89125.467 ▒   3260.221  us/op
WriteUnordered.sortThenBitmapOf                  0.5  10000000  avgt    5  1039797.302 ▒  32671.824  us/op
WriteUnordered.sortThenBitmapOf                  0.9     10000  avgt    5      625.968 ▒     96.403  us/op
WriteUnordered.sortThenBitmapOf                  0.9    100000  avgt    5     7564.899 ▒    139.118  us/op
WriteUnordered.sortThenBitmapOf                  0.9   1000000  avgt    5    93368.292 ▒  10483.635  us/op
WriteUnordered.sortThenBitmapOf                  0.9  10000000  avgt    5  1070815.944 ▒ 117055.656  us/op

@richardstartin richardstartin changed the title Speed up incremental writes Speed up incremental sequential writes and batch creation from unsorted data Dec 18, 2017

@coveralls

This comment has been minimized.

Copy link

coveralls commented Dec 18, 2017

Coverage Status

Coverage increased (+0.08%) to 90.529% when pulling 0d4a981 on richardstartin:speed-up-incremental-writes into b9a2107 on RoaringBitmap:master.

@lemire

This comment has been minimized.

Copy link
Member

lemire commented Dec 18, 2017

Amazing.

@lemire

This comment has been minimized.

Copy link
Member

lemire commented Dec 18, 2017

This seems to partially address the following issue: #190

@richardstartin

This comment has been minimized.

Copy link
Contributor

richardstartin commented Dec 18, 2017

I didn't set out to address #190 - this was a very easy modification of the cache idea in #196 (which I don't think is worth exploring further). I think the second half of #190 is of dubious merit in Java given that you can do Java 8+ friendly forEach iteration already: I'm unlikely to spend the time finding out.

Also, #160 proposes to solve the performance issue with unsorted data, but I think this gives a larger and more consistent speed up than was reported, though I would be interested if anyone can contrive scenarios that cause bitmapOfUnordered to perform poorly.

@richardstartin

This comment has been minimized.

Copy link
Contributor

richardstartin commented Dec 18, 2017

Made two simple tweaks OrderedWriter to make it faster for ordered sequential writes than RoaringBitmap.bitmapOf

Benchmark                                    (randomness)    (size)  Mode  Cnt       Score       Error  Units
WriteSequential.buildRoaringBitmap                    0.1     10000  avgt    5      54.263 ▒     3.393  us/op
WriteSequential.buildRoaringBitmap                    0.1    100000  avgt    5     355.188 ▒    15.234  us/op
WriteSequential.buildRoaringBitmap                    0.1   1000000  avgt    5    3567.839 ▒   135.149  us/op
WriteSequential.buildRoaringBitmap                    0.1  10000000  avgt    5   31982.046 ▒  1227.325  us/op
WriteSequential.buildRoaringBitmap                    0.5     10000  avgt    5      53.855 ▒     0.887  us/op
WriteSequential.buildRoaringBitmap                    0.5    100000  avgt    5     357.671 ▒    14.111  us/op
WriteSequential.buildRoaringBitmap                    0.5   1000000  avgt    5    3556.152 ▒   243.671  us/op
WriteSequential.buildRoaringBitmap                    0.5  10000000  avgt    5   34385.971 ▒  3864.143  us/op
WriteSequential.buildRoaringBitmap                    0.9     10000  avgt    5      59.354 ▒    10.385  us/op
WriteSequential.buildRoaringBitmap                    0.9    100000  avgt    5     374.245 ▒    54.485  us/op
WriteSequential.buildRoaringBitmap                    0.9   1000000  avgt    5    3712.684 ▒   657.964  us/op
WriteSequential.buildRoaringBitmap                    0.9  10000000  avgt    5   37223.976 ▒  4691.297  us/op
WriteSequential.incrementalNativeAdd                  0.1     10000  avgt    5     115.213 ▒    31.909  us/op
WriteSequential.incrementalNativeAdd                  0.1    100000  avgt    5     911.925 ▒   127.922  us/op
WriteSequential.incrementalNativeAdd                  0.1   1000000  avgt    5    8889.490 ▒   320.821  us/op
WriteSequential.incrementalNativeAdd                  0.1  10000000  avgt    5  102819.877 ▒ 14247.868  us/op
WriteSequential.incrementalNativeAdd                  0.5     10000  avgt    5     116.878 ▒    28.232  us/op
WriteSequential.incrementalNativeAdd                  0.5    100000  avgt    5     947.076 ▒   128.255  us/op
WriteSequential.incrementalNativeAdd                  0.5   1000000  avgt    5    7190.443 ▒   202.012  us/op
WriteSequential.incrementalNativeAdd                  0.5  10000000  avgt    5   98843.303 ▒  4325.924  us/op
WriteSequential.incrementalNativeAdd                  0.9     10000  avgt    5     101.694 ▒     6.579  us/op
WriteSequential.incrementalNativeAdd                  0.9    100000  avgt    5     816.411 ▒    65.678  us/op
WriteSequential.incrementalNativeAdd                  0.9   1000000  avgt    5    9114.624 ▒   412.152  us/op
WriteSequential.incrementalNativeAdd                  0.9  10000000  avgt    5  108793.694 ▒ 22562.527  us/op
WriteSequential.incrementalUseOrderedWriter           0.1     10000  avgt    5      23.573 ▒     5.962  us/op
WriteSequential.incrementalUseOrderedWriter           0.1    100000  avgt    5     289.588 ▒    36.814  us/op
WriteSequential.incrementalUseOrderedWriter           0.1   1000000  avgt    5    2785.659 ▒    49.385  us/op
WriteSequential.incrementalUseOrderedWriter           0.1  10000000  avgt    5   29489.758 ▒  2601.390  us/op
WriteSequential.incrementalUseOrderedWriter           0.5     10000  avgt    5      23.570 ▒     1.536  us/op
WriteSequential.incrementalUseOrderedWriter           0.5    100000  avgt    5     276.488 ▒     9.662  us/op
WriteSequential.incrementalUseOrderedWriter           0.5   1000000  avgt    5    2799.408 ▒   198.770  us/op
WriteSequential.incrementalUseOrderedWriter           0.5  10000000  avgt    5   28313.626 ▒  1976.042  us/op
WriteSequential.incrementalUseOrderedWriter           0.9     10000  avgt    5      22.345 ▒     1.574  us/op
WriteSequential.incrementalUseOrderedWriter           0.9    100000  avgt    5     280.205 ▒    36.987  us/op
WriteSequential.incrementalUseOrderedWriter           0.9   1000000  avgt    5    2779.732 ▒    93.456  us/op
WriteSequential.incrementalUseOrderedWriter           0.9  10000000  avgt    5   30568.591 ▒  2140.826  us/op

Benchmark                               (randomness)    (size)  Mode  Cnt        Score        Error  Units
WriteUnordered.bitmapOf                          0.1     10000  avgt    5     1058.106 ▒     76.013  us/op
WriteUnordered.bitmapOf                          0.1    100000  avgt    5    12323.905 ▒    976.680  us/op
WriteUnordered.bitmapOf                          0.1   1000000  avgt    5   171812.526 ▒   9593.879  us/op
WriteUnordered.bitmapOf                          0.1  10000000  avgt    5  3376296.157 ▒ 170362.195  us/op
WriteUnordered.bitmapOf                          0.5     10000  avgt    5     1096.663 ▒    477.795  us/op
WriteUnordered.bitmapOf                          0.5    100000  avgt    5    12836.177 ▒   1674.540  us/op
WriteUnordered.bitmapOf                          0.5   1000000  avgt    5   171998.126 ▒   4176.000  us/op
WriteUnordered.bitmapOf                          0.5  10000000  avgt    5  3707804.439 ▒ 974532.361  us/op
WriteUnordered.bitmapOf                          0.9     10000  avgt    5     1124.881 ▒     65.673  us/op
WriteUnordered.bitmapOf                          0.9    100000  avgt    5    14585.589 ▒   1894.788  us/op
WriteUnordered.bitmapOf                          0.9   1000000  avgt    5   198506.813 ▒   8552.218  us/op
WriteUnordered.bitmapOf                          0.9  10000000  avgt    5  3723942.934 ▒ 423704.363  us/op
WriteUnordered.bitmapOfUnordered                 0.1     10000  avgt    5      174.583 ▒     17.475  us/op
WriteUnordered.bitmapOfUnordered                 0.1    100000  avgt    5     1768.613 ▒     86.543  us/op
WriteUnordered.bitmapOfUnordered                 0.1   1000000  avgt    5    17889.705 ▒    135.714  us/op
WriteUnordered.bitmapOfUnordered                 0.1  10000000  avgt    5   192645.352 ▒   6482.726  us/op
WriteUnordered.bitmapOfUnordered                 0.5     10000  avgt    5      157.351 ▒      3.254  us/op
WriteUnordered.bitmapOfUnordered                 0.5    100000  avgt    5     1674.919 ▒     90.138  us/op
WriteUnordered.bitmapOfUnordered                 0.5   1000000  avgt    5    16900.458 ▒    778.999  us/op
WriteUnordered.bitmapOfUnordered                 0.5  10000000  avgt    5   185399.320 ▒   4383.485  us/op
WriteUnordered.bitmapOfUnordered                 0.9     10000  avgt    5      145.642 ▒      1.257  us/op
WriteUnordered.bitmapOfUnordered                 0.9    100000  avgt    5     1515.845 ▒     82.914  us/op
WriteUnordered.bitmapOfUnordered                 0.9   1000000  avgt    5    15807.597 ▒    811.048  us/op
WriteUnordered.bitmapOfUnordered                 0.9  10000000  avgt    5   167863.490 ▒   3501.132  us/op
WriteUnordered.partialSortThenBitmapOf           0.1     10000  avgt    5     1060.152 ▒    168.802  us/op
WriteUnordered.partialSortThenBitmapOf           0.1    100000  avgt    5    10942.731 ▒    347.583  us/op
WriteUnordered.partialSortThenBitmapOf           0.1   1000000  avgt    5   100606.506 ▒  24705.341  us/op
WriteUnordered.partialSortThenBitmapOf           0.1  10000000  avgt    5  1035448.545 ▒ 157383.713  us/op
WriteUnordered.partialSortThenBitmapOf           0.5     10000  avgt    5     1029.883 ▒    100.291  us/op
WriteUnordered.partialSortThenBitmapOf           0.5    100000  avgt    5    10472.509 ▒    832.719  us/op
WriteUnordered.partialSortThenBitmapOf           0.5   1000000  avgt    5   101144.032 ▒  16908.087  us/op
WriteUnordered.partialSortThenBitmapOf           0.5  10000000  avgt    5   958242.087 ▒  39650.946  us/op
WriteUnordered.partialSortThenBitmapOf           0.9     10000  avgt    5     1008.413 ▒     70.999  us/op
WriteUnordered.partialSortThenBitmapOf           0.9    100000  avgt    5    10458.340 ▒    600.416  us/op
WriteUnordered.partialSortThenBitmapOf           0.9   1000000  avgt    5   103945.644 ▒   2026.260  us/op
WriteUnordered.partialSortThenBitmapOf           0.9  10000000  avgt    5  1065638.269 ▒ 102257.059  us/op
WriteUnordered.setupCost                         0.1     10000  avgt    5        6.577 ▒      0.121  us/op
WriteUnordered.setupCost                         0.1    100000  avgt    5       61.378 ▒     24.113  us/op
WriteUnordered.setupCost                         0.1   1000000  avgt    5     1021.588 ▒    536.680  us/op
WriteUnordered.setupCost                         0.1  10000000  avgt    5    13182.341 ▒    196.773  us/op
WriteUnordered.setupCost                         0.5     10000  avgt    5        7.139 ▒      2.216  us/op
WriteUnordered.setupCost                         0.5    100000  avgt    5       60.847 ▒     23.395  us/op
WriteUnordered.setupCost                         0.5   1000000  avgt    5      800.888 ▒     14.711  us/op
WriteUnordered.setupCost                         0.5  10000000  avgt    5    13431.625 ▒    553.440  us/op
WriteUnordered.setupCost                         0.9     10000  avgt    5        6.599 ▒      0.090  us/op
WriteUnordered.setupCost                         0.9    100000  avgt    5       60.946 ▒     22.511  us/op
WriteUnordered.setupCost                         0.9   1000000  avgt    5      813.445 ▒      4.896  us/op
WriteUnordered.setupCost                         0.9  10000000  avgt    5    13374.943 ▒    349.314  us/op
WriteUnordered.sortThenBitmapOf                  0.1     10000  avgt    5      636.230 ▒     13.423  us/op
WriteUnordered.sortThenBitmapOf                  0.1    100000  avgt    5     7411.756 ▒    174.264  us/op
WriteUnordered.sortThenBitmapOf                  0.1   1000000  avgt    5    92299.305 ▒   3651.161  us/op
WriteUnordered.sortThenBitmapOf                  0.1  10000000  avgt    5  1096374.443 ▒ 162575.234  us/op
WriteUnordered.sortThenBitmapOf                  0.5     10000  avgt    5      634.957 ▒     47.447  us/op
WriteUnordered.sortThenBitmapOf                  0.5    100000  avgt    5     7939.074 ▒    409.328  us/op
WriteUnordered.sortThenBitmapOf                  0.5   1000000  avgt    5    93505.427 ▒   5409.749  us/op
WriteUnordered.sortThenBitmapOf                  0.5  10000000  avgt    5  1147933.592 ▒  57485.510  us/op
WriteUnordered.sortThenBitmapOf                  0.9     10000  avgt    5      661.072 ▒      6.717  us/op
WriteUnordered.sortThenBitmapOf                  0.9    100000  avgt    5     7915.506 ▒    356.148  us/op
WriteUnordered.sortThenBitmapOf                  0.9   1000000  avgt    5    93403.343 ▒   5454.583  us/op
WriteUnordered.sortThenBitmapOf                  0.9  10000000  avgt    5  1095960.734 ▒  85753.917  us/op
@coveralls

This comment has been minimized.

Copy link

coveralls commented Dec 18, 2017

Coverage Status

Coverage increased (+0.09%) to 90.53% when pulling 83b49de on richardstartin:speed-up-incremental-writes into b9a2107 on RoaringBitmap:master.

1 similar comment
@coveralls

This comment has been minimized.

Copy link

coveralls commented Dec 18, 2017

Coverage Status

Coverage increased (+0.09%) to 90.53% when pulling 83b49de on richardstartin:speed-up-incremental-writes into b9a2107 on RoaringBitmap:master.

@lemire

This comment has been minimized.

Copy link
Member

lemire commented Dec 19, 2017

Ignore deleted comment

@lemire

lemire approved these changes Dec 19, 2017

Copy link
Member

lemire left a comment

Your code is fine. I suggest a small rewrite that might reduce the amount of code, and thus make maintenance and porting easier. Please consider my suggestion.

*/
public void flush() {
if (dirty) {
int cardinality = computeCardinality();

This comment has been minimized.

@lemire

lemire Dec 19, 2017

Member

Unnecessary (see follow-up comments)

Arrays.fill(bitmap, 0L);
}

private int computeCardinality() {

This comment has been minimized.

@lemire

lemire Dec 19, 2017

Member

This function is not necessary

}
}

private Container chooseBestContainer(int cardinality) {

This comment has been minimized.

@lemire

lemire Dec 19, 2017

Member

You can simplify code as follows:

Container c = new BitmapContainer(bitmap,-1).repairAfterLazy().runOptimize();
if(c instanceof BitmapContainer) {
return c.clone();
} else {
return c;
}

Main benefit is to reduce code redundancy.

(Your code is fine and I'm happy to merge it as-is, but please consider my proposal.)

This comment has been minimized.

@richardstartin

richardstartin Dec 19, 2017

Contributor

I think that's a very helpful suggestion. I made this change.

@coveralls

This comment has been minimized.

Copy link

coveralls commented Dec 19, 2017

Coverage Status

Coverage increased (+0.2%) to 90.605% when pulling c0fd881 on richardstartin:speed-up-incremental-writes into b9a2107 on RoaringBitmap:master.

@richardstartin

This comment has been minimized.

Copy link
Contributor

richardstartin commented Dec 19, 2017

The suggested changes were a small win for performance and a big win for test coverage. I am happy for you to merge this PR, but please squash the commits.

@richardstartin

This comment has been minimized.

Copy link
Contributor

richardstartin commented Dec 20, 2017

Probably worth disabling tiered compilation in the JMH script...

@lemire

This comment has been minimized.

Copy link
Member

lemire commented Dec 21, 2017

Probably worth disabling tiered compilation in the JMH script...

Can you elaborate?

@richardstartin

This comment has been minimized.

Copy link
Contributor

richardstartin commented Dec 22, 2017

I would assume we want to capture steady state behaviour in a server environment. I would suggest disabling tiered compilation to that end. I implemented your suggestion - do we need to do anything else to merge this?

@lemire

This comment has been minimized.

Copy link
Member

lemire commented Dec 22, 2017

Please don't take small delays as an indication of doubts regarding the PR. I'll look at it again and probably merge.

@richardstartin

This comment has been minimized.

Copy link
Contributor

richardstartin commented Dec 22, 2017

That wasn’t what I meant. :)

@lemire

This comment has been minimized.

Copy link
Member

lemire commented Dec 22, 2017

Sure. I think I'll add a bit of documentation after merging, to make sure people are aware of the caveats.

@lemire

This comment has been minimized.

Copy link
Member

lemire commented Dec 23, 2017

I'm holding this PR until we know that the maven config is sane enough for releases, see

#201

@lemire

This comment has been minimized.

Copy link
Member

lemire commented Dec 23, 2017

Finally, issue #201 is fixed.

@lemire lemire merged commit 6125ee9 into RoaringBitmap:master Dec 23, 2017

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.2%) to 90.605%
Details
@lemire

This comment has been minimized.

Copy link
Member

lemire commented Dec 23, 2017

It has been released. I added some quick comments and a code sample.

@richardstartin richardstartin deleted the richardstartin:speed-up-incremental-writes branch Apr 15, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment