fast coalesce function for data.table #3608

MichaelChirico · 2019-05-27T17:35:39Z

Timings using script https://gist.github.com/MichaelChirico/3ed5276838e9114f89a607594e101ab2

`logical` column:

`integer` column:

`numeric` column:

`character` column:

          type   n_rows na_pct replace_cols  data.table        dplyr      hutils     HUvDT      DPvDT
        <char>    <int>  <num>        <int>       <num>        <num>       <num>     <num>      <num>
  1: character     1000   0.00            1      5932.0      16712.0      2117.5 0.3569622   2.817262
  2: character     1000   0.00            3      6453.0      38549.5      2410.5 0.3735472   5.973888
  3: character     1000   0.00            5      7546.0      61727.0      2622.5 0.3475351   8.180095
  4: character     1000   0.00           20      9922.0     223526.0      3645.0 0.3673655  22.528321
  5: character     1000   0.01            1      6631.5      18163.0     10877.5 1.6402775   2.738898
  6: character     1000   0.01            3      6946.0      40198.5     12109.0 1.7433055   5.787288
  7: character     1000   0.01            5     13097.0     144440.5     27365.5 2.0894480  11.028518
  8: character     1000   0.01           20     10494.0     226325.0     20014.0 1.9071851  21.567086
  9: character     1000   0.05            1      7074.5      19311.5     11946.5 1.6886706   2.729734
 10: character     1000   0.05            3      7513.0      41593.0     16367.0 2.1784906   5.536137
 11: character     1000   0.05            5      8071.0      63826.0     17689.0 2.1916739   7.908066
 12: character     1000   0.05           20     10478.0     221995.5     23040.5 2.1989406  21.186820
 13: character     1000   0.50            1     13909.0      28405.0     20719.0 1.4896110   2.042203
 14: character     1000   0.50            3     17405.5      61720.5     38631.0 2.2194709   3.546034
 15: character     1000   0.50            5     18032.0      86038.5     46836.0 2.5973824   4.771434
 16: character     1000   0.50           20     21599.0     250447.5     69514.0 3.2183897  11.595328
 17:   integer     1000   0.00            1      3961.0      15330.5      1918.0 0.4842212   3.870361
 18:   integer     1000   0.00            3      4366.0      35355.5      2195.0 0.5027485   8.097916
 19:   integer     1000   0.00            5      4534.0      54269.5      2362.0 0.5209528  11.969453
 20:   integer     1000   0.00           20      7112.5     196750.0      3548.0 0.4988401  27.662566
 21:   integer     1000   0.01            1      5092.5      17726.0     10870.5 2.1346097   3.480805
 22:   integer     1000   0.01            3      5139.5      36144.0     11611.0 2.2591692   7.032591
 23:   integer     1000   0.01            5      5284.0      55281.5     12341.5 2.3356359  10.462055
 24:   integer     1000   0.01           20      7366.5     195094.0     17252.0 2.3419534  26.483948
 25:   integer     1000   0.05            1      5016.5      17766.0     11205.5 2.2337287   3.541513
 26:   integer     1000   0.05            3      5659.0      40934.0     15802.5 2.7924545   7.233433
 27:   integer     1000   0.05            5      5875.0      56718.5     16347.5 2.7825532   9.654213
 28:   integer     1000   0.05           20      7831.0     195807.0     21672.5 2.7675265  25.004086
 29:   integer     1000   0.50            1      9283.0      23151.5     16817.0 1.8115911   2.493967
 30:   integer     1000   0.50            3     11928.5      52571.5     31920.0 2.6759442   4.407218
 31:   integer     1000   0.50            5     14188.0      75281.0     41402.5 2.9181350   5.305963
 32:   integer     1000   0.50           20     16639.0     216887.0     62611.5 3.7629365  13.034858
 33:   logical     1000   0.00            1      3989.5      15740.0      2113.5 0.5297656   3.945357
 34:   logical     1000   0.00            3      5026.0      36245.5      2538.5 0.5050736   7.211600
 35:   logical     1000   0.00            5      4562.5      54553.5      2598.0 0.5694247  11.956932
 36:   logical     1000   0.00           20      6868.5     194274.0      3680.0 0.5357793  28.284778
 37:   logical     1000   0.01            1      4711.0      16890.5     10227.0 2.1708767   3.585332
 38:   logical     1000   0.01            3      4941.5      36597.5     11565.0 2.3403825   7.406152
 39:   logical     1000   0.01            5      5575.5      56369.0     12661.0 2.2708277  10.110125
 40:   logical     1000   0.01           20      7701.0     194386.5     17835.0 2.3159330  25.241722
 41:   logical     1000   0.05            1      5046.0      17929.5     11190.0 2.2175981   3.553210
 42:   logical     1000   0.05            3      5215.5      37606.0     15497.0 2.9713354   7.210430
 43:   logical     1000   0.05            5      5679.5      56888.5     16623.0 2.9268422  10.016463
 44:   logical     1000   0.05           20      7913.0     199658.5     25777.5 3.2576141  25.231707
 45:   logical     1000   0.50            1      8753.5      23620.0     16947.0 1.9360256   2.698349
 46:   logical     1000   0.50            3     12653.5      58039.5     34903.0 2.7583673   4.586834
 47:   logical     1000   0.50            5     13011.5      75396.5     42022.5 3.2296430   5.794605
 48:   logical     1000   0.50           20     15925.0     217161.0     59149.5 3.7142543  13.636484
 49:   numeric     1000   0.00            1      7922.0      18521.0      3598.0 0.4541782   2.337920
 50:   numeric     1000   0.00            3      8337.5      45128.0      3794.5 0.4551124   5.412654
 51:   numeric     1000   0.00            5      8605.5      68236.0      3979.5 0.4624368   7.929348
 52:   numeric     1000   0.00           20     10797.0     244202.0      5116.0 0.4738353  22.617579
 53:   numeric     1000   0.01            1      8564.0      19545.5     12476.0 1.4567959   2.282286
 54:   numeric     1000   0.01            3      8975.0      45513.5     15692.5 1.7484680   5.071142
 55:   numeric     1000   0.01            5      9243.5      69114.5     21103.0 2.2830097   7.477092
 56:   numeric     1000   0.01           20     11630.0     244776.5     27416.0 2.3573517  21.046991
 57:   numeric     1000   0.05            1      9570.0      22267.5     14210.0 1.4848485   2.326803
 58:   numeric     1000   0.05            3      9748.5      47189.0     21691.0 2.2250603   4.840642
 59:   numeric     1000   0.05            5      9591.5      70494.0     22208.5 2.3154355   7.349632
 60:   numeric     1000   0.05           20     12334.5     248807.5     26852.5 2.1770238  20.171673
 61:   numeric     1000   0.50            1     16037.5      27467.5     20003.5 1.2472954   1.712705
 62:   numeric     1000   0.50            3     20966.5      62252.5     38052.5 1.8149190   2.969141
 63:   numeric     1000   0.50            5     22849.5      89607.5     50831.0 2.2246001   3.921639
 64:   numeric     1000   0.50           20     25423.0     267560.0     81957.5 3.2237541  10.524328
 65: character   100000   0.00            1    320522.0     714551.0     47552.0 0.1483580   2.229335
 66: character   100000   0.00            3    322401.5    1934283.5     48707.0 0.1510756   5.999611
 67: character   100000   0.00            5    323474.5    5589912.0     49790.5 0.1539240  17.280843
 68: character   100000   0.00           20    332092.5   24019690.0     50607.0 0.1523883  72.328312
 69: character   100000   0.01            1    337333.0     758926.5    315037.5 0.9339066   2.249784
 70: character   100000   0.01            3    368799.0    2011977.0    527635.0 1.4306845   5.455484
 71: character   100000   0.01            5    349709.5    5705843.0    522898.0 1.4952353  16.315951
 72: character   100000   0.01           20    351693.0   22880015.5    510467.5 1.4514577  65.056784
 73: character   100000   0.05            1    394094.0     857495.0    438587.5 1.1129007   2.175864
 74: character   100000   0.05            3    439923.0    2109477.5    762484.0 1.7332215   4.795106
 75: character   100000   0.05            5    462533.5    5750846.0    979328.0 2.1173126  12.433361
 76: character   100000   0.05           20    425652.5   25324855.0    823296.5 1.9341987  59.496549
 77: character   100000   0.50            1   1100659.0    1904290.5   1431910.5 1.3009574   1.730137
 78: character   100000   0.50            3   1500699.0    5582363.0   3008770.5 2.0049127   3.719842
 79: character   100000   0.50            5   1538074.0    7992246.5   3341107.5 2.1722671   5.196269
 80: character   100000   0.50           20   1811595.0   26799062.0   7106634.5 3.9228605  14.793076
 81:   integer   100000   0.00            1     89396.5     473667.0     29229.0 0.3269591   5.298496
 82:   integer   100000   0.00            3     91308.5    1302190.5     30707.5 0.3363049  14.261438
 83:   integer   100000   0.00            5     90038.0    2127184.5     30743.0 0.3414447  23.625408
 84:   integer   100000   0.00           20     93391.0   14036654.5     32679.0 0.3499159 150.299863
 85:   integer   100000   0.01            1    104375.0     449351.5    210376.0 2.0155784   4.305164
 86:   integer   100000   0.01            3    114833.5    1285874.5    377378.5 3.2863102  11.197730
 87:   integer   100000   0.01            5    114987.5    2147093.5    385747.5 3.3546907  18.672408
 88:   integer   100000   0.01           20    122800.5   13960827.0    529042.0 4.3081421 113.687053
 89:   integer   100000   0.05            1    171648.0     550623.0    306407.0 1.7850893   3.207861
 90:   integer   100000   0.05            3    180163.0    1367275.0    560074.0 3.1087071   7.589100
 91:   integer   100000   0.05            5    180593.5    2250939.5    767200.0 4.2482149  12.464122
 92:   integer   100000   0.05           20    193265.0   14211810.5    791227.5 4.0940031  73.535356
 93:   integer   100000   0.50            1    845426.0    1248660.0    989707.0 1.1706607   1.476960
 94:   integer   100000   0.50            3   1112980.5    2841024.5   2047275.5 1.8394532   2.552627
 95:   integer   100000   0.50            5   1218435.5    4176352.0   2648774.0 2.1739140   3.427635
 96:   integer   100000   0.50           20   1365508.0   16094775.5   6204160.5 4.5434816  11.786658
 97:   logical   100000   0.00            1     79330.0     418037.0     50262.0 0.6335812   5.269595
 98:   logical   100000   0.00            3     90325.0    1229678.0     51703.5 0.5724163  13.613927
 99:   logical   100000   0.00            5     80701.5    2116145.0     52109.5 0.6457067  26.221879
100:   logical   100000   0.00           20    102466.0   14026416.5     53629.5 0.5233882 136.888495
101:   logical   100000   0.01            1    103895.5     448980.0    204184.0 1.9652824   4.321458
102:   logical   100000   0.01            3    115204.5    1287910.0    389872.0 3.3841734  11.179338
103:   logical   100000   0.01            5    124491.5    2159182.0    554072.5 4.4506854  17.344011
104:   logical   100000   0.01           20    115648.0   14049663.0    412939.5 3.5706584 121.486433
105:   logical   100000   0.05            1    166411.0     547439.0    301020.0 1.8088948   3.289680
106:   logical   100000   0.05            3    182022.0    1367664.5    588468.5 3.2329526   7.513732
107:   logical   100000   0.05            5    181474.0    2567820.0    783859.5 4.3194039  14.149796
108:   logical   100000   0.05           20    177541.5   14068579.5    656816.5 3.6995097  79.241076
109:   logical   100000   0.50            1    836043.5    1237601.5    983486.5 1.1763580   1.480308
110:   logical   100000   0.50            3   1114749.5    2792142.0   2035300.5 1.8257918   2.504726
111:   logical   100000   0.50            5   1180586.5    4048974.5   2571835.0 2.1784384   3.429630
112:   logical   100000   0.50           20   1341120.5   15956940.5   6275856.5 4.6795620  11.898215
113:   numeric   100000   0.00            1    496539.0     767671.0    187408.0 0.3774286   1.546044
114:   numeric   100000   0.00            3    490569.0    2162623.5    185539.5 0.3782129   4.408398
115:   numeric   100000   0.00            5    492463.0    5402569.5    185691.0 0.3770659  10.970508
116:   numeric   100000   0.00           20    492638.5   23545536.0    186949.0 0.3794852  47.794754
117:   numeric   100000   0.01            1    503604.5     785947.0    399915.5 0.7941063   1.560643
118:   numeric   100000   0.01            3    518728.0    2289587.0    881052.5 1.6984865   4.413849
119:   numeric   100000   0.01            5    528399.5    5510336.0    879884.0 1.6651870  10.428352
120:   numeric   100000   0.01           20    511831.5   20982373.5    940907.0 1.8383140  40.994690
121:   numeric   100000   0.05            1    569052.0     883269.5    517215.0 0.9089064   1.552177
122:   numeric   100000   0.05            3    598659.0    2313521.5   1065702.5 1.7801495   3.864506
123:   numeric   100000   0.05            5    590520.5    5540227.5   1287339.0 2.1800073   9.381939
124:   numeric   100000   0.05           20    577252.0   20189578.5   1281808.5 2.2205354  34.975329
125:   numeric   100000   0.50            1   1291828.0    1594985.5   1232879.0 0.9543678   1.234673
126:   numeric   100000   0.50            3   1693015.0    3876348.5   2698185.5 1.5937162   2.289613
127:   numeric   100000   0.50            5   1820733.0    7428112.5   3514920.0 1.9304972   4.079737
128:   numeric   100000   0.50           20   2152732.5   22562634.5   9482398.5 4.4048197  10.480928
129: character 10000000   0.00            1  50800038.0  235855519.5   6073162.5 0.1195504   4.642822
130: character 10000000   0.00            3  42739595.0  473103680.5   6082649.5 0.1423188  11.069447
131: character 10000000   0.00            5  40507782.5  750471216.5   6066795.0 0.1497686  18.526593
132: character 10000000   0.00           20  40414074.5 2538540805.5   6070666.5 0.1502117  62.813286
133: character 10000000   0.01            1  43982298.0  106583557.0  44527168.0 1.0123884   2.423329
134: character 10000000   0.01            3  44077685.0  336523925.5  88758685.5 2.0136875   7.634791
135: character 10000000   0.01            5  44116497.5  723328713.0 106152350.0 2.4061826  16.395878
136: character 10000000   0.01           20  43985113.5 2511135036.0  85126024.5 1.9353372  57.090566
137: character 10000000   0.05            1  55633025.0  127647348.0  64390272.0 1.1574109   2.294453
138: character 10000000   0.05            3  57144435.0  393951547.5 111554972.0 1.9521581   6.893962
139: character 10000000   0.05            5  56993719.5  770233548.0 153793295.0 2.6984253  13.514358
140: character 10000000   0.05           20  57426462.0 2559252977.0 152444366.5 2.6546014  44.565744
141: character 10000000   0.50            1 116073466.0  217970168.0 151695858.5 1.3068952   1.877864
142: character 10000000   0.50            3 149168356.0  584027885.5 318103120.5 2.1325107   3.915226
143: character 10000000   0.50            5 169805328.0 1012960775.5 412543409.0 2.4295080   5.965424
144: character 10000000   0.50           20 183955782.0 2813413465.5 697093088.5 3.7894601  15.293966
145:   integer 10000000   0.00            1   9511513.5   52565310.0   3279316.0 0.3447733   5.526493
146:   integer 10000000   0.00            3   9527060.5  150688935.0   3215855.0 0.3375496  15.816939
147:   integer 10000000   0.00            5   9639395.5  311991882.5   3287509.0 0.3410493  32.366333
148:   integer 10000000   0.00           20   9482700.5 1030586684.0   3235361.0 0.3411856 108.680716
149:   integer 10000000   0.01            1  13395790.0   57129621.5  25887761.5 1.9325297   4.264744
150:   integer 10000000   0.01            3  13425587.0  208196713.0  61003367.5 4.5438138  15.507457
151:   integer 10000000   0.01            5  13368104.5  311750926.0  82853192.5 6.1978265  23.320503
152:   integer 10000000   0.01           20  13309867.5 1032154137.0  62017681.0 4.6595266  77.548040
153:   integer 10000000   0.05            1  21929097.0   69565866.5  38374451.0 1.7499330   3.172309
154:   integer 10000000   0.05            3  22737534.5  177697854.0  72673806.0 3.1962043   7.815177
155:   integer 10000000   0.05            5  22810900.5  322621839.0 106761972.5 4.6803050  14.143319
156:   integer 10000000   0.05           20  22830768.0 1045807494.0 123240686.5 5.3980088  45.806934
157:   integer 10000000   0.50            1  82479991.0  142359624.5 110174836.0 1.3357765   1.725990
158:   integer 10000000   0.50            3 109728015.0  383278615.0 235464137.0 2.1458890   3.492988
159:   integer 10000000   0.50            5 118512187.5  514789955.0 299621565.0 2.5281920   4.343772
160:   integer 10000000   0.50           20 129016588.5 1263367817.5 557652124.5 4.3223289   9.792290
161:   logical 10000000   0.00            1   9521381.5   52835840.5   4954387.0 0.5203433   5.549178
162:   logical 10000000   0.00            3   9579305.5  200576654.5   4954705.0 0.5172301  20.938538
163:   logical 10000000   0.00            5   9478532.0  311413325.0   4956443.5 0.5229126  32.854594
164:   logical 10000000   0.00           20   9511628.0 1031928316.0   4992276.5 0.5248604 108.491240
165:   logical 10000000   0.01            1  13413096.0   57231391.5  25954191.5 1.9349889   4.266829
166:   logical 10000000   0.01            3  13502655.0  209770906.0  59309867.5 4.3924597  15.535530
167:   logical 10000000   0.01            5  13372155.5  313868499.0  63872855.5 4.7765564  23.471795
168:   logical 10000000   0.01           20  13283216.0 1032157537.0  65324045.5 4.9177884  77.703889
169:   logical 10000000   0.05            1  21893350.5   69580969.5  38284088.5 1.7486628   3.178178
170:   logical 10000000   0.05            3  22898160.5  206116919.5  72815191.0 3.1799581   9.001462
171:   logical 10000000   0.05            5  22841569.5  325706615.5 110407783.0 4.8336338  14.259380
172:   logical 10000000   0.05           20  22637612.0 1043248492.5 130880424.0 5.7815473  46.084741
173:   logical 10000000   0.50            1  82528924.5  141855136.0 109828906.5 1.3307929   1.718854
174:   logical 10000000   0.50            3 111818598.5  389407463.5 237397706.0 2.1230610   3.482493
175:   logical 10000000   0.50            5 119519530.5  540749684.0 299332996.0 2.5044693   4.524363
176:   logical 10000000   0.50           20 129570426.5 1543077084.5 555320482.5 4.2858583  11.909177
177:   numeric 10000000   0.00            1  51246623.5   85740256.5  18457973.5 0.3601793   1.673091
178:   numeric 10000000   0.00            3  51332612.5  285752817.0  18469147.5 0.3597936   5.566691
179:   numeric 10000000   0.00            5  52114717.5  432952536.5  18486931.0 0.3547353   8.307683
180:   numeric 10000000   0.00           20  51113278.0 1551106762.5  18431514.5 0.3606013  30.346454
181:   numeric 10000000   0.01            1  54733192.0   89558797.0  46360216.5 0.8470220   1.636279
182:   numeric 10000000   0.01            3  55246466.0  243834866.0 109239864.5 1.9773186   4.413583
183:   numeric 10000000   0.01            5  55438720.5  434968311.0 123057446.0 2.2197021   7.845930
184:   numeric 10000000   0.01           20  54765682.5 1555582339.0 163964243.0 2.9939231  28.404327
185:   numeric 10000000   0.05            1  66132224.0  103792421.0  60734331.0 0.9183773   1.569468
186:   numeric 10000000   0.05            3  67652848.0  260506804.5 120259673.0 1.7775996   3.850641
187:   numeric 10000000   0.05            5  67813006.5  448409198.0 184746005.5 2.7243447   6.612436
188:   numeric 10000000   0.05           20  67634383.0 1574740853.0 204058659.0 3.0170847  23.283141
189:   numeric 10000000   0.50            1 133198420.5  184348790.0 140353111.5 1.0537145   1.384016
190:   numeric 10000000   0.50            3 172706137.0  441002245.5 303470400.0 1.7571489   2.553483
191:   numeric 10000000   0.50            5 200942574.0  678868465.0 396554312.5 1.9734709   3.378420
192:   numeric 10000000   0.50           20 217046584.0 2019039291.0 873049025.5 4.0224039   9.302332
          type   n_rows na_pct replace_cols  data.table        dplyr      hutils     HUvDT      DPvDT

codecov · 2019-05-27T17:57:38Z

Codecov Report

Merging #3608 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #3608      +/-   ##
==========================================
+ Coverage   98.24%   98.25%   +0.01%     
==========================================
  Files          67       69       +2     
  Lines       12972    13088     +116     
==========================================
+ Hits        12744    12860     +116     
  Misses        228      228

Impacted Files	Coverage Δ
src/init.c	`100% <ø> (ø)`	⬆️
R/coalesce.R	`100% <100%> (ø)`
src/coalesce.c	`100% <100%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c9c8d09...c228c7e. Read the comment docs.

jangorecki

some initial comments.
add some Rraw with examples, or put them in comments next to R code, so it is easier to reproduce.
we can think what we could parallelize here... we could eventually run all columns passed to ... in parallel

R/coalesce.R

src/coalesce.c

jangorecki · 2019-05-28T04:35:21Z

I will push some helpers to new branch, which are likely to be useful here.

jangorecki · 2019-05-28T05:35:55Z

#3609
I pushed working example of parallel anyNA, which

for atomic vector parallelize over rows

cc(F)
x = sample(1e8)
any_na_integer(x)

and for list input parallelize over columns

cc(F)
x = list(sample(1e8), c(sample(1e8-1), NA), sample(1e8))
list_any_na_integer(x)

anyNA would be best to incorporate into core loop in coalesce then code from my branch won't apply here. Otherwise it should be possible to re-use it maybe tweaking for different data types, except character.

update: #3608 (comment)

src/coalesce.c

jangorecki

some more detailed feedback

src/coalesce.c

jangorecki · 2019-05-29T08:44:20Z

I reduced number of passes over x to one, in most pessimistic case, when all values are NA, then we do 2 passes. First pass produces which(is.na(x)), then second pass iterates only on NA indexes.
No parallelism, no long vec support (difficult to write which for that). Added option for in-place coalesce. Waiting for new benchmark results.
@MichaelChirico

table presents median time, could we also get first run timing? it is useful to see if there is any significant difference on that
also could we format numbers to relative? It is difficult to compare visually 34612729.5 vs 319625360.0 vs 49262548.0. Now it is not even clear what those numbers are, milliseconds?
if possible add unexported setcoalesce to comparison
1e6 input is pretty small, could you add 1e7 at least? or ideally also 1e8 (probably not for 50 columns). No need to run 100 times then

src/coalesce.c

jangorecki · 2019-06-03T16:48:53Z

~~Date class left to do~~

jangorecki · 2019-06-04T10:10:57Z

~~tests for POSIXct left~~, nanotime and integer64 to review after eddelbuettel/nanotime#46

man/coalesce.Rd

HughParsonage · 2019-06-04T15:41:55Z

Should long vectors be supported?

jangorecki · 2019-06-05T04:01:00Z

@HughParsonage in the long run they should but I was not able to code which_eq functions to support that. It will also impose relatively significant overhead. We allocate memory for integer vector of the same length as x, just in case if x is all NAs. So now we allocate length(x)*sizeof(int), after switch to long vector we will need to allocate twice as much memory length(x)*sizeof(int64_t).
As long as it is not being requested by users I wouldn't bother.

jangorecki · 2019-06-24T08:22:22Z

some benchmarking of Matt's approach (strictly single pass handling everything) vs single pass which-only followed by a pass over which indices: https://gist.github.com/jangorecki/e065890e5ddd97c49670c33effcec688
generally Matt's solution faster, up to 100% faster, in some rare cases might be 20% slower. It will additionally consume less memory, because there is no need to allocate integer vector for which results.
One more thing to keep in mind that coalesce by many groups might eventually have big overhead, due to parallelism initialised for every group. This is IMO not the common use case, and can be easily avoided by separating call into two, something like DT[, replacement:=min(val), by=grp][, new_val:=coalesce(val, replacement)].

initial work on data.table::coalesce

4b1249d

jangorecki reviewed May 28, 2019

View reviewed changes

jangorecki added the WIP label May 28, 2019

basic working version done; to be benchmarked & iterated

a67a399

MichaelChirico changed the title ~~WIP: fast coalesce function for data.table~~ WIP: #3424 fast coalesce function for data.table May 29, 2019

jangorecki mentioned this pull request May 29, 2019

parallel any_na #3609

Closed

jangorecki reviewed May 29, 2019

View reviewed changes

src/coalesce.c Outdated Show resolved Hide resolved

jangorecki reviewed May 29, 2019

View reviewed changes

src/coalesce.c Outdated Show resolved Hide resolved

jangorecki reviewed May 29, 2019

View reviewed changes

src/coalesce.c Outdated Show resolved Hide resolved

jangorecki reviewed May 29, 2019

View reviewed changes

src/coalesce.c Outdated Show resolved Hide resolved

jangorecki reviewed May 29, 2019

View reviewed changes

src/coalesce.c Outdated Show resolved Hide resolved

jangorecki reviewed May 29, 2019

View reviewed changes

src/coalesce.c Outdated Show resolved Hide resolved

jangorecki reviewed May 29, 2019

View reviewed changes

jangorecki added 2 commits May 29, 2019 11:30

resolving mmy feedback in PR

7cd9758

proper pragma omp def

9c02162

MichaelChirico commented May 29, 2019

View reviewed changes

src/coalesce.c Outdated Show resolved Hide resolved

rewrite coalesce for less passes

890cd1a

jangorecki changed the title ~~WIP: #3424 fast coalesce function for data.table~~ #3424 fast coalesce function for data.table May 29, 2019

jangorecki added 3 commits May 29, 2019 17:22

int64 support, precaalc value lens and re-use

e04069d

add alt interface for dots to coalesce, as requested in #3424

0415de5

remove extra allocation, add verbose

eb1d9d1

jangorecki changed the title ~~#3424 fast coalesce function for data.table~~ fast coalesce function for data.table May 29, 2019

simplify which_char

475446a

jangorecki reviewed May 30, 2019

View reviewed changes

src/coalesce.c Outdated Show resolved Hide resolved

jangorecki added 2 commits May 30, 2019 12:23

reorg code, which_eq double NaN handling

7d09730

ncoalesce not yet exported

6f00521

jangorecki added 3 commits June 3, 2019 10:54

early escape for no input

55c6be2

setcoalesce and which_eq test coverage

5e9e85c

resolve pending unit tests related to classes and factor levels

7116086

check classMiss only for x class, add nanotime tests

607c469

jangorecki added 2 commits June 4, 2019 18:01

update manual

69cd482

test coverage and posixct tests

a7054f7

HughParsonage reviewed Jun 4, 2019

View reviewed changes

man/coalesce.Rd Outdated Show resolved Hide resolved

jangorecki added 3 commits June 5, 2019 10:22

handle nanotime class in coalesce, explained in eddelbuettel/nanotime#48

1499060

Hugh minor feedback to manual

a34407b

which_eq nanotime support

b926584

jangorecki removed the WIP label Jun 6, 2019

jangorecki approved these changes Jun 16, 2019

View reviewed changes

mattdowle added this to the 1.12.4 milestone Jun 17, 2019

mattdowle added 7 commits June 17, 2019 15:14

Merge branch 'master' into coalesce

86eac0d

major simplification (rewrite) and parallelized. (Was +641 -2 lines)

9c386ef

singletons by using final; passing tests

4aff5cd

fixed missing error() argument, failed just on windows

c98ce94

actually fixed this time

ffaa3b4

news item benchmark and removed .dots

0acc636

coverage

c228c7e

Rdatatable deleted a comment from codecov bot Jun 18, 2019

mattdowle merged commit 024399a into master Jun 18, 2019

mattdowle deleted the coalesce branch June 18, 2019 10:01

jangorecki restored the coalesce branch June 19, 2019 13:20

jangorecki deleted the coalesce branch June 19, 2019 14:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fast coalesce function for data.table #3608

fast coalesce function for data.table #3608

MichaelChirico commented May 27, 2019 •

edited by mattdowle

Loading

codecov bot commented May 27, 2019 •

edited

Loading

jangorecki left a comment

jangorecki commented May 28, 2019

jangorecki commented May 28, 2019 •

edited

Loading

jangorecki left a comment

jangorecki commented May 29, 2019 •

edited

Loading

jangorecki commented Jun 3, 2019 •

edited

Loading

jangorecki commented Jun 4, 2019 •

edited

Loading

HughParsonage commented Jun 4, 2019

jangorecki commented Jun 5, 2019

jangorecki commented Jun 24, 2019 •

edited

Loading

fast coalesce function for data.table #3608

fast coalesce function for data.table #3608

Conversation

MichaelChirico commented May 27, 2019 • edited by mattdowle Loading

logical column:

integer column:

numeric column:

character column:

codecov bot commented May 27, 2019 • edited Loading

Codecov Report

jangorecki left a comment

Choose a reason for hiding this comment

jangorecki commented May 28, 2019

jangorecki commented May 28, 2019 • edited Loading

jangorecki left a comment

Choose a reason for hiding this comment

jangorecki commented May 29, 2019 • edited Loading

jangorecki commented Jun 3, 2019 • edited Loading

jangorecki commented Jun 4, 2019 • edited Loading

HughParsonage commented Jun 4, 2019

jangorecki commented Jun 5, 2019

jangorecki commented Jun 24, 2019 • edited Loading

MichaelChirico commented May 27, 2019 •

edited by mattdowle

Loading

`logical` column:

`integer` column:

`numeric` column:

`character` column:

codecov bot commented May 27, 2019 •

edited

Loading

jangorecki commented May 28, 2019 •

edited

Loading

jangorecki commented May 29, 2019 •

edited

Loading

jangorecki commented Jun 3, 2019 •

edited

Loading

jangorecki commented Jun 4, 2019 •

edited

Loading

jangorecki commented Jun 24, 2019 •

edited

Loading