Refactoring some IO-related code by AdamGS · Pull Request #846 · vortex-data/vortex

AdamGS · 2024-09-17T12:33:52Z

Add the ability to specify a per-row filter to filter out records using additional information sources. Also some drive-by refactoring work, mostly variable renaming and some comments that I think improve readability by a bit

Some drive-by refactoring from spending time on doing better io pruning/row selection during reads, which will be in a separate PR.

github-actions

Vortex benchmarks

Details

Benchmark suite	Current: `7efc1fa`	Previous: `f14f45a`	Ratio
`tpch_q1/vortex-in-memory-no-pushdown`	`454918649` ns/iter (`± 1946730`)	`461879688` ns/iter (`± 3077475`)	`0.98`
`tpch_q1/vortex-in-memory-pushdown`	`520477139` ns/iter (`± 2410381`)	`523149740` ns/iter (`± 3032183`)	`0.99`
`tpch_q1/arrow`	`441723177` ns/iter (`± 1178388`)	`447032260` ns/iter (`± 3098751`)	`0.99`
`tpch_q1/parquet`	`644798806` ns/iter (`± 1897362`)	`648948151` ns/iter (`± 7956680`)	`0.99`
`tpch_q1/vortex-file-compressed`	`644656129` ns/iter (`± 2132224`)	`610890208` ns/iter (`± 9958849`)	`1.06`
`tpch_q1/vortex-file-uncompressed`	`823963931` ns/iter (`± 3663368`)	`632439600` ns/iter (`± 5723978`)	`1.30`
`tpch_q2/vortex-in-memory-no-pushdown`	`144063802` ns/iter (`± 1093154`)	`148541579` ns/iter (`± 4302659`)	`0.97`
`tpch_q2/vortex-in-memory-pushdown`	`142172262` ns/iter (`± 712663`)	`150887621` ns/iter (`± 4453065`)	`0.94`
`tpch_q2/arrow`	`121973459` ns/iter (`± 530117`)	`124031546` ns/iter (`± 2202920`)	`0.98`
`tpch_q2/parquet`	`154852740` ns/iter (`± 371236`)	`158896173` ns/iter (`± 2073068`)	`0.97`
`tpch_q2/vortex-file-compressed`	`155936096` ns/iter (`± 921084`)	`156750455` ns/iter (`± 1674197`)	`0.99`
`tpch_q2/vortex-file-uncompressed`	`168543902` ns/iter (`± 981951`)	`163293598` ns/iter (`± 1190892`)	`1.03`
`tpch_q3/vortex-in-memory-no-pushdown`	`149937413` ns/iter (`± 366413`)	`149693356` ns/iter (`± 2796232`)	`1.00`
`tpch_q3/vortex-in-memory-pushdown`	`184303826` ns/iter (`± 735835`)	`192024654` ns/iter (`± 7333517`)	`0.96`
`tpch_q3/arrow`	`145253007` ns/iter (`± 1309898`)	`151173886` ns/iter (`± 2769794`)	`0.96`
`tpch_q3/parquet`	`329988287` ns/iter (`± 1073044`)	`342704692` ns/iter (`± 5812958`)	`0.96`
`tpch_q3/vortex-file-compressed`	`336710758` ns/iter (`± 1629073`)	`286885933` ns/iter (`± 2929862`)	`1.17`
`tpch_q3/vortex-file-uncompressed`	`606602812` ns/iter (`± 7168399`)	`356609340` ns/iter (`± 2092025`)	`1.70`
`tpch_q4/vortex-in-memory-no-pushdown`	`105379180` ns/iter (`± 259095`)	`107185765` ns/iter (`± 1077663`)	`0.98`
`tpch_q4/vortex-in-memory-pushdown`	`138974319` ns/iter (`± 1031451`)	`143524344` ns/iter (`± 2500008`)	`0.97`
`tpch_q4/arrow`	`100542419` ns/iter (`± 342164`)	`107655854` ns/iter (`± 1319134`)	`0.93`
`tpch_q4/parquet`	`211677182` ns/iter (`± 584806`)	`222921259` ns/iter (`± 2550700`)	`0.95`
`tpch_q4/vortex-file-compressed`	`276670543` ns/iter (`± 1503640`)	`268313587` ns/iter (`± 4793722`)	`1.03`
`tpch_q4/vortex-file-uncompressed`	`487189326` ns/iter (`± 2532458`)	`314603548` ns/iter (`± 2273681`)	`1.55`
`tpch_q5/vortex-in-memory-no-pushdown`	`304159754` ns/iter (`± 7078557`)	`302192759` ns/iter (`± 4110096`)	`1.01`
`tpch_q5/vortex-in-memory-pushdown`	`304873208` ns/iter (`± 1358737`)	`311208286` ns/iter (`± 3549608`)	`0.98`
`tpch_q5/arrow`	`287618609` ns/iter (`± 2927866`)	`296813796` ns/iter (`± 2263770`)	`0.97`
`tpch_q5/parquet`	`441603632` ns/iter (`± 2876286`)	`448221090` ns/iter (`± 4376687`)	`0.99`
`tpch_q5/vortex-file-compressed`	`339276390` ns/iter (`± 3367293`)	`348447719` ns/iter (`± 2972916`)	`0.97`
`tpch_q5/vortex-file-uncompressed`	`398598253` ns/iter (`± 2317289`)	`349344160` ns/iter (`± 2129832`)	`1.14`
`tpch_q6/vortex-in-memory-no-pushdown`	`39976322` ns/iter (`± 99846`)	`38066285` ns/iter (`± 864563`)	`1.05`
`tpch_q6/vortex-in-memory-pushdown`	`87674245` ns/iter (`± 219160`)	`91880919` ns/iter (`± 383334`)	`0.95`
`tpch_q6/arrow`	`35015203` ns/iter (`± 193255`)	`34761982` ns/iter (`± 481222`)	`1.01`
`tpch_q6/parquet`	`149005222` ns/iter (`± 1913518`)	`148576284` ns/iter (`± 1296662`)	`1.00`
`tpch_q6/vortex-file-compressed`	`118469088` ns/iter (`± 1031929`)	`77163514` ns/iter (`± 769309`)	`1.54`
`tpch_q6/vortex-file-uncompressed`	`368941373` ns/iter (`± 6819565`)	`169717128` ns/iter (`± 5789579`)	`2.17`
`tpch_q7/vortex-in-memory-no-pushdown`	`554653458` ns/iter (`± 3332319`)	`556393001` ns/iter (`± 23698489`)	`1.00`
`tpch_q7/vortex-in-memory-pushdown`	`603984427` ns/iter (`± 4446450`)	`599744491` ns/iter (`± 3301311`)	`1.01`
`tpch_q7/arrow`	`548861325` ns/iter (`± 5089039`)	`539553105` ns/iter (`± 6411940`)	`1.02`
`tpch_q7/parquet`	`701840681` ns/iter (`± 5665045`)	`705749956` ns/iter (`± 11950297`)	`0.99`
`tpch_q7/vortex-file-compressed`	`718648149` ns/iter (`± 4398366`)	`661269547` ns/iter (`± 7820481`)	`1.09`
`tpch_q7/vortex-file-uncompressed`	`985861205` ns/iter (`± 13015037`)	`717390415` ns/iter (`± 3812166`)	`1.37`
`tpch_q8/vortex-in-memory-no-pushdown`	`216273673` ns/iter (`± 429923`)	`214106345` ns/iter (`± 740542`)	`1.01`
`tpch_q8/vortex-in-memory-pushdown`	`228915843` ns/iter (`± 861432`)	`226319703` ns/iter (`± 3484983`)	`1.01`
`tpch_q8/arrow`	`216389883` ns/iter (`± 1285463`)	`214715994` ns/iter (`± 3771551`)	`1.01`
`tpch_q8/parquet`	`479177709` ns/iter (`± 1092940`)	`473554749` ns/iter (`± 2538560`)	`1.01`
`tpch_q8/vortex-file-compressed`	`276799017` ns/iter (`± 1531698`)	`266486592` ns/iter (`± 1857795`)	`1.04`
`tpch_q8/vortex-file-uncompressed`	`367721393` ns/iter (`± 1705759`)	`291502758` ns/iter (`± 5980310`)	`1.26`
`tpch_q9/vortex-in-memory-no-pushdown`	`408446584` ns/iter (`± 3927066`)	`395192675` ns/iter (`± 1943894`)	`1.03`
`tpch_q9/vortex-in-memory-pushdown`	`403930681` ns/iter (`± 2808797`)	`397415770` ns/iter (`± 834027`)	`1.02`
`tpch_q9/arrow`	`415277652` ns/iter (`± 8637537`)	`389569639` ns/iter (`± 7320816`)	`1.07`
`tpch_q9/parquet`	`691381363` ns/iter (`± 2619258`)	`684888505` ns/iter (`± 7376860`)	`1.01`
`tpch_q9/vortex-file-compressed`	`453640057` ns/iter (`± 3021044`)	`438258368` ns/iter (`± 7598713`)	`1.04`
`tpch_q9/vortex-file-uncompressed`	`496319712` ns/iter (`± 6761019`)	`480955306` ns/iter (`± 5038800`)	`1.03`
`tpch_q10/vortex-in-memory-no-pushdown`	`227505684` ns/iter (`± 1232437`)	`226046551` ns/iter (`± 2595417`)	`1.01`
`tpch_q10/vortex-in-memory-pushdown`	`262516460` ns/iter (`± 4339073`)	`261828979` ns/iter (`± 1723691`)	`1.00`
`tpch_q10/arrow`	`219254203` ns/iter (`± 1644737`)	`214356978` ns/iter (`± 549396`)	`1.02`
`tpch_q10/parquet`	`473140290` ns/iter (`± 2610644`)	`471864858` ns/iter (`± 3960785`)	`1.00`
`tpch_q10/vortex-file-compressed`	`461615564` ns/iter (`± 2171143`)	`452272980` ns/iter (`± 5087412`)	`1.02`
`tpch_q10/vortex-file-uncompressed`	`630678555` ns/iter (`± 3963073`)	`389063182` ns/iter (`± 7024937`)	`1.62`
`tpch_q11/vortex-in-memory-no-pushdown`	`222570283` ns/iter (`± 2114864`)	`215392172` ns/iter (`± 5359998`)	`1.03`
`tpch_q11/vortex-in-memory-pushdown`	`219687005` ns/iter (`± 894101`)	`216949528` ns/iter (`± 2550297`)	`1.01`
`tpch_q11/arrow`	`175218987` ns/iter (`± 1233204`)	`176409366` ns/iter (`± 4702977`)	`0.99`
`tpch_q11/parquet`	`184041374` ns/iter (`± 1106798`)	`190107370` ns/iter (`± 6890211`)	`0.97`
`tpch_q11/vortex-file-compressed`	`224289951` ns/iter (`± 922334`)	`224448436` ns/iter (`± 3024412`)	`1.00`
`tpch_q11/vortex-file-uncompressed`	`232739894` ns/iter (`± 1895302`)	`231659525` ns/iter (`± 4099349`)	`1.00`
`tpch_q12/vortex-in-memory-no-pushdown`	`179484092` ns/iter (`± 227665`)	`177740634` ns/iter (`± 1683642`)	`1.01`
`tpch_q12/vortex-in-memory-pushdown`	`260200395` ns/iter (`± 572762`)	`254860784` ns/iter (`± 1161144`)	`1.02`
`tpch_q12/arrow`	`170736553` ns/iter (`± 936909`)	`165273140` ns/iter (`± 961182`)	`1.03`
`tpch_q12/parquet`	`357374044` ns/iter (`± 1605625`)	`352068445` ns/iter (`± 2636228`)	`1.02`
`tpch_q12/vortex-file-compressed`	`637599916` ns/iter (`± 1729981`)	`575933257` ns/iter (`± 5820825`)	`1.11`
`tpch_q12/vortex-file-uncompressed`	`552501434` ns/iter (`± 4057722`)	`351856000` ns/iter (`± 2918588`)	`1.57`
`tpch_q13/vortex-in-memory-no-pushdown`	`176712082` ns/iter (`± 2024955`)	`164279523` ns/iter (`± 6866203`)	`1.08`
`tpch_q13/vortex-in-memory-pushdown`	`173209915` ns/iter (`± 2027480`)	`163140660` ns/iter (`± 14031464`)	`1.06`
`tpch_q13/arrow`	`163150878` ns/iter (`± 4140614`)	`160955641` ns/iter (`± 17729929`)	`1.01`
`tpch_q13/parquet`	`335468448` ns/iter (`± 8536060`)	`305841673` ns/iter (`± 3061812`)	`1.10`
`tpch_q13/vortex-file-compressed`	`209014759` ns/iter (`± 2972733`)	`206909714` ns/iter (`± 1734778`)	`1.01`
`tpch_q13/vortex-file-uncompressed`	`202744933` ns/iter (`± 2924355`)	`214283652` ns/iter (`± 4051306`)	`0.95`
`tpch_q14/vortex-in-memory-no-pushdown`	`39917526` ns/iter (`± 248576`)	`40431818` ns/iter (`± 592849`)	`0.99`
`tpch_q14/vortex-in-memory-pushdown`	`88723704` ns/iter (`± 229320`)	`88516398` ns/iter (`± 892898`)	`1.00`
`tpch_q14/arrow`	`39286924` ns/iter (`± 175070`)	`39784495` ns/iter (`± 434423`)	`0.99`
`tpch_q14/parquet`	`217734687` ns/iter (`± 755624`)	`222673183` ns/iter (`± 1380481`)	`0.98`
`tpch_q14/vortex-file-compressed`	`128294679` ns/iter (`± 1171490`)	`88214842` ns/iter (`± 258223`)	`1.45`
`tpch_q14/vortex-file-uncompressed`	`361683613` ns/iter (`± 3373124`)	`144920148` ns/iter (`± 2027347`)	`2.50`
`tpch_q15/vortex-in-memory-no-pushdown`	`68246483` ns/iter (`± 307996`)	`69925514` ns/iter (`± 379219`)	`0.98`
`tpch_q15/vortex-in-memory-pushdown`	`120039811` ns/iter (`± 268510`)	`124757304` ns/iter (`± 521822`)	`0.96`
`tpch_q15/arrow`	`64238967` ns/iter (`± 426906`)	`66342686` ns/iter (`± 690736`)	`0.97`
`tpch_q15/parquet`	`288112050` ns/iter (`± 727268`)	`289717873` ns/iter (`± 2569075`)	`0.99`
`tpch_q15/vortex-file-compressed`	`227609481` ns/iter (`± 963872`)	`152168506` ns/iter (`± 1408894`)	`1.50`
`tpch_q15/vortex-file-uncompressed`	`697031116` ns/iter (`± 4263979`)	`276373214` ns/iter (`± 11059601`)	`2.52`
`tpch_q16/vortex-in-memory-no-pushdown`	`118386513` ns/iter (`± 320536`)	`119345226` ns/iter (`± 593243`)	`0.99`
`tpch_q16/vortex-in-memory-pushdown`	`122593417` ns/iter (`± 402711`)	`124708820` ns/iter (`± 556574`)	`0.98`
`tpch_q16/arrow`	`104075824` ns/iter (`± 255521`)	`106797434` ns/iter (`± 1502989`)	`0.97`
`tpch_q16/parquet`	`120728617` ns/iter (`± 1232255`)	`121776100` ns/iter (`± 714772`)	`0.99`
`tpch_q16/vortex-file-compressed`	`139363118` ns/iter (`± 443971`)	`137711153` ns/iter (`± 668423`)	`1.01`
`tpch_q16/vortex-file-uncompressed`	`145369116` ns/iter (`± 649778`)	`136383860` ns/iter (`± 508263`)	`1.07`
`tpch_q17/vortex-in-memory-no-pushdown`	`650148004` ns/iter (`± 8971946`)	`664879117` ns/iter (`± 13393600`)	`0.98`
`tpch_q17/vortex-in-memory-pushdown`	`646304647` ns/iter (`± 20536480`)	`661808705` ns/iter (`± 12848360`)	`0.98`
`tpch_q17/arrow`	`552889431` ns/iter (`± 15625605`)	`550112729` ns/iter (`± 8934105`)	`1.01`
`tpch_q17/parquet`	`577692527` ns/iter (`± 1977057`)	`587249827` ns/iter (`± 4112007`)	`0.98`
`tpch_q17/vortex-file-compressed`	`612251282` ns/iter (`± 8296580`)	`639554678` ns/iter (`± 16936248`)	`0.96`
`tpch_q17/vortex-file-uncompressed`	`684350713` ns/iter (`± 6473055`)	`683183788` ns/iter (`± 17491683`)	`1.00`
`tpch_q18/vortex-in-memory-no-pushdown`	`998555291` ns/iter (`± 14059404`)	`1011255791` ns/iter (`± 9664724`)	`0.99`
`tpch_q18/vortex-in-memory-pushdown`	`1053998310` ns/iter (`± 34845054`)	`1016027714` ns/iter (`± 8122498`)	`1.04`
`tpch_q18/arrow`	`1001839057` ns/iter (`± 4652561`)	`1051209569` ns/iter (`± 36763090`)	`0.95`
`tpch_q18/parquet`	`1173961781` ns/iter (`± 5296510`)	`1170968312` ns/iter (`± 14508995`)	`1.00`
`tpch_q18/vortex-file-compressed`	`1041455676` ns/iter (`± 9671343`)	`1161414580` ns/iter (`± 11105024`)	`0.90`
`tpch_q18/vortex-file-uncompressed`	`1108266130` ns/iter (`± 6666258`)	`1205800514` ns/iter (`± 10444648`)	`0.92`
`tpch_q19/vortex-in-memory-no-pushdown`	`165200541` ns/iter (`± 248232`)	`163337569` ns/iter (`± 231117`)	`1.01`
`tpch_q19/vortex-in-memory-pushdown`	`258187629` ns/iter (`± 579174`)	`257806045` ns/iter (`± 3217667`)	`1.00`
`tpch_q19/arrow`	`152122373` ns/iter (`± 388742`)	`149829086` ns/iter (`± 349165`)	`1.02`
`tpch_q19/parquet`	`470586935` ns/iter (`± 557894`)	`465770389` ns/iter (`± 2920116`)	`1.01`
`tpch_q19/vortex-file-compressed`	`669107795` ns/iter (`± 1784833`)	`714911338` ns/iter (`± 3433738`)	`0.94`
`tpch_q19/vortex-file-uncompressed`	`552641486` ns/iter (`± 2351396`)	`364227394` ns/iter (`± 2508336`)	`1.52`
`tpch_q20/vortex-in-memory-no-pushdown`	`259632029` ns/iter (`± 1461504`)	`249184954` ns/iter (`± 1711428`)	`1.04`
`tpch_q20/vortex-in-memory-pushdown`	`284587484` ns/iter (`± 2440931`)	`279440275` ns/iter (`± 1770692`)	`1.02`
`tpch_q20/arrow`	`238796692` ns/iter (`± 1831650`)	`237107149` ns/iter (`± 1116060`)	`1.01`
`tpch_q20/parquet`	`357309273` ns/iter (`± 1567287`)	`355145951` ns/iter (`± 1872562`)	`1.01`
`tpch_q20/vortex-file-compressed`	`352253767` ns/iter (`± 5250921`)	`309752390` ns/iter (`± 1959280`)	`1.14`
`tpch_q20/vortex-file-uncompressed`	`628323971` ns/iter (`± 4350147`)	`400674605` ns/iter (`± 4444505`)	`1.57`
`tpch_q21/vortex-in-memory-no-pushdown`	`838428268` ns/iter (`± 8237048`)	`833772887` ns/iter (`± 2694226`)	`1.01`
`tpch_q21/vortex-in-memory-pushdown`	`901272430` ns/iter (`± 9060372`)	`890630562` ns/iter (`± 3037851`)	`1.01`
`tpch_q21/arrow`	`820433451` ns/iter (`± 4372728`)	`824462975` ns/iter (`± 925823`)	`1.00`
`tpch_q21/parquet`	`963986479` ns/iter (`± 4357565`)	`980620646` ns/iter (`± 12885923`)	`0.98`
`tpch_q21/vortex-file-compressed`	`1185347893` ns/iter (`± 7812960`)	`1186910300` ns/iter (`± 11798799`)	`1.00`
`tpch_q21/vortex-file-uncompressed`	`1652503003` ns/iter (`± 11350552`)	`1323855427` ns/iter (`± 12143244`)	`1.25`
`tpch_q22/vortex-in-memory-no-pushdown`	`102072261` ns/iter (`± 798749`)	`99183298` ns/iter (`± 1007675`)	`1.03`
`tpch_q22/vortex-in-memory-pushdown`	`98889009` ns/iter (`± 505119`)	`98745983` ns/iter (`± 833551`)	`1.00`
`tpch_q22/arrow`	`66039678` ns/iter (`± 213715`)	`66089268` ns/iter (`± 588688`)	`1.00`
`tpch_q22/parquet`	`95039849` ns/iter (`± 586246`)	`101936791` ns/iter (`± 1278450`)	`0.93`
`tpch_q22/vortex-file-compressed`	`106259267` ns/iter (`± 626217`)	`110140769` ns/iter (`± 1661515`)	`0.96`
`tpch_q22/vortex-file-uncompressed`	`119577429` ns/iter (`± 771280`)	`120238517` ns/iter (`± 841608`)	`0.99`

This comment was automatically generated by workflow using github-action-benchmark.

github-actions

Vortex bytes_at

Details

Benchmark suite	Current: `7efc1fa`	Previous: `f14f45a`	Ratio
`bytes_at/array_data`	`610` ns/iter (`± 13`)	`603` ns/iter (`± 1`)	`1.01`
`bytes_at/array_data #2`	`1024` ns/iter (`± 1`)	`1021` ns/iter (`± 18`)	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

github-actions

Vortex random_access

Details

Benchmark suite	Current: `7efc1fa`	Previous: `f14f45a`	Ratio
`vortex/tokio local disk`	`1215845` ns/iter (`± 27580`)	`1204308` ns/iter (`± 40526`)	`1.01`
`vortex/localfs`	`1358665` ns/iter (`± 39688`)	`1363453` ns/iter (`± 28062`)	`1.00`
`parquet/tokio local disk`	`180759838` ns/iter (`± 3767543`)	`178573668` ns/iter (`± 4407348`)	`1.01`

This comment was automatically generated by workflow using github-action-benchmark.

github-actions

Vortex DataFusion

Details

Benchmark suite	Current: `7efc1fa`	Previous: `f14f45a`	Ratio
`arrow/planning`	`793560` ns/iter (`± 7922`)	`801394` ns/iter (`± 12474`)	`0.99`
`arrow/exec`	`1739302` ns/iter (`± 4929`)	`1754329` ns/iter (`± 4485`)	`0.99`
`vortex-pushdown-compressed/planning`	`503436` ns/iter (`± 14057`)	`507344` ns/iter (`± 1752`)	`0.99`
`vortex-pushdown-compressed/exec`	`2990853` ns/iter (`± 6831`)	`3046599` ns/iter (`± 26784`)	`0.98`
`vortex-pushdown-uncompressed/planning`	`501696` ns/iter (`± 8623`)	`506905` ns/iter (`± 2326`)	`0.99`
`vortex-pushdown-uncompressed/exec`	`3486041` ns/iter (`± 70945`)	`2919791` ns/iter (`± 14408`)	`1.19`
`vortex-nopushdown-compressed/planning`	`702283` ns/iter (`± 2299`)	`707699` ns/iter (`± 1898`)	`0.99`
`vortex-nopushdown-compressed/exec`	`13339221` ns/iter (`± 337658`)	`14350777` ns/iter (`± 266929`)	`0.93`
`vortex-nopushdown-uncompressed/planning`	`703032` ns/iter (`± 1867`)	`709099` ns/iter (`± 2558`)	`0.99`
`vortex-nopushdown-uncompressed/exec`	`1976353` ns/iter (`± 8365`)	`1990897` ns/iter (`± 5181`)	`0.99`

This comment was automatically generated by workflow using github-action-benchmark.

github-actions

Vortex Compression

Details

Benchmark suite	Current: `7efc1fa`	Previous: `f14f45a`	Ratio
`Yellow Taxi Trip Data/compress`	`141996248` ns/iter (`± 502660`)	`140163042` ns/iter (`± 801888`)	`1.01`
`Public BI Benchmark/AirlineSentiment`	`511317` ns/iter (`± 208`)	`511415` ns/iter (`± 222`)	`1.00`
`Public BI Benchmark/Arade`	`3721685116` ns/iter (`± 8702678`)	`3761025905` ns/iter (`± 5143244`)	`0.99`
`Public BI Benchmark/Euro2016`	`2875403449` ns/iter (`± 40314595`)	`2860438858` ns/iter (`± 39993075`)	`1.01`
`Public BI Benchmark/Food`	`1351351621` ns/iter (`± 4088634`)	`1369332280` ns/iter (`± 5333366`)	`0.99`
`Public BI Benchmark/HashTags`	`3958558915` ns/iter (`± 14340315`)	`4063545682` ns/iter (`± 24055373`)	`0.97`
`TPCH l_comment Column/compress-default`	`237928268` ns/iter (`± 347908`)	`240363739` ns/iter (`± 692055`)	`0.99`
`TPCH l_comment Column/compress-fsst-chunked`	`1104088055` ns/iter (`± 8173041`)	`1098840233` ns/iter (`± 12999438`)	`1.00`
`TPCH l_comment Column/compress-fsst-canonicalized`	`697336398` ns/iter (`± 7275797`)	`700600384` ns/iter (`± 6058583`)	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

Follows #846, Eval expressions by estimated cost

robert3005

I think there's still few leftovers

robert3005 · 2024-09-23T13:19:19Z

+        let deserializer =
+            LayoutDeserializer::new(self.ctx.clone(), Arc::new(LayoutContext::default()));
+
+        let message_cache = Arc::new(RwLock::new(LayoutMessageCache::default()));
+
+        let mut builder = LayoutReaderBuilder::new(read_at.clone(), deserializer.clone())
+            .with_message_cache(message_cache.clone());

        if let Some(batch_size) = self.batch_size {
            builder = builder.with_batch_size(batch_size);
        }

-        if let Some(predicate) = self
+        let row_filter = self
            .predicate
            .clone()
            .map(convert_expr_to_vortex)
            .transpose()?
-        {
-            builder = builder.with_row_filter(RowFilter::new(predicate));
+            .map(RowFilter::new);
+
+        if let Some(row_filter) = row_filter {
+            builder = builder.with_row_filter(row_filter);


~~All this logic should be in the stream or builder~~ these likely need to be reverted?

Ok, this is because of externalizing the message cache, we should revert that change for now and the diff here disappears

robert3005 · 2024-09-23T13:20:14Z

+    }
+}
+
+pub fn expr_is_filter(expr: &Arc<dyn VortexExpr>) -> bool {


This has a weird name? Column expressions are valid filters

not sure I have a better idea 😅

AdamGS added 3 commits September 16, 2024 17:23

.

af12072

more work

05dd3df

.

0dad209

AdamGS marked this pull request as draft September 17, 2024 12:54

lwwmanning reviewed Sep 17, 2024

View reviewed changes

Comment thread vortex-schema/src/lib.rs Outdated

Comment thread vortex-serde/src/layouts/read/buffered.rs

AdamGS added 2 commits September 17, 2024 14:15

.

73f35de

More code, but it has some bugs/other issues

ac24a52

robert3005 reviewed Sep 17, 2024

View reviewed changes

Comment thread vortex-datafusion/src/persistent/opener.rs Outdated

AdamGS added 6 commits September 17, 2024 19:58

Merge branch 'develop' into adamg/separate-filtering-and-io

9ce8971

.

3201a0f

Merge branch 'develop' into adamg/separate-filtering-and-io

060d202

.

4547429

.

f3e4fe9

.

5110ee4

AdamGS mentioned this pull request Sep 18, 2024

Fix take on sliced RunEnd array #859

Merged

Merge branch 'develop' into adamg/separate-filtering-and-io

aa83c91

AdamGS added the action/benchmark Trigger full benchmarks to run on this PR label Sep 18, 2024

github-actions Bot removed the action/benchmark Trigger full benchmarks to run on this PR label Sep 18, 2024

.

a3b6227

github-actions Bot reviewed Sep 18, 2024

View reviewed changes

AdamGS added 3 commits September 18, 2024 13:11

Merge branch 'develop' into adamg/separate-filtering-and-io

33c1629

Merge branch 'develop' into adamg/separate-filtering-and-io

6235282

Move build_selection to vortex-serde

ef3b149

AdamGS mentioned this pull request Sep 18, 2024

[WIP] Evaluate conjunctions by estimated cost #865

Merged

Merge branch 'develop' into adamg/separate-filtering-and-io

2accb58

AdamGS added the action/benchmark Trigger full benchmarks to run on this PR label Sep 18, 2024

github-actions Bot removed the action/benchmark Trigger full benchmarks to run on this PR label Sep 18, 2024

Merge branch 'develop' into adamg/separate-filtering-and-io

7efc1fa

AdamGS added the action/benchmark Trigger full benchmarks to run on this PR label Sep 19, 2024

github-actions Bot removed the action/benchmark Trigger full benchmarks to run on this PR label Sep 19, 2024

github-actions Bot reviewed Sep 19, 2024

View reviewed changes

AdamGS added 7 commits September 20, 2024 13:09

Merge branch 'develop' into adamg/separate-filtering-and-io

d7b1f1e

[WIP] Evaluate conjunctions by estimated cost (#865)

654610e

Follows #846, Eval expressions by estimated cost

.

3cca890

.

9906ab3

.

2ae93e1

.

b518fdb

.

a3ec71e

AdamGS changed the title ~~Add row selection mechanism to file scans~~ Refactoring some IO-related code Sep 23, 2024

AdamGS marked this pull request as ready for review September 23, 2024 12:35

AdamGS requested review from lwwmanning and robert3005 September 23, 2024 12:36

robert3005 requested changes Sep 23, 2024

View reviewed changes

AdamGS added 2 commits September 23, 2024 14:50

.

33a8356

.

ae09254

robert3005 reviewed Sep 23, 2024

View reviewed changes

Comment thread vortex-serde/Cargo.toml Outdated

robert3005 reviewed Sep 23, 2024

View reviewed changes

Comment thread vortex-serde/Cargo.toml Outdated

.

102a60d

robert3005 approved these changes Sep 23, 2024

View reviewed changes

AdamGS merged commit 17fe851 into develop Sep 23, 2024

AdamGS deleted the adamg/separate-filtering-and-io branch September 23, 2024 14:59

Conversation

AdamGS commented Sep 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Vortex benchmarks

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Vortex bytes_at

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Vortex random_access

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Vortex DataFusion

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Vortex Compression

Uh oh!

robert3005 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

robert3005 Sep 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robert3005 Sep 23, 2024

Choose a reason for hiding this comment

Uh oh!

robert3005 Sep 23, 2024

Choose a reason for hiding this comment

Uh oh!

AdamGS Sep 23, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AdamGS commented Sep 17, 2024 •

edited

Loading

github-actions Bot left a comment •

edited

Loading

robert3005 Sep 23, 2024 •

edited

Loading