-
Notifications
You must be signed in to change notification settings - Fork 2k
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Is your feature request related to a problem or challenge?
@blaginin is trying to speed up execution of tests
Right now, when running the tests by default it takes 8 wallclock seconds with 16 cores.
cargo test --profile=ci --test sqllogictests
...
Running with 16 test threads (available parallelism: 16)
Completed 407 test files in 8 secondsHowever, if you run time, you can see this only consumes 53 coresec out of a theoretical 128 coresec ( 16 cores * 8 seconds = 128 core*sec)
time cargo test --profile=ci --test sqllogictests
Finished `ci` profile [unoptimized] target(s) in 0.34s
Running bin/sqllogictests.rs (target/ci/deps/sqllogictests-91ecf623e0e3b627)
Running with 16 test threads (available parallelism: 16)
Completed 407 test files in 8 seconds
real 0m8.232s
user 0m53.201s
sys 0m1.247sThis means the tests could in theory go twice as fast with more cores
I when looked briefly at this, I think one problem is that the push_down_filter.slt test takes a long time and is single threaded for that time
[00:00:05] ####################-------------------- 52/107 "push_down_filter.slt" - 4 took > 500 ms
Describe the solution you'd like
I would like the sqllogictests to run on all cores
Describe alternatives you've considered
Maybe we can start by splitting up push_down_filter.slt into a few smaller files that can run in parallel
Additional context
No response
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request