Skip to content

Speedup execution of sqllogictests with more parallelization #20524

@alamb

Description

@alamb

Is your feature request related to a problem or challenge?

@blaginin is trying to speed up execution of tests

Right now, when running the tests by default it takes 8 wallclock seconds with 16 cores.

cargo test --profile=ci  --test sqllogictests
...
Running with 16 test threads (available parallelism: 16)
Completed 407 test files in 8 seconds

However, if you run time, you can see this only consumes 53 coresec out of a theoretical 128 coresec ( 16 cores * 8 seconds = 128 core*sec)

time cargo test --profile=ci  --test sqllogictests
    Finished `ci` profile [unoptimized] target(s) in 0.34s
     Running bin/sqllogictests.rs (target/ci/deps/sqllogictests-91ecf623e0e3b627)
Running with 16 test threads (available parallelism: 16)
Completed 407 test files in 8 seconds
real	0m8.232s
user	0m53.201s
sys	0m1.247s

This means the tests could in theory go twice as fast with more cores

I when looked briefly at this, I think one problem is that the push_down_filter.slt test takes a long time and is single threaded for that time

[00:00:05] ####################--------------------      52/107     "push_down_filter.slt" - 4 took > 500 ms

Describe the solution you'd like

I would like the sqllogictests to run on all cores

Describe alternatives you've considered

Maybe we can start by splitting up push_down_filter.slt into a few smaller files that can run in parallel

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions