Group logically similar tests into the same module in the D2 testsuite

[dlangBugzillaToGithub]

Iain Buclaw (@ibuclaw) reported this on 2018-01-14T11:59:39Z

Transferred from https://issues.dlang.org/show_bug.cgi?id=18235

CC List

• Walter Bright (@WalterBright)
• Dennis
• Seb
• Eddy Schauman-Haigh
• Iain Buclaw (@ibuclaw)

Description

Each new file added to the testsuite may as well be adding 5 seconds onto the overall runtime.  Logical groupings could be done to improve the overall CI speed.

For example, all fail_compilation tests that check coverage of error messages in `dmd/dinterpret.d` could be put into a single module `fail_compilation/fail_dinterpret.d`

[dlangBugzillaToGithub]

greensunny12 commented on 2018-04-05T03:55:30Z

Not saying anything about the logical grouping, but this is a bit exaggerated imho:

> Each new file added to the testsuite may as well be adding 5 seconds onto the overall runtime.  

_All_ 1283 fail_compilation tests compile in ~11s on my machine.
The 774 compilable tests are slightly slower with ~25s ... and you don't want to know how long runnable takes :/
(of course the numbers are taken with full parallelism on an i7 4th Gen)

Anyhow, if someone wants to make the testsuite faster, the runnable tests are the real bad boys and should be focussed on first.

[dlangBugzillaToGithub]

moonlightsentinel commented on 2021-05-20T18:15:50Z

Grouping test files doesn't achieve much in regard to the general test execution time. The real time sink are the permutations for DMD's optimizer/inliner (compare `./run.d` with `./run.d quick`)

[dlangBugzillaToGithub]

hey commented on 2022-01-12T17:25:14Z

Hi! I'm looking to contribute to D and have experience as an SDET so this issue looks perfect. However, from the comments and age of the issue, it's not clear if this is still required. Is it a genuine open issue?

[dlangBugzillaToGithub]

dkorpel commented on 2022-01-12T18:20:13Z

(In reply to Eddy Schauman-Haigh from comment #3)
> However, from the comments and age of the issue, it's
> not clear if this is still required. Is it a genuine open issue?

Yes, it's still an open issue, the test suite has only grown since. Contributions are appreciated! You can find the test suite here:

https://github.com/dlang/dmd/tree/master/test

[dlangBugzillaToGithub]

hey commented on 2022-01-12T18:36:24Z

(In reply to Dennis from comment #4)
> (In reply to Eddy Schauman-Haigh from comment #3)
> > However, from the comments and age of the issue, it's
> > not clear if this is still required. Is it a genuine open issue?
> 
> Yes, it's still an open issue, the test suite has only grown since.
> Contributions are appreciated! You can find the test suite here:
> 
> https://github.com/dlang/dmd/tree/master/test

Great! In which case I'll start having a look at this issue. Thanks for the swift reply Dennis!

[dlangBugzillaToGithub]

ibuclaw (@ibuclaw) commented on 2022-01-13T09:57:54Z

Just be sure to benchmark, benchmark, and benchmark some more!

Things that have changed since the original raising of this include:

- Running the most expensive tests first had reduced the time it took to complete by about a minute when using parallelism.
- Fail compilation tests no longer run with all permutations.
- Neither do compilable tests.