Enable expensive tests

[t-b]

It also includes a backport of a fix for t0027 from junio/master.
All tests pass.

[t-b]

@sschuberth I added some info into the commit message. I did not feel any heavier than without expensive tests cpu usage.

@dscho My intentition is to increase test coverage. I also don't like the concept of expensive tests, as everything which deviates from default setup is likely not tested. This is also the reason why I propose to change that in the source code level, rather on the test invocation level. With the expensive tests enabled, also every contributor will execute them, if he is doing that locally.

[dscho]

With the expensive tests enabled, also every contributor will execute them, if he is doing that locally.

We really have to be pragmatic here. If running tests takes an unbearable time, nobody will test them locally. So I would vote to let Jenkins do the heavy tests and let the default be at least somewhat bearable.

[sschuberth]

I tend to agree with Dscho. Also, a jump from 18 to 24 minutes test time (on your high end machine) is quite big. I'd expect the VM to take much longer, and it's already taking too long for my taste. Finally, it seems we'd be the only platform enabling expensive tests by default, so that's probably not a good idea.

[t-b]

I just checked all expensive tests separately.
Only t3302-notes-index-expensive.sh is really expensive with 6 minutes(!!). The other expensive tests finish within 50 seconds.

From the expensive tests I care mostly about t0027 because:

• it has a huge MINGW only section
• EOL fun is especially common on MINGW

t0027 takes here only 40 seconds.

So how about instead of enabling the EXPENSIVE tests, we remove the EXPENSIVE flag from t0027?
The test suite will not take noticibly longer to execute, but we have some important tests more?

[dscho]

I just checked all expensive tests separately.
Only t3302-notes-index-expensive.sh is really expensive with 6 minutes(!!). The other expensive tests finish within 50 seconds.

You could make an argument that some of the tests marked "expensive" are not expensive. That still does not change the fact that we should not force the expensive tests down the throat of contributors. Because if we tried that stunt, we'd only make sure that no contributor would run the tests at all...

So how about simply adding a Jenkins job that runs the expensive tests once a day, risk that regressions in those expensive tests are identified only with an up-to-24h delay, have happier contributors and quicker turn-arounds from BuildHive/Jenkins and be done with this issue?

[t-b]

So how about simply adding a Jenkins job that runs the expensive tests once a day, risk that regressions in those expensive tests are identified only with an up-to-24h delay, have happier contributors and quicker turn-arounds from BuildHive/Jenkins and be done with this issue?

Fine with me. But then we also should cherry-pick t-b@4f355f1.

[sschuberth]

So how about instead of enabling the EXPENSIVE tests, we remove the EXPENSIVE flag from t0027?

I like that idea. If the test is not (very) expensive, there is no need to flag it as such, and I'd like to run these MinGW-specific test by default.

[dscho]

How about dscho@t0027-mingw?

[dscho]

we also should cherry-pick t-b/git@4f355f1.

Agreed!

[sschuberth]

Dscho, when you push dscho@t0027-mingw, could you also take care or cherry-picking t-b@4f355f1? Thanks in advance! I'll close this PR then.

[dscho]

See #7.