Travis and --test-lang option

[nbrady-techempower]

Travis now limits execution to 200 jobs in the matrix. If you go over 200 jobs, travis will not start any of the jobs. Also, travis time limit is 50 mins. So, a potential solution is to group jobs by language. This may not be suitable for all languages as the execution time may be exceeded if there are too many tests. However, I think in a lot of cases this will also speed up runs because of docker caching in a single job. Travis diffing logic would have to change some to avoid full runs, but it will anyway for the travis docker branch.

Total execution time separately for the C languages I've grouped here is just over 60 minutes. Curious to see how long one job would take here.

See #3324

If we do move forward with something like this, i'd like to see a --test-lang C in the toolset as well, and we can do - "TESTLANG=C" in the build jobs for cleaner display.

[mention-bot]

Thanks @nbrady-techempower for contributing to The Framework Benchmarks! @TechEmpower, code you've worked on has been modified. If you have the chance, please review. If you wish to unsubscribe from these notices, please open a Pull Request with the commit message [ci skip] and your github name added to the userBlacklist array in the .mention-bot file.

[mention-bot]

Thanks @nbrady-techempower for contributing to The Framework Benchmarks! @TechEmpower, code you've worked on has been modified. If you have the chance, please review. If you wish to unsubscribe from these notices, please open a Pull Request with the commit message [ci skip] and your github name added to the userBlacklist array in the .mention-bot file.

[RX14]

Isn't the startup time of each travis VM the most time consuming factor here (given that almost all builds are partial)? Surely it'd be far faster to have a single travis job and simply execute the few tests that change in order?

Unfortunately that breaks down when you adjust some core infrastructure and have to rebuild everything... Hmm

[RX14]

Isn't the startup time of each travis VM the most time consuming factor here (given that almost all builds are partial)? Surely it'd be far faster to have a single travis job and simply execute the few tests that change in order?

Unfortunately that breaks down when you adjust some core infrastructure and have to rebuild everything... Hmm

[RX14]

Perhaps the optimal solution is to randomly partition the languages into 4 VMs (given only 4 jobs in the matrix can run in parallel) which gets you max parallelism when everything needs to be redone, but reduces the overhead of (in the common case) starting 26 new VMs just to not use 25 of them.

[RX14]

Perhaps the optimal solution is to randomly partition the languages into 4 VMs (given only 4 jobs in the matrix can run in parallel) which gets you max parallelism when everything needs to be redone, but reduces the overhead of (in the common case) starting 26 new VMs just to not use 25 of them.

[nbrady-techempower]

@RX14 You cant partition a full run into 4 VMs because you have a max execution time of 50 minutes.

[nbrady-techempower]

@RX14 You cant partition a full run into 4 VMs because you have a max execution time of 50 minutes.

[RX14]

@nbrady-techempower oh, that makes sense. Quite a difficult binpacking problem so I guess it's far easier to just be less efficient. This PR is an improvement at least.

[RX14]

@nbrady-techempower oh, that makes sense. Quite a difficult binpacking problem so I guess it's far easier to just be less efficient. This PR is an improvement at least.

[nbrady-techempower]

So this worked out well. I added the --test-lang option and made travis decipher between TESTDIR and TESTLANG environment variables. All of the C tests ran in about half the time. Doing this for a few languages will free up job spaces and generate quicker full runs. Probably best to do the ones that are worked on the least to start, Elixir, Erlang, etc. I'll have to modify the diffing tool as well so that only the framework tests that changed inside TESTLANG run but I think this is 👍

ping @msmith-techempower

[nbrady-techempower]

So this worked out well. I added the --test-lang option and made travis decipher between TESTDIR and TESTLANG environment variables. All of the C tests ran in about half the time. Doing this for a few languages will free up job spaces and generate quicker full runs. Probably best to do the ones that are worked on the least to start, Elixir, Erlang, etc. I'll have to modify the diffing tool as well so that only the framework tests that changed inside TESTLANG run but I think this is 👍

ping @msmith-techempower