Permalink
Browse files

Scale test parallelism on core count

Travis currently only provides two cores to each CI VM. So scale the
test parallelism based on that value rather than hardcoding it. I also
hope that reducing the parallelism will make some of the tests, like
`signal`, less flakey.

Also, ensure we have symbolic debug info by using `-g` on the builds.
  • Loading branch information...
krader1961 committed Dec 6, 2017
1 parent 02b3a65 commit c71c2815b4a3c40f79a5a01f1781ac212ec63108
Showing with 12 additions and 1 deletion.
  1. +9 −1 .travis.yml
  2. +3 −0 meson.build
View
@@ -1,5 +1,13 @@
sudo: required
env:
global:
# Travis currently only provides two cores per VM:
# https://docs.travis-ci.com/user/reference/overview/.
# There is at least one open issue asking that this info be provided to the
# client environment: https://github.com/travis-ci/travis-ci/issues/4696.
# For now just hardcode the expected value.
- CORE_COUNT=2
matrix:
- OS_TYPE=fedora
INSTALL_REQUIREMENTS="dnf repolist; dnf install -y gcc meson sudo langpacks-zh_CN"
@@ -26,7 +34,7 @@ script:
ninja;
echo ==== Running unit tests;
ulimit -n 1024;
export MESON_TESTTHREADS=16;
export MESON_TESTTHREADS=$(( 2 * CORE_COUNT ));

This comment has been minimized.

Show comment
Hide comment
@siteshwar

siteshwar Dec 6, 2017

Collaborator

I had delibrately choosen a rather high value. I checked the Travis build and it takes more than 6 minutes to complete now. Probably just set it to 8 * CORE_COUNT and build time should reduce..

@siteshwar

siteshwar Dec 6, 2017

Collaborator

I had delibrately choosen a rather high value. I checked the Travis build and it takes more than 6 minutes to complete now. Probably just set it to 8 * CORE_COUNT and build time should reduce..

This comment has been minimized.

Show comment
Hide comment
@krader1961

krader1961 Dec 6, 2017

Collaborator

This is basically an experiment to see if reducing test parallelism also reduces the number of unreproducible failures. Ultimately we'll want a really stressful test environment to induce race conditions. But right now I'd like to try and minimize how many failures we see from Travis that are solely due to race conditions. It's really tiresome to have to keep investigating the all too numerous Travis test failures. Obviously I don't have any reason to believe test parallelism is a factor which is why this is an experiment. If a couple of weeks go by with no discernible decrease in test failures we can increase the parallelism in order to decrease the build/test time.

@krader1961

krader1961 Dec 6, 2017

Collaborator

This is basically an experiment to see if reducing test parallelism also reduces the number of unreproducible failures. Ultimately we'll want a really stressful test environment to induce race conditions. But right now I'd like to try and minimize how many failures we see from Travis that are solely due to race conditions. It's really tiresome to have to keep investigating the all too numerous Travis test failures. Obviously I don't have any reason to believe test parallelism is a factor which is why this is an experiment. If a couple of weeks go by with no discernible decrease in test failures we can increase the parallelism in order to decrease the build/test time.

This comment has been minimized.

Show comment
Hide comment
@krader1961

krader1961 Dec 6, 2017

Collaborator

Also, I've noticed that on my OpenSUSE VM running meson test signal never fails. It's only when I run all the tests that I see failures. Which tells me there is a timing issue with the signal tests that is impacted by the parallel running of other tests. So I'm going to also experiment with just telling Meson to not run the signal tests in parallel with any other test.

@krader1961

krader1961 Dec 6, 2017

Collaborator

Also, I've noticed that on my OpenSUSE VM running meson test signal never fails. It's only when I run all the tests that I see failures. Which tells me there is a timing issue with the signal tests that is impacted by the parallel running of other tests. So I'm going to also experiment with just telling Meson to not run the signal tests in parallel with any other test.

This comment has been minimized.

Show comment
Hide comment
@krader1961

krader1961 Dec 6, 2017

Collaborator

It does appear that the signal test cannot be run reliably in parallel with other tests. I modified the Meson config to force that test to be run in isolation and it no longer fails on my system or Travis. TBD is why that is the case. I'll open a Github issue tomorrow to remind us to investigate this issue. In the meantime I'm going to just tell Meson to exclude the signal test from being run in parallel with other tests.

@krader1961

krader1961 Dec 6, 2017

Collaborator

It does appear that the signal test cannot be run reliably in parallel with other tests. I modified the Meson config to force that test to be run in isolation and it no longer fails on my system or Travis. TBD is why that is the case. I'll open a Github issue tomorrow to remind us to investigate this issue. In the meantime I'm going to just tell Meson to exclude the signal test from being run in parallel with other tests.

if ! meson test; then cat meson-logs/testlog.txt; exit 1; fi;
"
View
@@ -8,6 +8,9 @@ add_project_arguments('-DMESON_BUILD=1', language: 'c')
# We should remove all #prototyped pragmas from headers
add_project_arguments('-Wno-unknown-pragmas', language: 'c')
# We sure we have symbolic debug info in the compiled binaries so that our
# `dump_backtrace()` function can provide some minimally useful information.
add_project_arguments('-g', language: 'c')
hosttype_cmd=run_command('bin/package')
hosttype=hosttype_cmd.stdout().strip()

0 comments on commit c71c281

Please sign in to comment.