Run out-of-tree tests in Travis CI #293
Conversation
| build: | ||
| - make V=0 ${make_target} | ||
| builddep: | ||
| - rm -rf /var/cache/dnf/* |
There was a problem hiding this comment.
rm -rf /var/cache/dnf/* && dnf makecache fast are a workaround around deadlocks/synchronization issues observed when you run DNF too fast :). I will open a ticket against DNF for this.
There was a problem hiding this comment.
Computers ... :( I don't want to know how much time got wasted to find the bug.
| cleanup: | ||
| - chown -R ${uid}:${gid} ${container_working_dir} | ||
| configure: | ||
| - autoreconf -i && ./configure |
| - make V=0 lint | ||
| prepare_tests: | ||
| - echo ${server_password} | kinit admin && ipa ping | ||
| - cp -r /etc/ipa/* ~/.ipa/. |
There was a problem hiding this comment.
What's with the extra dot at the end of the line?
| before_install: | ||
| - pip install pep8 | ||
| - pip3 install --user git+https://github.com/martbab/ipa-docker-test-runner |
There was a problem hiding this comment.
In Travis you can safely install to the global site-packages directory. The image gets destroyed after each run anyway.
martbab
force-pushed
the
travis-docker-testrunner
branch
from
7722968
to
8eb8773
Compare
| before_install: | ||
| - pip install pep8 | ||
| - > | ||
| pip3 install | ||
| git+https://github.com/martbab/ipa-docker-test-runner@release-0-2-0 |
There was a problem hiding this comment.
We moved this under FreeIPA organization on github, this link should be updated
martbab
force-pushed
the
travis-docker-testrunner
branch
2 times, most recently
from
d5e7284
to
53610f3
Compare
Prepare a configuration file for https://github.com/martbab/ipa-docker-test-runner. The latest freeipa-fedora-test-runner Docker image (F25 as of time of writing this message) will be used to run tests. Some of them will be purposefuly excluded from the test suite, namely: * test_integration and test_webui: for obvious reasons, CI tests require complicated multi-host setup which is currently not achievable in Travis CI * test_ipapython/test_keyring: Docker can not cope with storing and retrieving secrets from Kernel keyring, that is a known issue * test_xmlrpc/test_dns_plugin.py:test_dns_soa: There are 2-3 non-deterministic failures in this suite in Travis CI, this suite was disabled until the root cause is discovered and fixed/workarounded
https://github.com/martbab/ipa-docker-test-runner is now used to run the following tasks in Travis CI: * pull in a FreeIPA test runner Docker image * configure/make lint/make rpms * install rpms * install FreeIPA server and KRA * run out-of-tree tests For performance reasons (last two steps are very time-consuming) the available tests were split roughly in half and are run as two separate jobs to speed up the process. AD trust is not installed as part of tests since the enabled compat plugin causes false negative errors.
martbab
force-pushed
the
travis-docker-testrunner
branch
from
53610f3
to
e398793
Compare
|
Since I recently run into issues with ipa-server-install and low entropy somewhere around creation of kdb proxy which drastically increased install time, would it make sense to install haveged atop of our rpms to possibly mitigate the problem? |
|
We bind mount (non-blocking) /dev/urandom as /dev/random inside the container so we should actually never get stuck on low entropy in there. |
|
Good. I see the tests pass now and both @tiran's nitpicks and @mbasti-rh's comment have been resolved, so an ACK is in order. |
This PR implements out-of-tree tests in Travis CI using
https://github.com/martbab/ipa-docker-test-runner
There is couple of issues with this PR we may discuss:
Currently ipa-docker-test-runner is pulled and installed directly from git
master branch.
It will probably be better if I release some stable version to PyPI so that
changes in master do not break CI. This would also allow us to cache the
package and its dependencies between builds, saving some time
We have currently two concurrent jobs that run most tests split evenly
between them. This cuts the time from 42 or so minutes to around half an
hour. Any ideas to further speed the tests are welcome.
DNS tests are flaky in Travis CI. Some of them are ignored because they tend
to fail non-deterministically. I will try to determin the root causes and
work around them.