New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lighter weight Selenium test setup for PRs. #4925

Merged
merged 2 commits into from Nov 4, 2017

Conversation

Projects
None yet
4 participants
@jmchilton
Member

jmchilton commented Nov 3, 2017

The Dockerfile and image update in #4900 baked chrome, chrome-wrapper, xvfb, and a Selenium server into the default Galaxy testing image. This means we should no longer need to run three docker containers in a compose setup to run Selenium tests. This swaps the script target used by Jenkins to use this newer variant of the tests.

In addition to simply consuming less CPU and booting up much faster thanks to pre-installed dependencies and pre-migrated database, this setup is much easier to cleanup and so we don't need to restrict it to one test per host - these tests I think should run just like API and framework tests. This should also be easier for people running tests locally - this is probably now by far the easiest way to run these tests on Mac or Linux (non-Dockerized test setups are still faster once you get them setup - but it is non-trivial for Mac at least).

I've kept the old compose setup around and I'll setup a Jenkins job against dev that continues to run it periodically (just not on PRs). It does testing of proxy prefix things this variant doesn't and serves as a good reference implementation for multi-container Galaxy testing - which we may wish to do for various categories of tests in the future. The multi-container variant makes it much easier to bring in various services - lots of which one can imagine writing useful Galaxy tests for - AMQP, statistics collection, Docker itself, etc....

@martenson

This comment has been minimized.

Member

martenson commented Nov 3, 2017

I expected this to follow when merging #4900. Awesome!

@jmchilton jmchilton added the status/WIP label Nov 3, 2017

jmchilton added some commits Nov 3, 2017

Lighter weight Selenium test setup for PRs.
The Dockerfile and image update in #4900 baked chrome, chrome-wrapper, xvfb, and a Selenium server into the default Galaxy testing image. This means we should no longer need to run three docker containers in a compose setup to run Selenium tests. This swaps the script target used by Jenkins to use this newer variant of the tests.

In addition to simply consuming less CPU and booting up much faster thanks to pre-installed dependencies and pre-migrated database, this setup is much easier to cleanup and so we don't need to restrict it to one test per host - these tests I think should run just like API and framework tests. This should also be easier for people running tests locally.

I've kept the old compose setup around and I'll setup a Jenkins job against dev that continues to run it periodically (just not on PRs). It does testing of proxy prefix things this variant doesn't and serves as a good reference implementation for multi-container Galaxy testing - which we may wish to do for various categories of tests in the future. The multi-container variant makes it much easier to bring in various services - lots of which one can imagine writing useful Galaxy tests for - AMQP, statistics collection, Docker itself, etc....

@jmchilton jmchilton added status/review and removed status/WIP labels Nov 4, 2017

@jmchilton

This comment has been minimized.

Member

jmchilton commented Nov 4, 2017

Pulling this out of WIP. Two working Jenkins runs (here and here). The runtime for the tests is now closer to 35 minutes instead of 50 - at least when the workers aren't busy.

@galaxybot galaxybot added this to the 18.01 milestone Nov 4, 2017

@dannon

This comment has been minimized.

Member

dannon commented Nov 4, 2017

@jmchilton Tried to run this locally and everything looks like it's working until "Waiting on service postgres", which hangs forever. I'm probably just testing it wrong, but just to check, how should I execute this and other than a running Docker, do I need anything else?

@jmchilton

This comment has been minimized.

Member

jmchilton commented Nov 4, 2017

@dannon I think you should only see that "Waiting on service postgres" message with the older compose based variant of this. How did you run the test?

@dannon

This comment has been minimized.

Member

dannon commented Nov 4, 2017

Tried running both versions; I just pasted the wrong error, sorry!

The non-compose error was something like:

Launching docker container for testing with extra args -e GALAXY_TEST_UID=501 -v /var/folders/gm/53ksmsz51njf80yl3snw6g500000gn/T/tmp.q5or3yks:/tmp -e USE_SELENIUM=1 -e GALAXY_TEST_SELENIUM_RETRIES=1 -e GALAXY_TEST_ERRORS_DIRECTORY=database/test-errors ...
docker: Error response from daemon: Mounts denied:
The path /var/folders/gm/53ksmsz51njf80yl3snw6g500000gn/T/tmp.q5or3yks
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.
.
ERRO[0000] error waiting for container: context canceled

So do I need to set up special mounts for this?

@jmchilton

This comment has been minimized.

Member

jmchilton commented Nov 4, 2017

@dannon I see - yeah you can't mount in your temp directory because it isn't available inside your Docker host's VM. I'm guessing you can't run any of the Docker CI scripts.

You can probably run:

./run_tests.sh --dockerize --db postgres --clean_pyc --selenium

from Galaxy's root though I'd guess... that gets you close and avoids the tmp directory hacking we use on our Jenkins server to reduce the frequency of those "text file is busy" errors.

The .ci script in that folder just runs almost the same command as that but includes this argument --external_tmp which doesn't work out of the box with Docker for Mac I suppose.

If you want to hack your Mac to do that - it is pretty easy though:

screen shot 2017-11-04 at 5 18 43 pm

I've had to mount the /private tmp directory for other reasons also (planemo maybe sets some stuff up in there by default for instance - so if you ever run planemo test --docker it fails without these changes).

I guess I don't really recommend running the script .ci/selenium/run_tests.sh directly except for if you are debugging a Jenkins issue - the script in the GALAXY_HOME is probably the better entry point. That said - we could restrict adding that argument --external_tmp if on Mac OS X or plaster a big warning in the root run_tests.sh if you are on Mac OS X that this may fail.

@dannon

This comment has been minimized.

Member

dannon commented Nov 4, 2017

Ah-ha! That's working perfectly. Thanks for all the info!

(when I ran it before with run_tests.sh, I had inadvetently used the --external_tmp argument, after reading through the .ci scripts, causing my failure)

@dannon dannon merged commit ed0ac48 into galaxyproject:dev Nov 4, 2017

6 checks passed

api test Build finished. 307 tests run, 4 skipped, 0 failed.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
framework test Build finished. 162 tests run, 0 skipped, 0 failed.
Details
integration test Build finished. 57 tests run, 0 skipped, 0 failed.
Details
lgtm analysis: JavaScript No alert changes
Details
toolshed test Build finished. 577 tests run, 0 skipped, 0 failed.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment