New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatic-er database initialization. #4887

Merged
merged 1 commit into from Nov 1, 2017

Conversation

Projects
None yet
4 participants
@jmchilton
Member

jmchilton commented Oct 27, 2017

  • Use sqlalchemy-utils to create postgres / mysql database if they don't exist the way it happens automatically for sqlite.
  • Add a new configuration parameter called database_template that allows creating postgres databases from an existing database using a TEMPLATE statement.
  • Use the database template parameter to give each integration test case class a clean postgres database.
@jmchilton

This comment has been minimized.

Member

jmchilton commented Oct 27, 2017

Hopefully something like this allows us to fix failing integration tests on the job handler and collection state PRs that work when you run them alone.

@@ -33,6 +33,8 @@
from sqlalchemy.orm.collections import attribute_mapped_collection
from sqlalchemy.types import BigInteger
from sqlalchemy_utils import database_exists, create_database

This comment has been minimized.

@nsoranzo

nsoranzo Oct 27, 2017

Member

Import order

@galaxybot galaxybot added this to the 18.01 milestone Oct 27, 2017

Improvements to database initialization.
- Use sqlalchemy-utils to create postgres / mysql database if they don't exist the way it happens automatically for sqlite.
- Add a new configuration parameter called database_template that allows creating postgres databases from an existing database using a TEMPLATE statement.
- Use the database template parameter to give each integration test case class a clean postgres database.
@jmchilton

This comment has been minimized.

Member

jmchilton commented Oct 27, 2017

@nsoranzo Thanks I fixed that and another linting problem.

jmchilton added a commit to jmchilton/galaxy that referenced this pull request Oct 30, 2017

Another round of Selenium enhancements and fixes for 18.01.
- Bake Selenium (chrome, chrome-wrapper, xvfb, and Selenium server) into Docker test image. (1)
- Correct usage of common_startup.sh to fetch dev wheels (don't use pip directly).
- Fix/enhance the test wrapper file to not set GALAXY_CONFIG_OVERRIDE_DATABASE_CONNECATION - this prevents integration database templating (galaxyproject#4887) from working with the Docker image.
- Update target Dockerfile.

1) The Docker compose setup for Selenium tests currently used by Jenkins is nice in that it uses official Selenium and Postgres Docker images - but it is problematic in that it takes more resources than the other single container Docker jobs, it doesn't cleanup as nicely, it doesn't have a pre-migrated database or pre-installed dependencies so it takes longer to run, and isn't easily exposed to developers via run_tests.sh. A more pragmatic Jenkins job based on this work should be runnable for every PR - the current variant simply takes too long and requires too many resources.

@martenson martenson merged commit 6b68c5d into galaxyproject:dev Nov 1, 2017

6 checks passed

api test Build finished. 306 tests run, 4 skipped, 0 failed.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
framework test Build finished. 162 tests run, 0 skipped, 0 failed.
Details
integration test Build finished. 57 tests run, 0 skipped, 0 failed.
Details
lgtm analysis: JavaScript No alert changes
Details
toolshed test Build finished. 577 tests run, 0 skipped, 0 failed.
Details

jmchilton added a commit to jmchilton/planemo that referenced this pull request Feb 23, 2018

Enhancements to Galaxy profiles and workflow testing.
Add a new profile database type option "postgres_docker". The existing "postgres" type option expects a local database to be accessible via the "psql" command on Planemo's PATH - probably few people have postgres running directly on their development machine and fewer still have it configured to allow passwordless entry. This newer variant is a much simpler interface - it starts a named container for the official Docker postgres image and uses that to manage the profile. This should make using profiles easier for most developers.

This new modality does have some shortcomings - it should probably be redone to persist the database somewhere -or- start and stop the same container (currently when the container is stopped the data is gone).

This also updates the workflow test that encounters Galaxy locks to pickup the environment variable PLANEMO_TEST_WORKFLOW_RUN_PROFILE and PLANEMO_TEST_WORKFLOW_RUN_PROFILE_DATABASE_TYPE and use these to create and use a profile during testing if set as well as setting that variable in Travis to leverage the postgres service already configured for other tests. This more less fixes what I had in mind with galaxyproject#772 but it would be nice to leverage galaxyproject/galaxy#4887 for greater test isolation.

jmchilton added a commit to jmchilton/planemo that referenced this pull request Feb 23, 2018

Enhancements to Galaxy profiles and workflow testing.
Add a new profile database type option "postgres_docker". The existing "postgres" type option expects a local database to be accessible via the "psql" command on Planemo's PATH - probably few people have postgres running directly on their development machine and fewer still have it configured to allow passwordless entry. This newer variant is a much simpler interface - it starts a named container for the official Docker postgres image and uses that to manage the profile. This should make using profiles easier for most developers.

This new modality does have some shortcomings - it should probably be redone to persist the database somewhere -or- start and stop the same container (currently when the container is stopped the data is gone).

This also updates the workflow test that encounters Galaxy locks to pickup the environment variable PLANEMO_TEST_WORKFLOW_RUN_PROFILE and PLANEMO_TEST_WORKFLOW_RUN_PROFILE_DATABASE_TYPE and use these to create and use a profile during testing if set as well as setting that variable in Travis to leverage the postgres service already configured for other tests. This more less fixes what I had in mind with galaxyproject#772 but it would be nice to leverage galaxyproject/galaxy#4887 for greater test isolation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment