Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assorted Fixes for Dockerized Test Setup #3755

Merged
merged 4 commits into from Mar 20, 2017

Conversation

Projects
None yet
3 participants
@jmchilton
Copy link
Member

commented Mar 14, 2017

  • 77b2988 Fixes using sh run_tests.sh --dockerzied on Mac OS X.
  • d1eab26 Provides a slightly improved experience if using the Dockerfile directly.
  • d665483 Appears to fix the pip caching issue that currently breaks the tests for #3179.

jmchilton added some commits Mar 14, 2017

Fix run_tests.sh --dockerize for Mac OS X usage.
Without this I get the error message below and the tests do not execute.

```
./run_tests.sh: line 279: UID: readonly variable
```

This was likely broken with b75ee68.
Fix pip install cache inside Galaxy testing container.
It is causing Jenkins to fail throughout on #3179.
@@ -4,6 +4,8 @@ set -e
echo "Deleting galaxy user - it may not exist and this is fine."
deluser galaxy | true

GALAXY_TEST_UID=${GALAXY_TEST_UID:-"1"}

This comment has been minimized.

Copy link
@natefoo

natefoo Mar 15, 2017

Member

: ${GALAXY_TEST_UID:="1"} is a less repetitive way to do this. (also done below with $GALAXY_VIRTUAL_ENV)

@dannon

This comment has been minimized.

Copy link
Member

commented Mar 17, 2017

(for anyone else testing this, it's --dockerize and not --dockerized)

@dannon

This comment has been minimized.

Copy link
Member

commented Mar 17, 2017

I'm getting the following error, just using the default args (w/ dockerize). Looks like it's read-only mounted into the docker container, maybe?

$ sh run_tests.sh --dockerize

<snipped>

Activating virtualenv at /galaxy_venv
131 -> 132...
Traceback (most recent call last):
  File "./scripts/manage_db.py", line 23, in <module>
    invoke_migrate_main()
  File "./scripts/manage_db.py", line 19, in invoke_migrate_main
    main( repository=repo, url=db_url )
  File "/galaxy_venv/local/lib/python2.7/site-packages/migrate/versioning/shell.py", line 209, in main
    ret = command_func(**kwargs)
  File "/galaxy_venv/local/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
    return _migrate(url, repository, version, upgrade=True, err=err, **opts)
  File "<string>", line 2, in _migrate
  File "/galaxy_venv/local/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
    return f(*a, **kw)
  File "/galaxy_venv/local/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
    schema.runchange(ver, change, changeset.step)
  File "/galaxy_venv/local/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
    change.run(self.engine, step)
  File "/galaxy_venv/local/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
    script_func(engine)
  File "lib/galaxy/model/migrate/versions/0132_add_lastpasswordchange_to_user.py", line 12, in upgrade
    lpc.create(account)
  File "/galaxy_venv/local/lib/python2.7/site-packages/migrate/changeset/schema.py", line 571, in create
    engine._run_visitor(visitorcallable, self, connection, **kwargs)
  File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1856, in _run_visitor
    conn._run_visitor(visitorcallable, element, **kwargs)
  File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1481, in _run_visitor
    **kwargs).traverse_single(element)
  File "/galaxy_venv/local/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 56, in traverse_single
    ret = super(AlterTableVisitor, self).traverse_single(elem)
  File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
    return meth(obj, **kw)
  File "/galaxy_venv/local/lib/python2.7/site-packages/migrate/changeset/databases/sqlite.py", line 139, in visit_column
    super(SQLiteColumnGenerator,self).visit_column(column)
  File "/galaxy_venv/local/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 104, in visit_column
    self.execute()
  File "/galaxy_venv/local/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 44, in execute
    return self.connection.execute(self.buffer.getvalue())
  File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 906, in execute
    return self._execute_text(object, multiparams, params)
  File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1054, in _execute_text
    statement, parameters
  File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) attempt to write a readonly database [SQL: u'\nALTER TABLE galaxy_user ADD last_password_change DATETIME']
@dannon

This comment has been minimized.

Copy link
Member

commented Mar 17, 2017

The caching also isn't quite there yet, relevant snip:

The directory '/root/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/root/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
@jmchilton

This comment has been minimized.

Copy link
Member Author

commented Mar 20, 2017

@dannon Is this after rebuilding the testing Docker container or are you still using the existing Docker container - in particular the caching thing?

For the Mac OS X fix in particular - I'd try this command:

./run_tests.sh --dockerize -db postgres -api test/api/test_datatypes.py

before and after the fix to demonstrate this is fixing the variable problem.

I guess you are hitting a pre-existing problem related to the sqlite database inside the Docker container which is never used by Jenkins. That was probably introduced in #3387.

@dannon

This comment has been minimized.

Copy link
Member

commented Mar 20, 2017

A fresh container was pulled when I tested it, but I didn't manually rebuild anything locally. What do I need to do to test the new container?

I ran the exact command above, and still hit:

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) attempt to write a readonly database [SQL: u'\nALTER TABLE galaxy_user ADD last_password_change DATETIME']

@jmchilton

This comment has been minimized.

Copy link
Member Author

commented Mar 20, 2017

@dannon - Okay it looks like the sqlite thing is fixed in dev - there just wasn't a push to Docker hub with the fix. So to test the latest image I would remove the existing download

docker rmi galaxy/testing-base:17.01.0

Rebuild it locally:

docker build -t galaxy/testing-base:17.01.0 test/docker/base

and run

./run_tests.sh --dockerize --db postgres -api test/api/test_datatypes.py

My previous variant of this last command had -db postgres instead of --db postgres. This would eventually produce an error about unknown commands - but only once it had been sent into the container and the database upgraded - which cannot be done against the latest thing on Docker Hub - but should be fine with the Docker image defined by the Dockerfile in dev on in this PR.

@dannon

This comment has been minimized.

Copy link
Member

commented Mar 20, 2017

@jmchilton Ok, rebuilding and retesting now, will let you know if it works!

@jmchilton

This comment has been minimized.

Copy link
Member Author

commented Mar 20, 2017

@dannon Thanks a ton for testing and sorry about my old sloppiness masking problems with my current sloppiness.

@dannon

This comment has been minimized.

Copy link
Member

commented Mar 20, 2017

It works!

@dannon dannon merged commit cd36aa0 into galaxyproject:dev Mar 20, 2017

5 checks passed

api test Build finished. 263 tests run, 0 skipped, 0 failed.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
framework test Build finished. 140 tests run, 0 skipped, 0 failed.
Details
integration test Build finished. 25 tests run, 0 skipped, 0 failed.
Details
toolshed test Build finished. 580 tests run, 0 skipped, 0 failed.
Details
@jmchilton

This comment has been minimized.

Copy link
Member Author

commented Mar 20, 2017

Awesome - thanks @dannon!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.