Keep separate session for ToolShedRepositoryCache

[mvdbeek]

Should fix

Traceback (most recent call last):
  File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/jobs/runners/__init__.py", line 244, in prepare_job
    stderr_file=stderr_file,
  File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/jobs/runners/__init__.py", line 285, in build_command_line
    stderr_file=stderr_file,
  File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/jobs/command_factory.py", line 79, in build_command
    __handle_dependency_resolution(commands_builder, job_wrapper, remote_command_params)
  File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/jobs/command_factory.py", line 194, in __handle_dependency_resolution
    if local_dependency_resolution and job_wrapper.dependency_shell_commands:
  File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/jobs/__init__.py", line 1012, in dependency_shell_commands
    job_directory=self.working_directory
  File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/tools/__init__.py", line 1891, in build_dependency_shell_commands
    tool_instance=self
  File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/tool_util/deps/__init__.py", line 184, in dependency_shell_commands
    requirements_to_dependencies = self.requirements_to_dependencies(requirements, **kwds)
  File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/tool_util/deps/__init__.py", line 194, in requirements_to_dependencies
    requirement_to_dependency = self._requirements_to_dependencies_dict(requirements, **kwds)
  File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/tool_util/deps/__init__.py", line 283, in _requirements_to_dependencies_dict
    dependency = resolver.resolve(requirement, **kwds)
  File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/tool_util/deps/resolvers/galaxy_packages.py", line 79, in resolve
    return self._find_dep_versioned(name, version, type=type, **kwds)
  File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/tool_util/deps/resolvers/tool_shed_packages.py", line 25, in _find_dep_versioned
    path = self._get_package_installed_dependency_path(installed_tool_dependency, name, version)
  File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/tool_util/deps/resolvers/tool_shed_packages.py", line 43, in _get_package_installed_dependency_path
    tool_shed_repository = installed_tool_dependency.tool_shed_repository
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 480, in __get__
    return self.impl.get(state, dict_)
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 931, in get
    value = self.callable_(state, passive)
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 836, in _load_for_state
    % (orm_util.state_str(state), self.key)
sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <ToolDependency at 0x7fd3e5e82160> is not bound to a Session; lazy load operation of attribute 'tool_shed_repository' cannot proceed (Background on this error at: http://sqlalche.me/e/14/bhk3)

Which I think broke in
#11737 (comment).

Instead of hoping that we preload all attributes it seems more robust to
use a separate session for the cache that is not tied to a thread and
that will not be closed by flushes in other parts of the application.

(Please replace this header with a description of your pull request. Please include BOTH what you did and why you made the changes. The "why" may simply be citing a relevant Galaxy issue.)
(If fixing a bug, please add any relevant error or traceback)
(For UI components, it is recommended to include screenshots or screencasts)

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these contributions under Galaxy's current license.
• I agree to allow the Galaxy committers to license these and all my past contributions to the core galaxy codebase under the MIT license. If this condition is an issue, uncheck and just let us know why with an e-mail to galaxy-committers@lists.galaxyproject.org.

[natefoo]

This appears to work on test.galaxyproject.org.

[mvdbeek]

hmm, but those test errors are real

[natefoo]

Also, just got this on a reload_toolbox task 😢

galaxy.queue_worker ERROR 2021-08-30 16:11:58,123 [pN:test_handler2,p:30341,tN:Thread-1] Error running control task type: reload_toolbox
Traceback (most recent call last):
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1671, in _execute_context
    dialect, self, conn, execution_options, *args, **kw
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 995, in _init_compiled
    self.cursor = self.create_cursor()
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 1343, in create_cursor
    return self.create_default_cursor()
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 1346, in create_default_cursor
    return self._dbapi_connection.cursor()
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 982, in cursor
    return self.connection.cursor(*args, **kwargs)
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 139815756478272 and this is thread id 139809042638592.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/queue_worker.py", line 388, in process_task
    result = f(self.app, **body['kwargs'])
  File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/queue_worker.py", line 171, in reload_toolbox
    _get_new_toolbox(app, save_integrated_tool_panel)
  File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/queue_worker.py", line 185, in _get_new_toolbox
    app.tool_shed_repository_cache.rebuild()
  File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/tools/cache.py", line 304, in rebuild
    defer(ToolShedRepository.metadata), joinedload('tool_dependencies')
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2709, in all
    return self._iter().all()
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2847, in _iter
    execution_options={"_sa_orm_load_options": self.load_options},
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1689, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1582, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 324, in _execute_on_connection
    self, multiparams, params, execution_options
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1461, in _execute_clauseelement
    cache_hit=cache_hit,
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1677, in _execute_context
    e, util.text_type(statement), parameters, None, None
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1995, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1671, in _execute_context
    dialect, self, conn, execution_options, *args, **kw
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 995, in _init_compiled
    self.cursor = self.create_cursor()
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 1343, in create_cursor
    return self.create_default_cursor()
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 1346, in create_default_cursor
    return self._dbapi_connection.cursor()
  File "/cvmfs/test.galaxyproject.org/venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 982, in cursor
    return self.connection.cursor(*args, **kwargs)
sqlalchemy.exc.ProgrammingError: (sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 139815756478272 and this is thread id 139809042638592.

[mvdbeek]

Alright, it's green.