Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Drop allow_none for finding repository #1358

Merged
merged 2 commits into from Mar 8, 2023
Merged

Drop allow_none for finding repository #1358

merged 2 commits into from Mar 8, 2023

Conversation

mvdbeek
Copy link
Member

@mvdbeek mvdbeek commented Mar 8, 2023

This was only used in the RealizedRepository class where it is being caught anyway. Should make debugging mismatched in name/owner of .shed.yml much easier. Probably fixes #1353 ?

@mvdbeek
Drop allow_none for finding repository
80865bf 
This was only used in the RealizedRepository class where it is being
caught anyway. Should make debugging mismatched in name/owner of
.shed.yml much easier.
nsoranzo
nsoranzo approved these changes Mar 8, 2023
View reviewed changes
planemo/shed/__init__.py Outdated Show resolved Hide resolved
@mvdbeek @nsoranzo
Use f-string for exception message
622aa4a 
Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>
@nsoranzo nsoranzo enabled auto-merge March 8, 2023 12:34
@nsoranzo
Copy link
Member

nsoranzo commented Mar 8, 2023

Failing tests seem unrelated.

@mvdbeek
Copy link
Member Author

mvdbeek commented Mar 8, 2023

Yeah, probably a change in the dev branch that's not playing nice with sqlite:

galaxy.workflow.run DEBUG 2023-03-08 13:25:16,334 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] Workflow step 3 of invocation 1 invoked (1.195 ms)
galaxy.managers.collections DEBUG 2023-03-08 13:25:16,381 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] Created collection with 1 elements
galaxy.tools.actions.model_operations INFO 2023-03-08 13:25:16,383 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] Calling produce_outputs, tool is <galaxy.tools.ApplyRulesTool object at 0x7f025f54e650>
galaxy.tools.execute DEBUG 2023-03-08 13:25:16,383 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] Tool __APPLY_RULES__ created job None (14.443 ms)
galaxy.web_stack.handlers INFO 2023-03-08 13:25:16,413 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] (Job[id=3,tool_id=__APPLY_RULES__]) Handler '_default_' assigned using 'HANDLER_ASSIGNMENT_METHODS.DB_SKIP_LOCKED' assignment method
galaxy.tools.execute DEBUG 2023-03-08 13:25:16,415 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] Created 1 job(s) for tool __APPLY_RULES__ request (78.566 ms)
galaxy.workflow.run DEBUG 2023-03-08 13:25:16,420 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] Workflow step 2 of invocation 1 invoked (86.011 ms)
galaxy.tools.actions INFO 2023-03-08 13:25:16,436 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] Handled output named out_file1 for tool cat_list (1.032 ms)

galaxy.tools ERROR 2023-03-08 13:25:21,447 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] Exception caught while attempting to execute tool with id 'cat_list':
Traceback (most recent call last):
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
    cursor, statement, parameters, context
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/tmp99kwunn_/galaxy-dev/lib/galaxy/tools/__init__.py", line 2069, in handle_single_execution
    skip=skip,
  File "/tmp/tmp99kwunn_/galaxy-dev/lib/galaxy/tools/__init__.py", line 2155, in execute
    self, trans, incoming=incoming, set_output_hid=set_output_hid, history=history, **kwargs
  File "/tmp/tmp99kwunn_/galaxy-dev/lib/galaxy/tools/actions/__init__.py", line 638, in execute
    history.add_pending_items(set_output_hid=set_output_hid)
  File "/tmp/tmp99kwunn_/galaxy-dev/lib/galaxy/model/__init__.py", line 2900, in add_pending_items
    object_session(self), self._pending_additions, set_hid=set_output_hid, quota=False, flush=False
  File "/tmp/tmp99kwunn_/galaxy-dev/lib/galaxy/model/__init__.py", line 2992, in add_datasets
    self.add_dataset(dataset, parent_id=parent_id, genome_build=genome_build, set_hid=set_hid, quota=quota)
  File "/tmp/tmp99kwunn_/galaxy-dev/lib/galaxy/model/__init__.py", line 2960, in add_dataset
    dataset.hid = self._next_hid()
  File "/tmp/tmp99kwunn_/galaxy-dev/lib/galaxy/model/__init__.py", line 2928, in _next_hid
    conn.execute(update_stmt)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 335, in _execute_on_connection
    self, multiparams, params, execution_options
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1582, in _execute_clauseelement
    cache_hit=cache_hit,
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1944, in _execute_context
    e, statement, parameters, cursor, context
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2125, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
    cursor, statement, parameters, context
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: UPDATE history SET update_time=?, hid_counter=(history.hid_counter + ?) WHERE history.id = ?]
[parameters: ('2023-03-08 13:25:16.441239', 1, 1)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
galaxy.tools.execute WARNING 2023-03-08 13:25:21,452 [pN:main.1,p:20282,tN:WorkflowRequestMonitor.monitor_thread] There was a failure executing a job for tool [cat_list] - Error executing tool with id 'cat_list': (sqlite3.OperationalError) database is locked
[SQL: UPDATE history SET update_time=?, hid_counter=(history.hid_counter + ?) WHERE history.id = ?]
[parameters: ('2023-03-08 13:25:16.441239', 1, 1)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
galaxy.jobs.handler DEBUG 2023-03-08 13:25:22,113 [pN:main.1,p:20282,tN:JobHandlerQueue.monitor_thread] Grabbing Job failed (serialization failures are ok): (sqlite3.OperationalError) database is locked
[SQL: UPDATE job SET update_time=?, handler=? WHERE job.id IN (SELECT job.id 
FROM job 
WHERE job.handler IN (?) AND job.state = ? ORDER BY job.id)]
[parameters: ('2023-03-08 13:25:17.105546', 'main.1', '_default_', <states.NEW: 'new'>)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

kombu.mixins WARNING 2023-03-08 13:25:22,402 [pN:main.1,p:20282,tN:Thread-1] Connection to broker lost, trying to re-establish connection...
Traceback (most recent call last):
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
    cursor, statement, parameters, context
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/mixins.py", line 171, in run
    for _ in self.consume(limit=None, **kwargs):
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/mixins.py", line 193, in consume
    conn.drain_events(timeout=safety_interval)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/connection.py", line 316, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 971, in drain_events
    get(self._deliver, timeout=timeout)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/utils/scheduling.py", line 52, in get
    return self.fun(resource, callback, **kwargs)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 1009, in _drain_channel
    return channel.drain_events(callback=callback, timeout=timeout)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 736, in drain_events
    return self._poll(self.cycle, callback, timeout=timeout)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 400, in _poll
    return cycle.get(callback)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/utils/scheduling.py", line 52, in get
    return self.fun(resource, callback, **kwargs)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 403, in _get_and_deliver
    message = self._get(queue)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 167, in _get
    self.session.execute('BEGIN IMMEDIATE TRANSACTION')
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1714, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 335, in _execute_on_connection
    self, multiparams, params, execution_options
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1582, in _execute_clauseelement
    cache_hit=cache_hit,
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1944, in _execute_context
    e, statement, parameters, cursor, context
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2125, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
    cursor, statement, parameters, context
  File "/tmp/planemo-test-workspace/gx_venv_3_dev/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: BEGIN IMMEDIATE TRANSACTION]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

any idea on that @jdavcs ?

@mvdbeek mvdbeek disabled auto-merge March 8, 2023 14:27
@mvdbeek mvdbeek merged commit 0bf7c24 into galaxyproject:master Mar 8, 2023
9 of 11 checks passed
@jdavcs
Copy link
Member

jdavcs commented Mar 8, 2023

Yeah, probably a change in the dev branch that's not playing nice with sqlite:

It's very possible that it may be related/caused by recent changes to db transaction handling. I'll investigate.

@jdavcs
Copy link
Member

jdavcs commented Jun 8, 2023

The sqlite database locking error has been fixed in galaxyproject/galaxy#16208.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nsoranzo nsoranzo nsoranzo approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

shed_create with testtoolshed fails
3 participants
@mvdbeek @nsoranzo @jdavcs