Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catch DatabaseError in fetch/gather callbacks #301

Merged
merged 2 commits into from
Nov 7, 2017
Merged

Conversation

dlax
Copy link
Contributor

@dlax dlax commented Nov 2, 2017

I occasionally get the following errors on a production server:

  File "/home/opendata/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 183,
 in command
    gather_callback(consumer, method, header, body)
  File "/home/opendata/src/ckanext-harvest/ckanext/harvest/queue.py", line 276, in gather_ca
llback
    job = HarvestJob.get(id)
  File "/home/opendata/src/ckanext-harvest/ckanext/harvest/model/__init__.py", line 118, in
get
    o = cls.filter(**kwds).first()
  File "/home/opendata/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2334,
 in first
    ret = list(self[0:1])
  File "/home/opendata/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2201,
 in __getitem__
    return list(res)
  File "/home/opendata/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2405,
 in __iter__
    return self._execute_and_instances(context)
  File "/home/opendata/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2420,
 in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/home/opendata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 727
, in execute
    return meth(self, multiparams, params)
  File "/home/opendata/venv/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 32
2, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/opendata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 824
, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/opendata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 954
, in _execute_context
    context)
  File "/home/opendata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 111
6, in _handle_dbapi_exception
    exc_info
  File "/home/opendata/venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 189
, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/home/opendata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 947
, in _execute_context
    context)
  File "/home/opendata/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 435, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.DatabaseError: (DatabaseError) could not receive data from server: Connection timed out

This makes respective harvest jobs going in "limbo".

So in this PR (second commit) I changed the except sqlalchemy.exc.OperationalError into except sqlalchemy.exc.DatabaseError (the latter exception class being a superclass of the former).

Along the way, the first commit just cleans up the logging logic.

Denis Laxalde added 2 commits November 2, 2017 17:17
logging.exception() already logs an ERROR message with exception
information, so there's no need to call both log.exception() and
log.error().

Along the way, make messages uniform in fetch_callback() and
gather_callback().
I sometimes see "connection timed out" message which are reported as
sqlalchemy.exc.DatabaseError, so by catching the latter exception, it'd
avoid the harvester to be stuck in "limbo" state.

As DatabaseError is a super-class of OperationalError, the latter would
still be catched.
@metaodi metaodi self-assigned this Nov 7, 2017
@metaodi metaodi merged commit cccc469 into ckan:master Nov 7, 2017
@metaodi
Copy link
Member

metaodi commented Nov 7, 2017

Thanks for this PR @dlax

@dlax dlax deleted the db-errors branch November 7, 2017 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants