New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

repeated database locked (Operational Error) issues in high activity, db intensive application. #280

Closed
steelliberty opened this Issue Jul 25, 2017 · 8 comments

Comments

Projects
None yet
3 participants
@steelliberty

steelliberty commented Jul 25, 2017

Dear Sir :

I am using pony 0.7.1 with PyPy 5.6.0 and sqlite3 in an application that monitors hundreds of sites for being up using ping and http, tcp pings. Status records are generated every second of an hour logging information about the current 'ping.' I have on a frequent basis, an error as shown below.

I have felt comfortable with sqlite3 until recent increases in this error give me concern as to whether there is a better way to operate on a Status record ( I have added the volatility flag to the DB ) or if their is a typical way to wait and try again ( I do try again but always with the same exception) . I understand that poor programming can explain many issues with database access, but I have studied comments from various sources about this situation and I do not see where I am improperly attempting this operation.

I am reluctant to move away from sqlite3 because it is otherwise stable -- hoping to avoid a new collection of issues dealing with another db such as MySQL ..

File "scan_notify.py", line 669, in run
self.checkDownMonitors()
File "<auto generated wrapper of checkDownMonitors() function>", line 2, in checkDownMonitors
File "/tombstone/site-packages/pony/orm/core.py", line 413, in new_func
try: return func(*args, **kwargs)
File "scan_notify.py", line 357, in checkDownMonitors
mainCheck()
File "<auto generated wrapper of mainCheck() function>", line 2, in mainCheck
File "/tombstone/site-packages/pony/orm/core.py", line 413, in new_func
try: return func(*args, **kwargs)
File "scan_notify.py", line 156, in mainCheck
ss = Status.get_for_update(id=newsitesdowndict[nsdhost].id)
File "<auto generated wrapper of get_for_update() function>", line 2, in get_for_update
File "/tombstone/site-packages/pony/utils/utils.py", line 58, in cut_traceback
return func(*args, **kwargs)
File "/tombstone/site-packages/pony/orm/core.py", line 3586, in get_for_update
try: return entity.find_one(kwargs, True, nowait) # can throw MultipleObjectsFoundError
File "/tombstone/site-packages/pony/orm/core.py", line 3670, in find_one
if obj is None: obj = entity.find_in_db(avdict, unique, for_update, nowait)
File "/tombstone/site-packages/pony/orm/core.py", line 3729, in find_in_db
cursor = database._exec_sql(sql, arguments)
File "/tombstone/site-packages/pony/orm/core.py", line 696, in _exec_sql
connection = cache.prepare_connection_for_query_execution()
File "/tombstone/site-packages/pony/orm/core.py", line 1536, in prepare_connection_for_query_execution
except Exception as e: connection = cache.reconnect(e)
File "/tombstone/site-packages/pony/orm/core.py", line 1511, in reconnect
if not provider.should_reconnect(exc): reraise(*sys.exc_info())
File "/tombstone/site-packages/pony/orm/core.py", line 1535, in prepare_connection_for_query_execution
try: provider.set_transaction_mode(connection, cache) # can set cache.in_transaction
File "<auto generated wrapper of set_transaction_mode() function>", line 2, in set_transaction_mode
File "/tombstone/site-packages/pony/orm/dbapiprovider.py", line 53, in wrap_dbapi_exceptions
except dbapi_module.OperationalError as e: raise OperationalError(e)
OperationalError: database is locked

@kozlovsky

This comment has been minimized.

Show comment
Hide comment
@kozlovsky

kozlovsky Jul 25, 2017

Member

Hi, can you try it with Pony 0.7.2? we fixed some issues with SQLite3 locking, maybe it solves your problem

Member

kozlovsky commented Jul 25, 2017

Hi, can you try it with Pony 0.7.2? we fixed some issues with SQLite3 locking, maybe it solves your problem

@steelliberty

This comment has been minimized.

Show comment
Hide comment
@steelliberty

steelliberty Jul 25, 2017

I will do it ... and let you know.

I love PONY .. why would anyone do databases any other way .. surprising that many include db specific code in their python db-applications, when using pony abstracts away the issues with knowing multiple db code. Thanks -- I'll need to let it run a few hours before reporting results .

steelliberty commented Jul 25, 2017

I will do it ... and let you know.

I love PONY .. why would anyone do databases any other way .. surprising that many include db specific code in their python db-applications, when using pony abstracts away the issues with knowing multiple db code. Thanks -- I'll need to let it run a few hours before reporting results .

@steelliberty

This comment has been minimized.

Show comment
Hide comment
@steelliberty

steelliberty Jul 25, 2017

Okay .. it takes a few hours for a circumstance to cause the database lock. It happens during activity created because of sites being down -- The same errors appear again.

Would there be any difference if I used select for_update ?

File "scan_notify.py", line 148, in mainCheck
ss = Status.get_for_update(id=newsitesdowndict[nsdhost].id)
File "<auto generated wrapper of get_for_update() function>", line 2, in get_for_update
File "/tombstone/site-packages/pony/utils/utils.py", line 58, in cut_tombstoneback
return func(*args, **kwargs)
File "/tombstone/site-packages/pony/orm/core.py", line 3593, in get_for_update
try: return entity.find_one(kwargs, True, nowait) # can throw MultipleObjectsFoundError
File "/tombstone/site-packages/pony/orm/core.py", line 3677, in find_one
if obj is None: obj = entity.find_in_db(avdict, unique, for_update, nowait)
File "/tombstone/site-packages/pony/orm/core.py", line 3736, in find_in_db
cursor = database._exec_sql(sql, arguments)
File "/tombstone/site-packages/pony/orm/core.py", line 699, in _exec_sql
connection = cache.prepare_connection_for_query_execution()
File "/tombstone/site-packages/pony/orm/core.py", line 1539, in prepare_connection_for_query_execution
except Exception as e: connection = cache.reconnect(e)
File "/tombstone/site-packages/pony/orm/core.py", line 1514, in reconnect
if not provider.should_reconnect(exc): reraise(*sys.exc_info())
File "/tombstone/site-packages/pony/orm/core.py", line 1538, in prepare_connection_for_query_execution
try: provider.set_transaction_mode(connection, cache) # can set cache.in_transaction
File "<auto generated wrapper of set_transaction_mode() function>", line 2, in set_transaction_mode
File "/tombstone/site-packages/pony/orm/dbapiprovider.py", line 53, in wrap_dbapi_exceptions
except dbapi_module.OperationalError as e: raise OperationalError(e)
OperationalError: database is locked

steelliberty commented Jul 25, 2017

Okay .. it takes a few hours for a circumstance to cause the database lock. It happens during activity created because of sites being down -- The same errors appear again.

Would there be any difference if I used select for_update ?

File "scan_notify.py", line 148, in mainCheck
ss = Status.get_for_update(id=newsitesdowndict[nsdhost].id)
File "<auto generated wrapper of get_for_update() function>", line 2, in get_for_update
File "/tombstone/site-packages/pony/utils/utils.py", line 58, in cut_tombstoneback
return func(*args, **kwargs)
File "/tombstone/site-packages/pony/orm/core.py", line 3593, in get_for_update
try: return entity.find_one(kwargs, True, nowait) # can throw MultipleObjectsFoundError
File "/tombstone/site-packages/pony/orm/core.py", line 3677, in find_one
if obj is None: obj = entity.find_in_db(avdict, unique, for_update, nowait)
File "/tombstone/site-packages/pony/orm/core.py", line 3736, in find_in_db
cursor = database._exec_sql(sql, arguments)
File "/tombstone/site-packages/pony/orm/core.py", line 699, in _exec_sql
connection = cache.prepare_connection_for_query_execution()
File "/tombstone/site-packages/pony/orm/core.py", line 1539, in prepare_connection_for_query_execution
except Exception as e: connection = cache.reconnect(e)
File "/tombstone/site-packages/pony/orm/core.py", line 1514, in reconnect
if not provider.should_reconnect(exc): reraise(*sys.exc_info())
File "/tombstone/site-packages/pony/orm/core.py", line 1538, in prepare_connection_for_query_execution
try: provider.set_transaction_mode(connection, cache) # can set cache.in_transaction
File "<auto generated wrapper of set_transaction_mode() function>", line 2, in set_transaction_mode
File "/tombstone/site-packages/pony/orm/dbapiprovider.py", line 53, in wrap_dbapi_exceptions
except dbapi_module.OperationalError as e: raise OperationalError(e)
OperationalError: database is locked

@steelliberty

This comment has been minimized.

Show comment
Hide comment
@steelliberty

steelliberty Jul 26, 2017

or do you believe there is an issue using sqlite3 and because of the moderate increase in database access I should have used MySQL or one of the other DBs ?

steelliberty commented Jul 26, 2017

or do you believe there is an issue using sqlite3 and because of the moderate increase in database access I should have used MySQL or one of the other DBs ?

@kozlovsky

This comment has been minimized.

Show comment
Hide comment
@kozlovsky

kozlovsky Jul 26, 2017

Member

This exception may arise if some transaction is very long and cannot be finished before the default timeout of 5 seconds. If this is the case, you can try to increase the timeout value. Unfortunately, Pony does not provide API for that. In the next release we need to add timeout option to bind method when using 'sqlite' provider.

Right now, in order to check that the reason of the problem is insufficiently big timeout value, you can find line 516 of file pony/orm/dbproviders/sqlite.py:

pool.con = con = sqlite.connect(filename, isolation_level=None)

and set a timeout (in seconds) to some value bigger then the default one

pool.con = con = sqlite.connect(filename, isolation_level=None, timeout=1000.0)

On the other hand, if all transactions are short and increasing of timeout value gives no result, then it looks like a bug in Pony, and we need to debug and fix it.

Member

kozlovsky commented Jul 26, 2017

This exception may arise if some transaction is very long and cannot be finished before the default timeout of 5 seconds. If this is the case, you can try to increase the timeout value. Unfortunately, Pony does not provide API for that. In the next release we need to add timeout option to bind method when using 'sqlite' provider.

Right now, in order to check that the reason of the problem is insufficiently big timeout value, you can find line 516 of file pony/orm/dbproviders/sqlite.py:

pool.con = con = sqlite.connect(filename, isolation_level=None)

and set a timeout (in seconds) to some value bigger then the default one

pool.con = con = sqlite.connect(filename, isolation_level=None, timeout=1000.0)

On the other hand, if all transactions are short and increasing of timeout value gives no result, then it looks like a bug in Pony, and we need to debug and fix it.

@steelliberty

This comment has been minimized.

Show comment
Hide comment
@steelliberty

steelliberty Jul 26, 2017

steelliberty commented Jul 26, 2017

@steelliberty

This comment has been minimized.

Show comment
Hide comment
@steelliberty

steelliberty Jul 28, 2017

steelliberty commented Jul 28, 2017

@kozlovsky kozlovsky self-assigned this Aug 25, 2017

@kozlovsky kozlovsky closed this in 3a649fe Aug 25, 2017

@sashaaero

This comment has been minimized.

Show comment
Hide comment
@sashaaero

sashaaero Aug 25, 2017

Member

Now it's possible to specify timeout option, as well as other keyword arguments for sqlite3.connect:

db.bind(provider='sqlite', filename='db.sqlite', timeout=10)
Member

sashaaero commented Aug 25, 2017

Now it's possible to specify timeout option, as well as other keyword arguments for sqlite3.connect:

db.bind(provider='sqlite', filename='db.sqlite', timeout=10)

@kozlovsky kozlovsky added this to the 0.7.3 milestone Aug 25, 2017

kozlovsky added a commit that referenced this issue Oct 23, 2017

Pony ORM Release 0.7.3 (2017-10-23)
# New features

* `where()` method added to query
* `coalesce()` function added
* `between(x, a, b)` function added
* #295: Add `_table_options_` for entity class to specify engine, tablespace, etc.
* Make debug flag thread-local
* `sql_debugging` context manager added
* `sql_debug` and show_values arguments to db_session added
* `set_sql_debug` function added as alias to (to be deprecated) `sql_debug` function
* Allow `db_session` to accept `ddl` parameter when used as context manager
* Add `optimistic=True` option to db_session
* Skip optimistic checks for queries in `db_session` with `serializable=True`
* `fk_name` option added for attributes in order to specify foreign key name
* #280: Now it's possible to specify `timeout` option, as well as pass other keyword arguments for `sqlite3.connect` function
* Add support of explicit casting to int in queries using `int()` function
* Added modulo division % native support in queries

# Bugfixes

* Fix bugs with composite table names
* Fix invalid foreign key & index names for tables which names include schema name
* For queries like `select(x for x in MyObject if not x.description)` add "OR x.info IS NULL" for nullable string columns
* Add optimistic checking for `delete()` method
* Show updated attributes when `OptimisticCheckError` is being raised
* Fix incorrect aliases in nested queries
* Correctly pass exception from user-defined functions in SQLite
* More clear error messages for `UnrepeatableReadError`
* Fix `db_session(strict=True)` which was broken in 2d3afb2
* Fixes #170: Problem with a primary key column used as a part of another key
* Fixes #223: incorrect result of `getattr(entity, attrname)` when the same lambda applies to different entities
* Fixes #266: Add handler to `"pony.orm"` logger does not work
* Fixes #278: Cascade delete error: FOREIGN KEY constraint failed, with complex entity relationships
* Fixes #283: Lost Json update immediately after object creation
* Fixes #284: `query.order_by()` orders Json numbers like strings
* Fixes #288: Expression text parsing issue in Python 3
* Fixes #293: translation of if-expressions in expression
* Fixes #294: Real stack traces swallowed within IPython shell
* `Collection.count()` method should check if session is alive
* Set `obj._session_cache_` to None after exiting from db session for better garbage collection
* Unload collections which are not fully loaded after exiting from db session for better garbage collection
* Raise on unknown options for attributes that are part of relationship
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment