New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"db_session required" exception when working with aync def coroutines #380

Closed
amireldor opened this Issue Sep 10, 2018 · 12 comments

Comments

Projects
None yet
4 participants
@amireldor
Copy link

amireldor commented Sep 10, 2018

Hey there. I'm having an issue that reminds #126 which should've been resolved. I'm also using tornado. I've managed to reproduce it with this minimal code:

https://gist.github.com/amireldor/3c7f1701761d6c92855b868e5df07af0

I wrap a coroutine (async def) with db_session but an exception is thrown when doing a database action inside the coroutine. Regular wrapped functions or with blocks are fine.

This is Python 3.6.5 and pony 0.7.6; getting "pony.orm.core.TransactionError: db_session is required when working with the database".

@ranisalt

This comment has been minimized.

Copy link

ranisalt commented Sep 17, 2018

I am having the same issue on Python 3.7. I work around it using with inside async def functions, but the extra level of indentation is undesirable.

@kozlovsky kozlovsky closed this in fe56efa Sep 22, 2018

@kozlovsky

This comment has been minimized.

Copy link
Member

kozlovsky commented Sep 22, 2018

Thanks for reporting, it should work now!

@ranisalt

This comment has been minimized.

Copy link

ranisalt commented Sep 22, 2018

It does not. The following code:

class AuthCheckHandler(BaseHandler):
    @orm.db_session
    async def get(self):
        user = self.current_user
        if user:
            self.finish({
                'user': user.to_dict(only=['first_name', 'last_name']),
            })

throws

    Traceback (most recent call last):
      File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/pony/orm/core.py", line 544, in new_gen_func
        output = wrapped_interact(iterator)
      File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/pony/orm/core.py", line 531, in wrapped_interact
        rollback_and_reraise(sys.exc_info())
      File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/pony/orm/core.py", line 327, in rollback_and_reraise
        reraise(*exc_info)
      File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/pony/utils/utils.py", line 118, in reraise
        try: raise exc.with_traceback(tb)
      File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/pony/orm/core.py", line 521, in wrapped_interact
        output = interact(iterator, input, exc_info)
      File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/pony/orm/core.py", line 494, in interact
        return next(iterator) if input is None else iterator.send(input)
    StopIteration
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/tornado/web.py", line 1592, in _execute
        result = yield result
      File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run
        value = future.result()
    RuntimeError: generator raised StopIteration

I guess db_session decorator is consuming the return value in a way that Tornado can't handle.

@ranisalt

This comment has been minimized.

Copy link

ranisalt commented Sep 24, 2018

Here's a different, longer traceback of the same issue:

    Traceback (most recent call last):
      File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/tornado/web.py", line 1592, in _execute
        result = yield result
      File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run
        value = future.result()
      File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/pony/orm/core.py", line 544, in new_gen_func
        output = wrapped_interact(iterator)
      File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/pony/orm/core.py", line 531, in wrapped_interact
        rollback_and_reraise(sys.exc_info())
      File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/pony/orm/core.py", line 327, in rollback_and_reraise
        reraise(*exc_info)
      File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/pony/utils/utils.py", line 118, in reraise
        try: raise exc.with_traceback(tb)
      File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/pony/orm/core.py", line 525, in wrapped_interact
        'You need to manually commit() changes before exiting from the generator')
      File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/pony/utils/utils.py", line 129, in throw
        raise exc
    pony.orm.core.TransactionError: You need to manually commit() changes before exiting from the generator

@sashaaero sashaaero reopened this Oct 3, 2018

@sashaaero sashaaero closed this in 4f0f2ad Oct 20, 2018

@sashaaero

This comment has been minimized.

Copy link
Member

sashaaero commented Oct 20, 2018

This should solve the last problem. Please respond if it doesn't.

@ranisalt

This comment has been minimized.

Copy link

ranisalt commented Oct 20, 2018

It does not:

Traceback (most recent call last):
File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/tornado/web.py", line 1592, in _execute
  result = yield result
File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run
  value = future.result()
File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/pony/orm/core.py", line 544, in new_gen_func
  output = wrapped_interact(iterator)
File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/pony/orm/core.py", line 511, in wrapped_interact
  '@db_session-wrapped generator cannot be used inside another db_session')
File "/home/rsa/swartz/.venv/lib/python3.7/site-packages/pony/utils/utils.py", line 129, in throw
  raise exc
pony.orm.core.TransactionError: @db_session-wrapped generator cannot be used inside another db_session
@sashaaero

This comment has been minimized.

Copy link
Member

sashaaero commented Oct 20, 2018

Can you provide an example?

@kozlovsky

This comment has been minimized.

Copy link
Member

kozlovsky commented Oct 20, 2018

What exception says is true: "@db_session-wrapped generator cannot be used inside another db_session"

@kozlovsky

This comment has been minimized.

Copy link
Member

kozlovsky commented Oct 20, 2018

Well, actually we can temporarily switch outer db session to generator db session on resuming generator, but I'm not sure is it correct usage or not

@ranisalt

This comment has been minimized.

Copy link

ranisalt commented Oct 20, 2018

Can you provide an example?

I will try to isolate the issue in a gist

@kozlovsky

This comment has been minimized.

Copy link
Member

kozlovsky commented Oct 20, 2018

In my understanding, it looks incorrect when someone directly calls async (db_session-decorated) function from a normal (db_session-decorated) function. There should be some dispatcher from which async functions are actually called

@ranisalt

This comment has been minimized.

Copy link

ranisalt commented Oct 20, 2018

Ok, @kozlovsky answer helped me find the issue. While my code works fine, tests are also wrapped with db_session so there is a nested db_session function call. Something like this:

@db_session
async def get():
  u = self.db.User[self.id]
  do_something(u)

@db_session
def test_get(http_client):
  res = http_client.get('https://localhost/path/to/get()')

So it opens another db_session while the previous was already open.

Is there a way to make db_session reentrant or at least a known way to work around this scenario?

kozlovsky added a commit that referenced this issue Jan 17, 2019

PonyORM release 0.7.7 (2019-01-17)
# Major features

* Array type support for PostgreSQL and SQLite
* isinstance() support in queries
* Support of queries based on collections: select(x for x in y.items)

# Other features

* Support of Entity.select(**kwargs)
* Support of SKIP LOCKED option in 'SELECT ... FOR UPDATE'
* New function make_proxy(obj) to make cros-db_session proxy objects
* Specify ON DELETE CASCADE/SET NULL in foreign keys
* Support of LIMIT in `SELECT FROM (SELECT ...)` type of queries
* Support for negative JSON array indexes in SQLite

# Improvements

* Improved query prefetching: use fewer number of SQL queries
* Memory optimization: deduplication of values recieved from the database in the same session
* increase DBAPIProvider.max_params_count value

# Bugfixes

* #405: breaking change with cx_Oracle 7.0: DML RETURNING now returns a list
* #380: db_session should work with async functions
* #385: test fails with python3.6
* #386: release unlocked lock error in SQLite
* #390: TypeError: writable buffers are not hashable
* #398: add auto coversion of numpy numeric types
* #404: GAE local run detection
* Fix Flask compatibility: add support of LocalProxy object
* db_session(sql_debug=True) should log SQL commands also during db_session.__exit__()
* Fix duplicated table join in FROM clause
* Fix accessing global variables from hybrid methods and properties
* Fix m2m collection loading bug
* Fix composite index bug: stackoverflow.com/questions/53147694
* Fix MyEntity[obj.get_pk()] if pk is composite
* MySQL group_concat_max_len option set to max of 32bit platforms to avoid truncation
* Show all attribute options in show(Entity) call
* For nested db_session retry option should be ignored
* Fix py_json_unwrap
* Other minor fixes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment