Possibility to use @db_session w/ few concurrent ``multiprocessing.Process`` #118

Closed
surpr1ze opened this Issue Mar 25, 2015 · 3 comments

Projects

None yet

2 participants

@surpr1ze

When I start few processes, which are using the same @db_session, at the same time a get the following exception:

    record = get((row.id, row.user_id) for row in Record \
  File "<auto generated="" wrapper="" of="" get()="" function="">", line 2, in get
  File "/root/Projects/dns-stat-collector/.venv/lib/python2.7/site-packages/pony/utils.py", line 88, in cut_traceback
    return func(*args, **kwargs)
  File "/root/Projects/dns-stat-collector/.venv/lib/python2.7/site-packages/pony/orm/core.py", line 4260, in get
    return make_query(args, frame_depth=3).get()
  File "<auto generated="" wrapper="" of="" get()="" function="">", line 2, in get
  File "/root/Projects/dns-stat-collector/.venv/lib/python2.7/site-packages/pony/utils.py", line 88, in cut_traceback
    return func(*args, **kwargs)
  File "/root/Projects/dns-stat-collector/.venv/lib/python2.7/site-packages/pony/orm/core.py", line 4531, in get
    objects = query[:2]
  File "<auto generated="" wrapper="" of="" __getitem__()="" function="">", line 2, in __getitem__
  File "/root/Projects/dns-stat-collector/.venv/lib/python2.7/site-packages/pony/utils.py", line 88, in cut_traceback
    return func(*args, **kwargs)
  File "/root/Projects/dns-stat-collector/.venv/lib/python2.7/site-packages/pony/orm/core.py", line 4721, in __getitem__
    return query._fetch(range=(start, stop))
  File "/root/Projects/dns-stat-collector/.venv/lib/python2.7/site-packages/pony/orm/core.py", line 4433, in _fetch
    for sql_row in cursor.fetchall() ]
ProgrammingError: no results to fetch

Thanks!

@surpr1ze surpr1ze closed this Mar 30, 2015
@surpr1ze

Need to disconnect() from db, before spawning new process

@kozlovsky
Contributor

I'm glad that db.disconnect() works for you. But I think that ideally Pony should be able to determine by itself that the stored connection was created in another process. This way Pony will ensure that the user will never experience any problem even if he doesn't know anything about possible multiprocessing issues.

In order to do this we need to store process id along with the connection, and then, when getting the connection from the pool, we need to compare stored process id with the current pid value and discard connection if the pid value differs.

@kozlovsky kozlovsky reopened this Mar 30, 2015
@kozlovsky kozlovsky added a commit that closed this issue May 19, 2015
@kozlovsky kozlovsky Fixed #118: Pony should reconnect if previous connection was created …
…before process was forked
6089b32
@kozlovsky kozlovsky closed this in 6089b32 May 19, 2015
@kozlovsky
Contributor

Now this problem should be completely solved. It is still recommended to call db.disconnect() before forking, but if a programmer forgets to do it, Pony should notice this and open a new connection instead of using the old one.

The only restriction remaining is that the programmer should not to do forking while a db_session is active, but it looks like a common sense.

@kozlovsky kozlovsky added a commit that referenced this issue Jan 11, 2016
@kozlovsky kozlovsky Pony ORM Release 0.6.2 (2015-01-11)
The documentation was moved from this repo to a separate one at https://github.com/ponyorm/pony-doc
The compiled version can be found at https://docs.ponyorm.com

# New features

* Python 3.5 support
* #132, #145: raw_sql() function was added
* #126: Ability to use @db_session with generator functions
* #116: Add support to select by UUID
* Ability to get string SQL statement using the Query.get_sql() method
* New function delete(gen) and Query.delete(bulk=False)
* Now it is possible to override Entity.__init__() and declare custom entity methods

# Backward incompatible changes

* Normalizing table names for symmetric relationships
* Autostrip - automatically remove leading and trailing characters

# Bugfixes

* #87: Pony fails with pymysql installed as MySQLdb
* #118: Pony should reconnect if previous connection was created before process was forked
* #121: Unable to update value of unique attribute
* #122: AssertionError when changing part of a composite key
* #127: a workaround for incorrect pysqlite locking behavior
* #136: Cascade delete does not work correctly for one-to-one relationships
* #141, #143: remove restriction on adding new methods to entities
* #142: Entity.select_random() AssertionError
* #147: Add 'atom_expr' symbol handling for Python 3.5 grammar
ab09f64
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment