RuntimeError: Set changed size during iteration (utils.py clear_all) #101

Closed
karlcz opened this Issue Sep 23, 2011 · 2 comments

Comments

Projects
None yet
3 participants

karlcz commented Sep 23, 2011

We have a pretty complicated web.py app we've been running on CentOS 5.x with web.py packaged as python-webpy-0.32-3.el5 from Fedora EPEL, running under mod_wsgi packaged as mod_wsgi-3.2-1.el5.

I tried testing it on CentOS 6.0 using mod_wsgi-3.2-1.el6.x86_64 and web.py-0.36 installed manually, but ran into trouble. This application runs with multiple processes and threads per process using the daemon mode of mod_wsgi. During testing with concurrent requests (multiple client connections with pipelined requests), I have seen this internal server error several times on 0.36 which we never saw on our older systems:

mod_wsgi (pid=28570): Exception occurred processing WSGI script '/usr/lib/python2.6/site-packages/tagfiler/wsgi/tagfiler.wsgi'.
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/web/application.py", line 291, in cleanup
self._cleanup()
File "/usr/lib/python2.6/site-packages/web/application.py", line 110, in _cleanup
utils.ThreadedDict.clear_all()
File "/usr/lib/python2.6/site-packages/web/utils.py", line 1202, in clear_all
for t in ThreadedDict._instances:
RuntimeError: Set changed size during iteration

Does this seem like a simple race-condition in web.py? Ours is a tangle of an app, but I am struggling to see how our code could be affecting this low-level bit of the web.py app lifecycle.

We do have some global state (dictionaries) shared at the module level for caching of common database results, including strings and some web.Storage objects. Has the relationship between the app and the web.Storage objects been linked more deeply between 0.32 and 0.36?

karlcz commented Sep 24, 2011

I tried a simple patch to web/utils.py which seems to resolve this issue for us. Just wrap the instances in a list to take a snapshot rather than using an iterator, to avoid having the membership change during the loop iterations:

def clear_all():
    """Clears all ThreadedDict instances.                                                                                                                                                       
    """
    for t in list(ThreadedDict._instances):
        t.clear()

I also ran into this problem. I'm just starting to develop an app. And I was using ab to evaluate a database race problem. At -c 100 concurrency, 40% of requests fails with the above error on my macbook. The patch by karlcz did address this problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment