RecentlyUsedContainer is not threadsafe #7

Closed
christer opened this Issue Nov 4, 2011 · 4 comments

Projects

None yet

2 participants

christer commented Nov 4, 2011

I'm testing 1.0.1

As noted in the code, RecentlyUsedContainer is not threadsafe.

I'm getting: RuntimeError: deque mutated during iteration

Stacktrace:

  File "/home/chrsjo/src/arkenutils_omkomp/arken/hcap.py", line 153, in _request
    return self._pool.request(*args, **kwargs)
  File ".../urllib3-1.0.1-py2.7.egg/urllib3/request.py", line 65, in request
    **urlopen_kw)
  File ".../urllib3-1.0.1-py2.7.egg/urllib3/request.py", line 78, in request_encode_url
    return self.urlopen(method, url, **urlopen_kw)
  File ".../urllib3-1.0.1-py2.7.egg/urllib3/poolmanager.py", line 107, in urlopen
    conn = self.connection_from_url(url)
  File ".../urllib3-1.0.1-py2.7.egg/urllib3/poolmanager.py", line 98, in connection_from_url
    return self.connection_from_host(host, port=port, scheme=scheme)
  File ".../urllib3-1.0.1-py2.7.egg/urllib3/poolmanager.py", line 73, in connection_from_host
    pool = self.pools.get(pool_key)
  File "/home/chrsjo/.virtualenvs/arkenutils_omkomp/lib/python2.7/_abcoll.py", line 342, in get 
    return self[key]
  File ".../urllib3-1.0.1-py2.7.egg/urllib3/_collections.py", line 96, in __getitem__
    self._prune_invalidated_entries()
  File ".../urllib3-1.0.1-py2.7.egg/urllib3/_collections.py", line 77, in _prune_invalidated_entries
    self.access_log = deque(e for e in self.access_log if e.is_valid)
  File ".../urllib3-1.0.1-py2.7.egg/urllib3/_collections.py", line 77, in <genexpr>
    self.access_log = deque(e for e in self.access_log if e.is_valid)
RuntimeError: deque mutated during iteration

christer commented Nov 4, 2011

I quickly added some locking
christer@5923463

Owner
shazow commented Nov 4, 2011

Perfect, thank you @christer. I've been meaning to do this and I appreciate you pushing this through.

Looks good at first glance. I'll do a review later today and merge it in.

Owner
shazow commented Nov 4, 2011

I merged and ported over your changes to <Py26 (acquire/release instead of with) and made some other minor improvements here:

4a68690

I'm wondering whether we should expand the scope of the mutex to the entire methods rather than just that one deque. Hmm. Or perhaps I'll write some of the surrounding code a bit more defensively.

Owner
shazow commented Nov 4, 2011

Alright I reviewed the code and I feel fairly confident about it. I'll do a 1.0.2 release later today with these changes included.

@shazow shazow closed this Nov 4, 2011
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment