Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Infinite loop in sqlquerycount for MySQL #3185

Closed
mattrobenolt opened this issue May 6, 2016 · 3 comments · Fixed by #3191
Closed

Infinite loop in sqlquerycount for MySQL #3185

mattrobenolt opened this issue May 6, 2016 · 3 comments · Fixed by #3191

Comments

@mattrobenolt
Copy link
Contributor

mattrobenolt commented May 6, 2016

May 05 14:36:41 sentry-1-001 sentry[39317]: File "/opt/sentry/venv/lib/python2.7/site-packages/sentry/debug/utils/patch_context.py", line 27, in wrapped 
May 05 14:36:41 sentry-1-001 sentry[39317]: return self.callback(self.func, *args, **kwargs) 
May 05 14:36:41 sentry-1-001 sentry[39317]: File "/opt/sentry/venv/lib/python2.7/site-packages/sentry/utils/performance/sqlquerycount.py", line 57, in cursor 
May 05 14:36:41 sentry-1-001 sentry[39317]: result = func(self, *args, **kwargs) 
May 05 14:36:41 sentry-1-001 sentry[39317]: File "/opt/sentry/venv/lib/python2.7/site-packages/sentry/debug/utils/patch_context.py", line 27, in wrapped 
May 05 14:36:41 sentry-1-001 sentry[39317]: return self.callback(self.func, *args, **kwargs) 
May 05 14:36:41 sentry-1-001 sentry[39317]: File "/opt/sentry/venv/lib/python2.7/site-packages/sentry/utils/performance/sqlquerycount.py", line 57, in cursor 
May 05 14:36:41 sentry-1-001 sentry[39317]: result = func(self, *args, **kwargs)

(patch_context.py)

22 def patch(self): 
23 func = getattr(self.target, self.attr) 
24 
25 def wrapped(*args, **kwargs): 
26 __traceback_hide__ = True # NOQA 
27 return self.callback(self.func, *args, **kwargs) 
28 
29 wrapped.__name__ = func.__name__ 
30 if hasattr(func, '__doc__'): 
31 wrapped.__doc__ = func.__doc__ 
32 if hasattr(func, '__module__'): 
33 wrapped.__module__ = func.__module__

(sqlquerycount.py)

53 
54 
55 def get_cursor_wrapper(state): 
56 def cursor(func, self, *args, **kwargs): 
57 result = func(self, *args, **kwargs) 
58 
59 return CursorWrapper(result, self, state) 
60 return cursor 
61
May 05 14:36:41 sentry-1-001 sentry[39317]: [ERROR] maximum recursion depth exceeded in __instancecheck__
May 05 14:36:41 sentry-1-001 sentry[39317]: Traceback (most recent call last):
May 05 14:36:41 sentry-1-001 sentry[39317]: File "/opt/sentry/venv/lib/python2.7/site-packages/sentry/options/store.py", line 165, in get_store
May 05 14:36:41 sentry-1-001 sentry[39317]: value = self.model.objects.get(key=key.name).value
May 05 14:36:41 sentry-1-001 sentry[39317]: File "/opt/sentry/venv/lib/python2.7/site-packages/sentry/../django/db/models/manager.py", line 151, in get
May 05 14:36:41 sentry-1-001 sentry[39317]: return self.get_queryset().get(*args, **kwargs)
May 05 14:36:41 sentry-1-001 sentry[39317]: File "/opt/sentry/venv/lib/python2.7/site-packages/sentry/../django/db/models/query.py", line 304, in get
May 05 14:36:41 sentry-1-001 sentry[39317]: num = len(clone)
May 05 14:36:41 sentry-1-001 sentry[39317]: File "/opt/sentry/venv/lib/python2.7/site-packages/sentry/../django/db/models/query.py", line 77, in __len__
May 05 14:36:41 sentry-1-001 sentry[39317]: self._fetch_all()
May 05 14:36:41 sentry-1-001 sentry[39317]: File "/opt/sentry/venv/lib/python2.7/site-packages/sentry/../django/db/models/query.py", line 857, in _fetch_all
May 05 14:36:41 sentry-1-001 sentry[39317]: self._result_cache = list(self.iterator())
May 05 14:36:41 sentry-1-001 sentry[39317]: File "/opt/sentry/venv/lib/python2.7/site-packages/sentry/../django/db/models/query.py", line 220, in iterator
May 05 14:36:41 sentry-1-001 sentry[39317]: for row in compiler.results_iter():
May 05 14:36:41 sentry-1-001 sentry[39317]: File "/opt/sentry/venv/lib/python2.7/site-packages/sentry/../django/db/models/sql/compiler.py", line 713, in results_iter
May 05 14:36:41 sentry-1-001 sentry[39317]: for rows in self.execute_sql(MULTI):
May 05 14:36:41 sentry-1-001 sentry[39317]: File "/opt/sentry/venv/lib/python2.7/site-packages/sentry/../django/db/models/sql/compiler.py", line 785, in execute_sql
May 05 14:36:41 sentry-1-001 sentry[39317]: cursor = self.connection.cursor()
May 05 14:36:41 sentry-1-001 sentry[39317]: File "/opt/sentry/venv/lib/python2.7/site-packages/sentry/debug/utils/patch_context.py", line 27, in wrapped
May 05 14:36:41 sentry-1-001 sentry[39317]: return self.callback(self.func, *args, **kwargs)
May 05 14:36:41 sentry-1-001 sentry[39317]: File "/opt/sentry/venv/lib/python2.7/site-packages/sentry/utils/performance/sqlquerycount.py", line 57, in cursor
May 05 14:36:41 sentry-1-001 sentry[39317]: result = func(self, *args, **kwargs)
May 05 14:36:41 sentry-1-001 sentry[39317]: File "/opt/sentry/venv/lib/python2.7/site-packages/sentry/debug/utils/patch_context.py", line 27, in wrapped
May 05 14:36:41 sentry-1-001 sentry[39317]: return self.callback(self.func, *args, **kwargs)
May 05 14:36:41 sentry-1-001 sentry[39317]: File "/opt/sentry/venv/lib/python2.7/site-packages/sentry/utils/performance/sqlquerycount.py", line 57, in cursor
May 05 14:36:41 sentry-1-001 sentry[39317]: result = func(self, *args, **kwargs)

@mitsuhiko @dcramer

@mitsuhiko
Copy link
Member

Looking at this a bit it seems like this is a MySQL unrelated issue. The patcher is currently not thread safe so what might happen is that two things attempt to patch and then it recursively calls itself instead of the original function. I need to verify that this is actually what happened there but it would appear that this is very much possible.

@nepeat
Copy link

nepeat commented May 8, 2016

For my Sentry setup, this same exception has been happening across the whole app. Here is one of the tracebacks for the breakages that have been happening for me. https://gist.github.com/nepeat/9dd29a8d96f801631aeb141d71893b26

@bsergean
Copy link

bsergean commented May 9, 2016

I locally applied #3191 (used the version of patch_context.py which has a lock) and it works fine for me now.

bsergean pushed a commit to bsergean/sentry that referenced this issue May 13, 2016
mjumbewu added a commit to CityOfPhiladelphia/phl-sentry that referenced this issue Jun 1, 2016
I was going to hold of on the upgrade to 8.4.1 because it doesn't have breadcrumbs, but there appears to be a breaking change in 8.4.0 that is affecting us (getsentry/sentry#3185), fixed by getsentry/sentry#3191.
@github-actions github-actions bot locked and limited conversation to collaborators Dec 18, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants