Sentry with eventlet #1036

Randomneo · 2021-03-01T13:39:32Z

Environment

self-hosted sentry 9.0.0

python 3.9/3.7
sentry_sdk version: 0.20.3
eventlet version: 0.30.1
greenlet version: 1.0.0

Steps to Reproduce

import eventlet
import sentry_sdk

eventlet.monkey_patch()

sentry_sdk.init('<sentry_sdk_dsn>')
sentry_sdk.capture_message('aoe')

Expected Result

getting message in sentry

Actual Result

Nothing happens. No errors thrown. No clear documentation for that problem. Here issue for eventlet.

If sentry_sdk can't be used with eventlet and there no possible way to combine them it should be in documentation.
Please correct me if i'm missing something.

The text was updated successfully, but these errors were encountered:

untitaker · 2021-03-01T13:42:20Z

Can you add sentry_sdk.flush() as last line?

Randomneo · 2021-03-01T14:02:47Z

Same. Nothing happens. Without flush I got.

Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

Program ends by itself instantly.

With flush. No output and no message in sentry

Randomneo · 2021-03-01T14:05:36Z

Problem definitely in eventlet.monkey_patch() if that line is commented everything works fine.

temoto · 2021-03-01T14:40:26Z

Just for record, copying workaround from our tracker.

Calling monkey_patch before even importing sentry_sdk works.

import eventlet
eventlet.monkey_patch()

import sentry_sdk
...

I understand it's well below acceptable and yet don't have a better solution right now.

Successful fix would start with analyzing what side effects are produced by import sentry_sdk

Randomneo · 2021-03-01T15:36:45Z

@temoto Thanks alot. I had no idea that import sentry_sdk was the problem :)

untitaker · 2021-03-01T15:52:22Z

only work on import is this:

sentry-python/sentry_sdk/utils.py

Line 840 in 37105d9

if not _is_contextvars_broken():

However, flush etc should not really break regardless of whether contextvars or thread locals are used. So I think it's more related to threading patches that would affect sentry_sdk.worker.

untitaker · 2021-03-01T15:53:31Z

@Randomneo can you check Hub.current.client after init? Maybe locals are broken in a really bad way.

Randomneo · 2021-03-02T07:41:12Z

@untitaker I not quite get how I supposed to check it. But this code outputs:

import eventlet
import sentry_sdk
from sentry_sdk import Hub

eventlet.monkey_patch()

sentry_sdk.init('dsn')
print(Hub.current.client)
sentry_sdk.capture_message('aoe')

Output:

<sentry_sdk.client._Client object at 0x7f5d8354edf0>                                                                                                                                                             
Sentry is attempting to send 0 pending error messages                                                                                                                                                            
Waiting up to 2 seconds
Press Ctrl-C to quit

It also easy to reproduce on your machine using code sample in issue description. I'm using latest stable versions of packages

untitaker · 2021-03-03T11:04:54Z

I'm getting the following recursionerror in addition when it's broken. As you said, if I move the SDK import after the monkeypatch, the issue disappears. You need to add debug=True to the init() call to see the stacktrace.

Traceback (most recent call last):
  File "/Users/untitaker/projects/sentry-python/sentry_sdk/transport.py", line 326, in send_event_wrapper
    self._send_event(event)
  File "/Users/untitaker/projects/sentry-python/sentry_sdk/transport.py", line 232, in _send_event
    self._send_request(
  File "/Users/untitaker/projects/sentry-python/sentry_sdk/transport.py", line 176, in _send_request
    response = self._pool.request(
  File "/Users/untitaker/projects/sentry-python/.tox/py3.9-flask-dev/lib/python3.9/site-packages/urllib3/request.py", line 78, in request
    return self.request_encode_body(
  File "/Users/untitaker/projects/sentry-python/.tox/py3.9-flask-dev/lib/python3.9/site-packages/urllib3/request.py", line 170, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/Users/untitaker/projects/sentry-python/.tox/py3.9-flask-dev/lib/python3.9/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/untitaker/projects/sentry-python/.tox/py3.9-flask-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/Users/untitaker/projects/sentry-python/.tox/py3.9-flask-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/Users/untitaker/projects/sentry-python/.tox/py3.9-flask-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/Users/untitaker/projects/sentry-python/.tox/py3.9-flask-dev/lib/python3.9/site-packages/urllib3/connection.py", line 392, in connect
    self.ssl_context = create_urllib3_context(
  File "/Users/untitaker/projects/sentry-python/.tox/py3.9-flask-dev/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 303, in create_urllib3_context
    context.options |= options
  File "/Users/untitaker/.pyenv/versions/3.9.0/lib/python3.9/ssl.py", line 602, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/Users/untitaker/.pyenv/versions/3.9.0/lib/python3.9/ssl.py", line 602, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/Users/untitaker/.pyenv/versions/3.9.0/lib/python3.9/ssl.py", line 602, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  [Previous line repeated 479 more times]
RecursionError: maximum recursion depth exceeded

In addition, the issue can be reproduced without Sentry. The urllib3 usage is straight from their README:

import eventlet

import urllib3

eventlet.monkey_patch()
http = urllib3.PoolManager()

r = http.request('GET', 'https://httpbin.org/robots.txt')

This will crash in the same way.

temoto · 2021-03-03T11:51:45Z

@untitaker

I'm getting the following recursionerror in addition when it's broken. As you said, if I move the SDK import after the monkeypatch, the issue disappears. You need to add debug=True to the init() call to see the stacktrace.

This should be fixed in just released eventlet v0.30.2 but (AFAIK) is not related to current problem.

My two cents to checking Hub.current.client:

# test.py
import sentry_sdk
sentry_sdk.init('http://1234@foo/1')
print("[before eventlet] Hub.current.client:",sentry_sdk.Hub.current.client)

import eventlet
eventlet.monkey_patch()
print("[ after eventlet] Hub.current.client:", sentry_sdk.Hub.current.client)

[before eventlet] Hub.current.client: <sentry_sdk.client._Client object at 0x10b2b6198>
[ after eventlet] Hub.current.client: <sentry_sdk.client._Client object at 0x10b2b6198>

At the very least, it refers to same object after patching, so that's something about thread locals. Still maybe thread locals are messed up in another place.

untitaker · 2021-03-03T12:11:22Z

0.30.2 exhibits the same problem for me. You need to both send an event via the SDK and have debug=True added

github-actions · 2021-12-23T15:04:11Z

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

antonpirker · 2022-02-21T11:30:36Z

Quick update: I think this is a important issue to improve sentry with async Python. I will try to review and test this.

untitaker added the Type: Bug Something isn't working label Mar 1, 2021

guiscaranse mentioned this issue Nov 18, 2021

Patch eventlet under Sentry SDK #1258

Closed

github-actions bot added the Status: Stale label Dec 23, 2021

github-actions bot closed this as completed Dec 30, 2021

sl0thentr0py reopened this Jan 21, 2022

sl0thentr0py added Status: Backlog and removed Status: Stale labels Jan 21, 2022

antonpirker added the Integration: gevent/eventlet Green threads and the sorts. label Feb 21, 2022

hubertdeng123 removed the Status: Backlog label Jul 25, 2023

antonpirker added the triaged Has been looked at recently during old issue triage label Aug 28, 2023

szokeasaurusrex mentioned this issue Oct 25, 2023

Patch eventlet under Sentry SDK #2464

Merged

szokeasaurusrex self-assigned this Oct 25, 2023

szokeasaurusrex closed this as completed in #2464 Oct 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sentry with eventlet #1036

Sentry with eventlet #1036

Randomneo commented Mar 1, 2021

untitaker commented Mar 1, 2021

Randomneo commented Mar 1, 2021

Randomneo commented Mar 1, 2021

temoto commented Mar 1, 2021

Randomneo commented Mar 1, 2021

untitaker commented Mar 1, 2021 •

edited

untitaker commented Mar 1, 2021

Randomneo commented Mar 2, 2021

untitaker commented Mar 3, 2021 •

edited

temoto commented Mar 3, 2021 •

edited

untitaker commented Mar 3, 2021 •

edited

github-actions bot commented Dec 23, 2021

antonpirker commented Feb 21, 2022

Sentry with eventlet #1036

Sentry with eventlet #1036

Comments

Randomneo commented Mar 1, 2021

Environment

Steps to Reproduce

Expected Result

Actual Result

untitaker commented Mar 1, 2021

Randomneo commented Mar 1, 2021

Randomneo commented Mar 1, 2021

temoto commented Mar 1, 2021

Randomneo commented Mar 1, 2021

untitaker commented Mar 1, 2021 • edited

untitaker commented Mar 1, 2021

Randomneo commented Mar 2, 2021

untitaker commented Mar 3, 2021 • edited

temoto commented Mar 3, 2021 • edited

untitaker commented Mar 3, 2021 • edited

github-actions bot commented Dec 23, 2021

antonpirker commented Feb 21, 2022

untitaker commented Mar 1, 2021 •

edited

untitaker commented Mar 3, 2021 •

edited

temoto commented Mar 3, 2021 •

edited

untitaker commented Mar 3, 2021 •

edited