No IPv4 fallback and no error reporting on failed connection

[malexmave]

If you self-host a sentry instance and have a broken IPv6 setup on the server (i.e., you have an AAAA record set up for the domain, but it points at the wrong IP), but a working IPv4 setup, sentry-python will attempt to deliver the issue via IPv6 (if the network supports it), fail, and not retry with IPv4.

If this isn't intended behavior, I would recommend implementing an IPv4 fallback. At a minimum, it would be nice if the system would report an error if the connection failed - even with debug enabled, the log does not indicate that the connection failed, and is more or less indistinguishable from a log where the reporting worked.

In my eyes, an IPv4 fallback would be desirable either way, because some places have broken IPv6 networks (they provide v6 IPs, but do not let them access the internet). This is another scenario where the robustness of reporting would be improved with a v4 fallback.

Steps to reproduce:

• Set up Sentry with working A record, but AAAA record pointing at an incorrect IP
• Run the following code:

import sentry_sdk

sentry_sdk.init(YOUR_DSN, debug=True)

# Produce an exception:
1/0

[untitaker]

Just for clarity, what does "an incorrect IP" mean here? One where connecting times out? I am am puzzled as to the lack of error.

Do you get the same issue installing and importing requests in your environment and issuing a request against that server?

[malexmave]

Just double-checked the exact setup: I had it set up to point at an IPv6 address that accepts connections on :80 and :443 (try it yourself: 2a01:4f8:d1:3aa1::666). That is the IP of a firewall that proxies requests using HAProxy on IPv4, but not on IPv6.

I currently don't have time to try it out with requests, I'll try in a few hours if it is still relevant.

[malexmave]

Update: Interestingly, just using requests works. I am not sure if that is due to an integrated fallback mechanism, or what is going on. If you want to play around with it, you can try making requests to the domain https://maass.xyz, which currently has the same DNS records that the sentry instance used to have.

If necessary, I can restore sentry to the failing DNS setup for a day and provide you with an account and project so you can play with it (you can find my contact details on the domain linked above if you don't want to post your eMail address publicly).

[untitaker]

I'm able to fetch from that URL with both requests and raw urllib3. A test account would be great! My email is markus ÄT sentry.io

[malexmave]

You should have received an invite, and I also just updated the DNS to once again use the settings that created these issues for me. I also created a project for you in my sentry to play with. If you cannot reproduce with external machines, let me know - the issues occured when machines behind the same firewall attempted to communicate with sentry, so the situation may be more complex than a simple IPv6-to-4 fallback (although removing the AAAA record has fixed the issues, so it has to be something related to that).

[untitaker]

WFM, and I am pretty sure I have no working IPv6 🤷‍♂️ https://sentry.maass.xyz/sentry/sdk-testing/issues/7341/

idk what's going on tbh. Might try different machines in the same network.

[untitaker]

btw urllib3 already has an ipv6 fallback for this, from what I can tell by looking through their issues.

[malexmave]

I'll test it again this evening and will post the most detailed logs I can get my hands on.

[malexmave]

and I am pretty sure I have no working IPv6

Wait, you have no working IPv6? In that case, it would be expected to work, as it only uses v4 then.

[malexmave]

 [sentry] DEBUG: Setting up integrations (with default = True)
 [sentry] DEBUG: Setting up previously not enabled integration modules
 [sentry] DEBUG: Setting up previously not enabled integration threading
 [sentry] DEBUG: Setting up previously not enabled integration dedupe
 [sentry] DEBUG: Setting up previously not enabled integration stdlib
 [sentry] DEBUG: Setting up previously not enabled integration logging
 [sentry] DEBUG: Setting up previously not enabled integration argv
 [sentry] DEBUG: Setting up previously not enabled integration atexit
 [sentry] DEBUG: Setting up previously not enabled integration excepthook
 [sentry] DEBUG: Enabling integration modules
 [sentry] DEBUG: Enabling integration threading
 [sentry] DEBUG: Enabling integration dedupe
 [sentry] DEBUG: Enabling integration stdlib
 [sentry] DEBUG: Enabling integration logging
 [sentry] DEBUG: Enabling integration argv
 [sentry] DEBUG: Enabling integration atexit
 [sentry] DEBUG: Enabling integration excepthook
Traceback (most recent call last):
  File "/archiv/sentrytest.py", line 5, in <module>
 [sentry] DEBUG: Sending error event [73a9fa939f164d1f91525a7c49e15670] to sentry.maass.xyz project:10
    1/0
ZeroDivisionError: division by zero
 [sentry] DEBUG: atexit: got shutdown signal
 [sentry] DEBUG: atexit: shutting down client
 [sentry] DEBUG: Flushing HTTP transport
 [sentry] DEBUG: background worker got flush request
 [sentry] DEBUG: 0 event(s) pending on flush
Sentry is attempting to send 0 pending error messages
Waiting up to 2.0 seconds
Press Ctrl-C to quit
 [sentry] DEBUG: background worker flushed
 [sentry] DEBUG: Killing HTTP transport
 [sentry] DEBUG: background worker got kill request

This is the log I get from the code in the original issue using Python 3.5.2. As you can see in sentry, no issue was created. The VM the code was running on has an IPv4 and IPv6 connection to the outside world, and both work (i.e., ping6 heise.de gets responses).

$ pip freeze
certifi==2018.11.29
pkg-resources==0.0.0
sentry-sdk==0.7.6
urllib3==1.24.1

If you know a way to make urllib3 output useful logs, let me know.

[untitaker]

I just sent an event on a remote server with working ipv6 and ipv4 (same urllib3 version). Please check curl -6 -v http://sentry.maass.xyz/ and similar things, but I don't think this issue is unique to the sentry sdk or python.

[malexmave]

$ curl -6 -v http://sentry.maass.xyz/
*   Trying 2a01:4f8:d1:3aa1::666...
* connect to 2a01:4f8:d1:3aa1::666 port 80 failed: Keine Route zum Zielrechner
* Failed to connect to sentry.maass.xyz port 80: Keine Route zum Zielrechner
* Closing connection 0
curl: (7) Failed to connect to sentry.maass.xyz port 80: Keine Route zum Zielrechner

Hm, then it seems to be related to how our firewall handles internal connections, if it works from outside (don't have an IPv6-speaking device ready outside of that NAT). I guess this makes it less of a sentry problem than a configuration problem on our end.

However, I still think that the sentry logs in debug mode should reflect that the message could not be sent, and ideally contain the error that occured. I'd be happy to be a guinea pig to test any code that adds this functionality on my broken setup.

[untitaker]

@malexmave I just got reminded by the weekly report that your Sentry instance sends: Is this still an issue for you?

[malexmave]

Since I removed the v6 record again after this test, currently everything runs fine, but since I did not update anything, I would assume it would still break again if I re-added the AAAA record. As I said, I'd be happy to test any patches that make the fallback to IPv4 more robust / improve the logging of errors in this setup.

[untitaker]

@malexmave cool. fwiw I don't have a way to repro this... I tried this on uberspace.de which supposedly has ipv6 support all the way, but didn't find issues.

[untitaker]

Do you mind if I just close this? It seems not very actionable to me in its current state. A missing error during debug mode would be a bug but then I don't know how to reproduce the part where urllib3 swallows all errors.

[malexmave]

The setup is a bit more complicated than simple v6 support. The situation is the following: Sentry is running in a VM in a NATted network behind a firewall. The firewall uses HAProxy to terminate TLS connections and forward the contents to the correct VM. If connections come from outside the VM NAT network, they work fine on v4 and v6. If they come from a different VM inside the NATted network, they are lost at the firewall due to a misconfiguration (as I learned when I talked to the server admin).

So, the issue here is less the fact that in this specific setup, the messages don't get through, and more the fact that no log message indicates that the connection failed.

[malexmave]

If you want to play with this specific setup, I could probably get you a VM with SSH access. If you think that this is too specific to warrant your time, or if you suspect that the error lies in urllib and not sentry, I'd also be fine with closing this issue without a fix.

[untitaker]

I'm trying to gauge how many people are affected by this issue. Let's keep this open for now.

One last question: Could you try installing different versions of urllib3? You mentioned earlier that requests works fine. Does this also mean that it behaves/errors as expected when neither IPv6 nor IPv4 work in your specific (mis)configuration? I believe there might be discrepancies between the version used by requests vs the one used by the SDK. For example one might use a urllib3 provided by your Linux distro (distros are often happy to violently patch things) while the other installs from PyPI.

[sreedharbukya]

I have a similar issue with the latest version of sentry_sdk with self-managed sentry service.

sentry-sdk==0.9.0
urllib3==1.25.3

This is the code which initializes my sentry. It works good with sentry.io.
but doesn't work with my managed sentry.[ABC].io domain.

Wondering what could be the issue. duplicate issue #399

def before_send(event, hint):
	log.debug("Before Sending the Event, {} and Hint : {}".format(repr(event), repr(hint)))
	return event


def init_sentry():
	sentry_logging = LoggingIntegration(level=logging.INFO, event_level=logging.ERROR)

	sentry_client = sentry_sdk.init(dsn=SENTRY_DSN, integrations=[sentry_logging], environment=ENVIRONMENT,
									before_send=before_send)
	log.info("Sentry Initialized")
	return sentry_client


sentry_client = init_sentry()

[untitaker]

@sreedharbukya this issue is about something very specific in the HTTP library we're using, so I wouldn't call this the same issue just because the symptoms are

[github-actions]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[antonpirker]

As this is a very old issue, and it seems to be very specific to a certain network setup and thus not affecting a lot of users, I think it is save to close this issue.

If you come across this issue and what to help fixing this, please create a new Issue and link to this one. Thanks!