DisallowedHost errors should not trigger a Sentry error

[peterbe]

https://sentry.prod.mozaws.net/operations/normandy-prod/issues/5108824/

DisallowedHost: Invalid HTTP_HOST header: 'kite.zerodha.com'. You may need to add 'kite.zerodha.com' to ALLOWED_HOSTS.
  File "django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 93, in __call__
    response = self.process_request(request)
  File "django/middleware/common.py", line 56, in process_request
    host = request.get_host()
  File "django/http/request.py", line 105, in get_host
    raise DisallowedHost(msg)

DisallowedHost: Invalid HTTP_HOST header: 'kite.zerodha.com'. You may need to add 'kite.zerodha.com' to ALLOWED_HOSTS.
  File "django/core/handlers/exception.py", line 99, in get_exception_response
    response = callback(request, **dict(param_dict, exception=exception))
  File "django/utils/decorators.py", line 138, in _wrapped_view
    result = middleware.process_view(request, view_func, args, kwargs)
  File "django/middleware/csrf.py", line 266, in process_view
    good_referer = request.get_host()
  File "django/http/request.py", line 105, in get_host
    raise DisallowedHost(msg)

DisallowedHost: Invalid HTTP_HOST header: 'kite.zerodha.com'. You may need to add 'kite.zerodha.com' to ALLOWED_HOSTS.

[peterbe]

First of all, why is it even passed into Django? Shouldn't this host header check be done in Nginx or the load balancer or something? @sciurus ??

Incidentally, I noticed that on symbols.mozilla.org the same happens. The request appears to make it through Nginx into Django.

▶ curl -v -H 'Host: example.com' https://symbols.mozilla.org
...
> GET / HTTP/1.1
> Host: example.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 400 Bad Request
...

{"error": "Invalid HTTP_HOST header: 'example.com'. You may need to add 'example.com' to ALLOWED_HOSTS."}%

But at least it doesn't send anything to Sentry.

Second of all, it's a weird exception. Or is it just Sentry being weird?
Sentry reports it as POST request to https://kite.zerodha.com/api/orders/regular. The breadcrumbs (which I often ignore) mention GET /api/v1/classify_client/ and GET /__lbheartbeat__ and POST /api/orders/regular. Watt?!

[peterbe]

@sciurus We can move this to Bugzilla if you prefer.

[sciurus]

We don't have a standard practice of blocking based on the host header in nginx or the load balancer. Currently, applications where it matters can use their framework's preferred method of blocking access, e.g. ALLOWED_HOSTS for Django.

In Normandy in AWS we did add configuration to block unknown hosts in https://github.com/mozilla-services/cloudops-deployment/pull/1524/ . This was essentially "belt and suspenders" to deal with the cache poisoning security issue. Since ALLOWED_HOSTS is working, we just went back to our standard nginx configuration for Normandy in GCP. Presumably you're seeing this sentry error now because we cut a lot of Normandy's traffic over to GCP yesterday.

I think the idea of continuing to block within nginx is interesting. I could add it back just to Normandy, but @milescrabill is currently working on the next generation of our team's standard nginx configuration so it might be better to raise the idea with him.

For now though, I think that things are working as expected, except for the errors going to Sentry. You're in a better position to investigate and this that than I am.

[peterbe]

I can definitely tackle the not-sending-DisallowedHost-exceptions-to-Sentry problem. Mind you, I think we might switch from python-raven to sentry-sdk so that might change things.

And, as you said, this is working as expected in Django (apart from the noisy Sentry pings) which is maybe not a terrible thing. In our projects, we have it as a "policy" that Nginx should be rather "dumb" when it comes to serving static assets and allow Django (technically Whitenoise) to do that part.

[peterbe]

I'm struggling to solve this. The right place to add a list of ignore_exceptions is here:

normandy/normandy/settings.py

Lines 283 to 286 in 2dda0b0

	return {
	"dsn": values.URLValue(None, environ_name="RAVEN_CONFIG_DSN"),
	"release": values.Value(version, environ_name="RAVEN_CONFIG_RELEASE"),
	}

I did test it by adding: "ignore_exceptions": [NameError], to that. Before and after I put in a raise NameError('testing') and it did appear in Sentry but ceased once I added that line. (I used the symbols-dev Sentry DSN)

Two questions I don't know the answer to...

1. When I start Django like this: DJANGO_CONFIGURATION=ProductionInsecure DJANGO_ALLOWED_HOSTS=localhost DJANGO_DEBUG=False DJANGO_RAVEN_CONFIG_RELEASE=peterbetesting DJANGO_RAVEN_CONFIG_DSN=https://7f6...@sentry.prod.mozaws.net/196 ./manage.py runserver and then do curl -v -H 'Host: example.com' http://localhost:8000 I do get the 400 Bad Request but it's not sending to Sentry. I poked around in the Django code base (in the virtualenv) and sure enough the raise DisallowedHost(...) exception does happen.

2. Suppose I add a list of ignored exceptions, e.g. "ignore_exceptions": ["django.exceptions.http.DisallowedHost"], what were the other defaults? Any?

[mythmon]

I don't understand how (1) is a question.

For (2), the default is nothing. I found this by grepping the source of Raven for ignore_exceptions. I don't follow all of the logic there, but it's definitely setting an empty default.

I think is a good way to solve the issue.

[peterbe]

Regarding (1), what I meant to ask was; How do you reproduce causing a Sentry send on a DisallowedHost exception?

[mythmon]

Ah, I see now.

I tried to get Sentry to send a DisallowedHost exception locally as well, using similar steps. I got a 400 bad request to happen, and I saw a log line about it, but there wasn't anything sent to my mock sentry (actually a requestbin).

@sciurus Do you know if there is something special we need to be doing to get Raven to actually send errors to Sentry?

[peterbe]

Glad it's not just me! I do wonder if it's related to logging. Perhaps it's instructed somewhere to swallow certain exceptions. If so, it's sad that I can't get my local env to simulate how we do things in production.

[peterbe]

Stale. I'm tempted to go ahead and just do the RAVEN_CONFIG trick mentioned above. These errors are still bombarding our Sentry.

[jaredlockhart]

We'll just mark these as muted in sentry rather than adding custom code.