mandrill webhook: incorrect signature with auth

[patroqueeet]

dear, Im getting Mandrill webhook called with incorrect signature error with a Bad Request response towards mandrill.

Sentry logs say about the err location:

filename	
'base.py'
lineno	
216
pathname	
'/home/onlinegv/eggs/Django-1.9.9-py2.7.egg/django/core/handlers/base.py'

I triple checked my webhook and api key. my total settings look like this:

        ANYMAIL =  {
            'MANDRILL_API_KEY': get_config_or_none(
                config,'Email', 'mandrill_api_key', 'SOMEKEY-A'),
            'WEBHOOK_AUTHORIZATION':'SOMEAUTH:SOMEPW',
            'MANDRILL_WEBHOOK_KEY': get_config_or_none(
                config, 'Email', 'mandrill_webhook_key', 'SOMEKEY'),
        }

what else should I check to investigate the failure?

[medmunds]

If your Django app doesn't know its own public hostname (because Django is behind a proxy that doesn't set X-Forwarded-Host, e.g.), you'll need to set MANDRILL_WEBHOOK_URL to the exact webhook url you entered into Mandrill's dashboard.

Without that setting, Mandrill is computing a signature for something like https://yoursite.example.com/anymail/mandrill/tracking/, but Anymail thinks it was called at something like http://127.0.0.1:8000/anymail/mandrill/tracking/ -- so the signatures don't match.

(I'm making a note to update that error message to include the url Anymail thinks it was called at.)

If that's not the issue, it would be helpful to know things like Anymail and Django versions, whether the webhook has ever worked (and if so, what changed since then).

You could also scan up the stack trace in Sentry to find where anymail.webhooks.mandrill...validate_request is raising the error -- Sentry may have captured more information there that would help your investigation. In particular, signed_data will start with the url Anymail thinks it was called at (followed by your webhook key and the request params that were signed).

[patroqueeet]

dear, the x-forwarded-host is properly set to the domain of the webhook. but I realized that in mandrill, the webhook url registered contains the authentication credentials (as of your documentation). if that is used for signing, then it won't work. what do you think?

also signed_data is not reported to sentry. only mandrill_events, request headers and env variables are recorded. there stack trace is also missing. this happens usually if the exception is catched somewhere else and a logger reports the failure. in the sentry logger name is django.security.AnymailWebhookValidationFailure. what is the best way to have an insigt into signed_data?

and no, I just switched to anymail and it was never successfull.

[patroqueeet]

Update: I disabled WEBHOOK_AUTHORIZATION setting, removed the httpauth from the mandrill registered url and tried again. but that failed again with a 400.

[patroqueeet]

another Update: I managed to print signed_data on the instance:

u'https://SUBDOMAIN.DOMAIN.com/webhooks/mail/mandrill/tracking/mandrill_events[{"event":"send","msg":{"ts":1365109999,"subject":"This an example webhook message","email":"example.webhook@mandrillapp.com","sender":"example.sender@mandrillapp.com","tags":["webhook-example"],"opens":[],"clicks":[],"state":"sent","metadata":{"user_id":111},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa","_version":"exampleaaaaaaaaaaaaaaa"},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa","ts":1483429533},{"event":"deferral","msg":{"ts":1365109999,"subject":"This an example webhook message","email":"example.webhook@mandrillapp.com","sender":"example.sender@mandrillapp.com","tags":["webhook-example"],"opens":[],"clicks":[],"state":"deferred","metadata":{"user_id":111},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa1","_version":"exampleaaaaaaaaaaaaaaa","smtp_events":[{"destination_ip":"127.0.0.1","diag":"451 4.3.5 Temporarily unavailable, try again later.","source_ip":"127.0.0.1","ts":1365111111,"type":"deferred","size":0}]},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa1","ts":1483429533},{"event":"hard_bounce","msg":{"ts":1365109999,"subject":"This an example webhook message","email":"example.webhook@mandrillapp.com","sender":"example.sender@mandrillapp.com","tags":["webhook-example"],"state":"bounced","metadata":{"user_id":111},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa2","_version":"exampleaaaaaaaaaaaaaaa","bounce_description":"bad_mailbox","bgtools_code":10,"diag":"smtp;550 5.1.1 The email account that you tried to reach does not exist. Please try double-checking the recipient\'s email address for typos or unnecessary spaces."},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa2","ts":1483429533},{"event":"soft_bounce","msg":{"ts":1365109999,"subject":"This an example webhook message","email":"example.webhook@mandrillapp.com","sender":"example.sender@mandrillapp.com","tags":["webhook-example"],"state":"soft-bounced","metadata":{"user_id":111},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa3","_version":"exampleaaaaaaaaaaaaaaa","bounce_description":"mailbox_full","bgtools_code":22,"diag":"smtp;552 5.2.2 Over Quota"},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa3","ts":1483429533},{"event":"open","msg":{"ts":1365109999,"subject":"This an example webhook message","email":"example.webhook@mandrillapp.com","sender":"example.sender@mandrillapp.com","tags":["webhook-example"],"opens":[{"ts":1365111111}],"clicks":[{"ts":1365111111,"url":"http:\\/\\/mandrill.com"}],"state":"sent","metadata":{"user_id":111},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa4","_version":"exampleaaaaaaaaaaaaaaa"},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa4","ip":"127.0.0.1","location":{"country_short":"US","country":"United States","region":"Oklahoma","city":"Oklahoma City","latitude":35.4675598145,"longitude":-97.5164337158,"postal_code":"73101","timezone":"-05:00"},"user_agent":"Mozilla\\/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.8) Gecko\\/20100317 Postbox\\/1.1.3","user_agent_parsed":{"type":"Email Client","ua_family":"Postbox","ua_name":"Postbox 1.1.3","ua_version":"1.1.3","ua_url":"http:\\/\\/www.postbox-inc.com\\/","ua_company":"Postbox, Inc.","ua_company_url":"http:\\/\\/www.postbox-inc.com\\/","ua_icon":"http:\\/\\/cdn.mandrill.com\\/img\\/email-client-icons\\/postbox.png","os_family":"OS X","os_name":"OS X 10.6 Snow Leopard","os_url":"http:\\/\\/www.apple.com\\/osx\\/","os_company":"Apple Computer, Inc.","os_company_url":"http:\\/\\/www.apple.com\\/","os_icon":"http:\\/\\/cdn.mandrill.com\\/img\\/email-client-icons\\/macosx.png","mobile":false},"ts":1483429533},{"event":"click","msg":{"ts":1365109999,"subject":"This an example webhook message","email":"example.webhook@mandrillapp.com","sender":"example.sender@mandrillapp.com","tags":["webhook-example"],"opens":[{"ts":1365111111}],"clicks":[{"ts":1365111111,"url":"http:\\/\\/mandrill.com"}],"state":"sent","metadata":{"user_id":111},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa5","_version":"exampleaaaaaaaaaaaaaaa"},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa5","ip":"127.0.0.1","location":{"country_short":"US","country":"United States","region":"Oklahoma","city":"Oklahoma City","latitude":35.4675598145,"longitude":-97.5164337158,"postal_code":"73101","timezone":"-05:00"},"user_agent":"Mozilla\\/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.8) Gecko\\/20100317 Postbox\\/1.1.3","user_agent_parsed":{"type":"Email Client","ua_family":"Postbox","ua_name":"Postbox 1.1.3","ua_version":"1.1.3","ua_url":"http:\\/\\/www.postbox-inc.com\\/","ua_company":"Postbox, Inc.","ua_company_url":"http:\\/\\/www.postbox-inc.com\\/","ua_icon":"http:\\/\\/cdn.mandrill.com\\/img\\/email-client-icons\\/postbox.png","os_family":"OS X","os_name":"OS X 10.6 Snow Leopard","os_url":"http:\\/\\/www.apple.com\\/osx\\/","os_company":"Apple Computer, Inc.","os_company_url":"http:\\/\\/www.apple.com\\/","os_icon":"http:\\/\\/cdn.mandrill.com\\/img\\/email-client-icons\\/macosx.png","mobile":false},"url":"http:\\/\\/mandrill.com","ts":1483429533},{"event":"spam","msg":{"ts":1365109999,"subject":"This an example webhook message","email":"example.webhook@mandrillapp.com","sender":"example.sender@mandrillapp.com","tags":["webhook-example"],"opens":[{"ts":1365111111}],"clicks":[{"ts":1365111111,"url":"http:\\/\\/mandrill.com"}],"state":"sent","metadata":{"user_id":111},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa6","_version":"exampleaaaaaaaaaaaaaaa"},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa6","ts":1483429533},{"event":"unsub","msg":{"ts":1365109999,"subject":"This an example webhook message","email":"example.webhook@mandrillapp.com","sender":"example.sender@mandrillapp.com","tags":["webhook-example"],"opens":[{"ts":1365111111}],"clicks":[{"ts":1365111111,"url":"http:\\/\\/mandrill.com"}],"state":"sent","metadata":{"user_id":111},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa7","_version":"exampleaaaaaaaaaaaaaaa"},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa7","ts":1483429533},{"event":"reject","msg":{"ts":1365109999,"subject":"This an example webhook message","email":"example.webhook@mandrillapp.com","sender":"example.sender@mandrillapp.com","tags":["webhook-example"],"opens":[],"clicks":[],"state":"rejected","metadata":{"user_id":111},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa8","_version":"exampleaaaaaaaaaaaaaaa"},"_id":"exampleaaaaaaaaaaaaaaaaaaaaaaaaa8","ts":1483429533},{"type":"blacklist","action":"add","reject":{"reason":"hard-bounce","detail":"Example detail","last_event_at":"2014-02-01 12:43:56","email":"example.webhook@mandrillapp.com","created_at":"2014-01-15 11:32:19","expires_at":"2020-04-02 12:09:18","expired":false,"subaccount":"example_subaccount","sender":"example.sender@mandrillapp.com"},"ts":1483429533},{"type":"blacklist","action":"change","reject":{"reason":"hard-bounce","detail":"Example detail","last_event_at":"2014-02-01 12:43:56","email":"example.webhook@mandrillapp.com","created_at":"2014-01-15 11:32:19","expires_at":"2020-04-02 12:09:18","expired":false,"subaccount":"example_subaccount","sender":"example.sender@mandrillapp.com"},"ts":1483429533},{"type":"blacklist","action":"remove","reject":{"reason":"hard-bounce","detail":"Example detail","last_event_at":"2014-02-01 12:43:56","email":"example.webhook@mandrillapp.com","created_at":"2014-01-15 11:32:19","expires_at":"2020-04-02 12:09:18","expired":false,"subaccount":"example_subaccount","sender":"example.sender@mandrillapp.com"},"ts":1483429533},{"type":"whitelist","action":"add","entry":{"email":"example.webhook@mandrillapp.com","detail":"example details","created_at":"2014-01-15 12:03:19"},"ts":1483429533},{"type":"whitelist","action":"remove","entry":{"email":"example.webhook@mandrillapp.com","detail":"example details","created_at":"2014-01-15 12:03:19"},"ts":1483429533}]'

does this look right?

I also printed the expected_signature and the signature. they were both different.

[patroqueeet]

while investigating I noticed: in sentry the mandrill_events contains http:\/\/www.apple.com\/ but inside the signed_data it is: http:\\/\\/www.apple.com\\/

[medmunds]

Ah -- I think you're correct about basic auth affecting how Mandrill calculates the signature. (But I can't really tell from Mandrill's docs, and I don't have access to Mandrill for testing.)

But yes, let's try to get it working without basic auth first. Where you've added print debugging in MandrillSignatureMixin.validate_request, let's also print some other things and verify they're correct:

• print self.basic_auth should be [] (because you disabled WEBHOOK_AUTHORIZATION)
• print self.webhook_key should be the exact current webhook key from the Mandrill webhook dashboard. (Mandrill may have re-generated this key when you changed the webhook url to remove basic auth, so it wouldn't hurt to check again.)
• print self.webhook_url should be 'https://SUBDOMAIN.DOMAIN.com/webhooks/mail/mandrill/tracking/' -- and that should be the exact url in Mandrill's webhooks dashboard, too -- please double-check 'https:' vs 'http:', and that both have a trailing '/'.
• print request.build_absolute_uri() ideally would be the same as self.webhook_url. (But if not, the difference would tell us why you're needing to set MANDRILL_WEBHOOK_URL.)

If none of those shows a problem, I'm afraid I'm a bit mystified.

[patroqueeet]

hey, thanks for the guidance. will instantly tomorrow morning. seen my comment about the escaping?

[medmunds]

I think the escaping difference is just an artifact of printing to the console.

[patroqueeet]

hey, self.webhook_url is None, but according to signed_data = self.webhook_url or request.build_absolute_uri() this should be fine.

with printing I get

auth ['AUTH:AUTH2']
hook key 'KEY'
hook url None
uri `https://SUBDOMAIN.DOMAIN.com/webhooks/mail/mandrill/tracking/`

I compared the url 20 times with mandrill and it looks fine. even with the trailing slash.

[patroqueeet]

alright I finally got it: when using the authentication mechanism the url for the signed_data is retrieved like this: https://github.com/anymail/django-anymail/blob/master/anymail/webhooks/mandrill.py#L49 which means it either is using the automatically created uri or the setting. I had the setting empty (as you documented it optional). but the created uri does not contain the authentication data. hence, exaclty as documented ( https://mandrill.zendesk.com/hc/en-us/articles/205583257-How-to-Authenticate-Webhook-Requests#generate-a-signature ) the uri used MUST contain the authentication data (even if mandrill is hiding it inside it's own UI).

to make it short adding the same URI as entered into mandrill form (including the authentication data) as a setting for MANDRILL_WEBHOOK_URL did the trick for me.

maybe it would make sense to have this documented, that if you are using the authentication, the setting would be mandatory.

[medmunds]

Ah, great. Thanks for tracking this down, and glad you were able to get it working.

I'm going to treat this as a bug in the Anymail's Mandrill webhook verification. If Anymail knows you're using basic auth (and it does!), it should factor that into the signature calculation.

For anyone else encountering this, until there's a fix in place, the workaround is to set MANDRILL_WEBHOOK_URL to include your WEBHOOK_AUTHORIZATION secret:

ANYMAIL = {
    "MANDRILL_API_KEY": "<your API key>",
    "WEBHOOK_AUTHORIZATION": "random:random",  # use same random:random below
    "MANDRILL_WEBHOOK_URL": "https://random:random@yoursite.example.com/anymail/mandrill/tracking/",
    # ...
}

[medmunds]

@patroqueeet you mentioned "authentication data (even if mandrill is hiding it inside it's own UI)"... does that mean the Mandrill webhooks settings is extracting the basic auth from the url and showing it in some other UI fields? (Does it have a separate place for webhook username/password?)

I'd like to clarify Anymail's docs if that's how Mandrill handles webhook basic auth. If you could let me know the exact labels for things in the Mandrill webhook page -- or post a screenshot (empty webhooks form or sensitive data removed) -- that would be really helpful.

Thanks.

[patroqueeet]

hey, current mandrill form to add a webhook: https://www.dropbox.com/s/ec96qromh6rne76/Screenshot%202017-01-24%2007.50.15.png?dl=0 and this is the display. looks like they altered it. when we talked here, the auth data was not reflected by the status display url.

btw. you are the first ever open source maintainer which reflected a ticket and came back to me with the goal to make his great library even better...

[medmunds]

Thanks, looks like Mandrill has renamed a couple of things since the last time I saw that screen. I'll update the docs.

I really appreciate all the effort you put into researching this issue. It's been tricky maintaining support for Mandrill without access to their API, and it would be absolutely impossible without the help of users like you.

[patroqueeet]

I'm working with them every single day. ping me whenever you need any insights...