Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sending mails fails if SMTP server advertises CRAM-MD5 authentication #1637

Closed
XtraLarge opened this issue May 21, 2019 · 7 comments

Comments

Projects
None yet
2 participants
@XtraLarge
Copy link

commented May 21, 2019

Hello,

I've installed privacyidea 3.0 on a debian stretch in an virtualenv with python 2.7. There were a problem so the smtp module did not works as exprected.

If you try to use a smtp server privacyidea throws the exception: "character mapping must return integer, None or unicode"

Here's the stack:

[2019-05-21 09:59:31,017] ERROR in app: Exception on /smtpserver/send_test_email [POST]
 127.0.0.1:34212] Traceback (most recent call last):
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app
 127.0.0.1:34212]     response = self.full_dispatch_request()
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
 127.0.0.1:34212]     rv = self.handle_user_exception(e)
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
 127.0.0.1:34212]     reraise(exc_type, exc_value, tb)
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
 127.0.0.1:34212]     rv = self.dispatch_request()
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
 127.0.0.1:34212]     return self.view_functions[rule.endpoint](**req.view_args)
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/privacyidea/api/lib/prepolicy.py", line 129, in policy_wrapper
 127.0.0.1:34212]     return wrapped_function(*args, **kwds)
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/privacyidea/lib/log.py", line 154, in log_wrapper
 127.0.0.1:34212]     return func(*args, **kwds)
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/privacyidea/api/smtpserver.py", line 157, in test
 127.0.0.1:34212]     u"The configuration {} is working.".format(identifier))
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/privacyidea/lib/smtpserver.py", line 145, in send_or_enqueue_email
 127.0.0.1:34212]     return send(config, recipient, subject, body, sender, reply_to, mimetype)
 127.0.0.1:34212]   File "/opt/privacyidea/local/lib/python2.7/site-packages/privacyidea/lib/smtpserver.py", line 116, in test_email
 127.0.0.1:34212]     mail.login(config['username'], password)
 127.0.0.1:34212]   File "/usr/lib/python2.7/smtplib.py", line 607, in login
 127.0.0.1:34212]     (code, resp) = self.docmd(encode_cram_md5(resp, user, password))
 127.0.0.1:34212]   File "/usr/lib/python2.7/smtplib.py", line 571, in encode_cram_md5
 127.0.0.1:34212]     response = user + " " + hmac.HMAC(password, challenge).hexdigest()
 127.0.0.1:34212]   File "/usr/lib/python2.7/hmac.py", line 75, in __init__
 127.0.0.1:34212]     self.outer.update(key.translate(trans_5C))
 127.0.0.1:34212] TypeError: character mapping must return integer, None or unicode
 127.0.0.1:34212] /opt/privacyidea/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py:536: Warning: Data truncated for column 'info' at row 1
 127.0.0.1:34212]   cursor.execute(statement, parameters)
@fredreichbier

This comment has been minimized.

Copy link
Member

commented May 21, 2019

Thank you for your bug report. What SMTP server are you using? Judging from the traceback, I think we attempt to authenticate to the SMTP server using CRAM-MD5. The error is thrown because the type of password is unicode, which breaks the hmac module.

@XtraLarge

This comment has been minimized.

Copy link
Author

commented May 21, 2019

I've tried two servers both with tls encryption. The one fron selfhost.de and gmx.net.

@fredreichbier fredreichbier changed the title Privacyidea on Python2.7 Sending mails fails if SMTP server advertises CRAM-MD5 authentication May 21, 2019

@fredreichbier fredreichbier added bug and removed possible bug labels May 21, 2019

@fredreichbier

This comment has been minimized.

Copy link
Member

commented May 21, 2019

Thanks for the update! Indeed, it seems like sending mails is broken under Python 2 + privacyIDEA 3.0 if the SMTP server advertises CRAM-MD5.

I checked the behavior under privacyIDEA 2.23.5 to see if we had this bug prior to 3.0:

  • under 2.23.5, testing a SMTP server configuration with a CRAM-MD5 server fails with the same error ...
  • but if we just save a SMTP server configuration without testing, we can actually use it e.g. for email tokens, because the password is passed in a bytestring in this case.

So there might be 2.23.5 installations which currently work and would break with an update to 3.0 or 3.0.1.

fredreichbier added a commit that referenced this issue May 21, 2019

Fix sending mails in combination with CRAM-MD5 authentication
Under Python 2, we need to pass passwords as bytestrings, while Python 3
expects passwords as (unicode) strings.

Fixes #1637
@fredreichbier

This comment has been minimized.

Copy link
Member

commented May 21, 2019

I've pushed a fix to #1638. Could you test if this patch fixes the bug?

You mentioned you have installed privacyIDEA in a virtualenv. You can then apply the fix with the following command:

/path/to/your/virtualenv/bin/pip install https://github.com/privacyidea/privacyidea/archive/1637/smtp-cram-md5.zip

But beware, only do this in a test environment :)

@XtraLarge

This comment has been minimized.

Copy link
Author

commented May 21, 2019

Hello,
I tested the fix. I renamed the old files with the ending of ".py" to ".py.old" and ".pyc" to "pyc.old". After That I downloaded your fix as "smtpserver.py".

Then ...

I've tested it by using the eMail send dialog from:
Configuration -> System -> SMTP-Server
If I use the gmx server I can send eMails without an error if I use the eMail Account from gmx as sender.

But if I use the smtp server from the hoster of my dyndns domain, which allows every sender address in the dns domains of my hosted domain, I get the error: "character mapping must return integer, None or unicode" and the stack trace is:

[Tue May 21 22:11:07.286509 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192] [2019-05-21 22:11:07,285] ERROR in app: Exception on /smtpserver/send_test_email [POS
[Tue May 21 22:11:07.286559 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192] Traceback (most recent call last):
[Tue May 21 22:11:07.286565 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]   File "/opt/privacyidea/local/lib/python2.7/site-packages/flask/app.py", line 2292,
[Tue May 21 22:11:07.286570 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]     response = self.full_dispatch_request()
[Tue May 21 22:11:07.286575 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]   File "/opt/privacyidea/local/lib/python2.7/site-packages/flask/app.py", line 1815,
[Tue May 21 22:11:07.286579 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]     rv = self.handle_user_exception(e)
[Tue May 21 22:11:07.286584 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]   File "/opt/privacyidea/local/lib/python2.7/site-packages/flask/app.py", line 1718,
[Tue May 21 22:11:07.286589 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]     reraise(exc_type, exc_value, tb)
[Tue May 21 22:11:07.286593 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]   File "/opt/privacyidea/local/lib/python2.7/site-packages/flask/app.py", line 1813,
[Tue May 21 22:11:07.286598 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]     rv = self.dispatch_request()
[Tue May 21 22:11:07.286602 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]   File "/opt/privacyidea/local/lib/python2.7/site-packages/flask/app.py", line 1799,
[Tue May 21 22:11:07.286607 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]     return self.view_functions[rule.endpoint](**req.view_args)
[Tue May 21 22:11:07.286612 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]   File "/opt/privacyidea/local/lib/python2.7/site-packages/privacyidea/api/lib/prepol
[Tue May 21 22:11:07.286617 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]     return wrapped_function(*args, **kwds)
[Tue May 21 22:11:07.286621 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]   File "/opt/privacyidea/local/lib/python2.7/site-packages/privacyidea/lib/log.py", l
[Tue May 21 22:11:07.286626 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]     return func(*args, **kwds)
[Tue May 21 22:11:07.286630 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]   File "/opt/privacyidea/local/lib/python2.7/site-packages/privacyidea/api/smtpserver
[Tue May 21 22:11:07.286635 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]     u"The configuration {} is working.".format(identifier))
[Tue May 21 22:11:07.286640 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]   File "/opt/privacyidea/local/lib/python2.7/site-packages/privacyidea/lib/smtpserver
[Tue May 21 22:11:07.286662 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]     :return: True if the job is sent to the queue, return value of ``SMTPServer.test_
[Tue May 21 22:11:07.286666 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]   File "/opt/privacyidea/local/lib/python2.7/site-packages/privacyidea/lib/smtpserver
[Tue May 21 22:11:07.286670 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]     if password == FAILED_TO_DECRYPT_PASSWORD:
[Tue May 21 22:11:07.286674 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]   File "/usr/lib/python2.7/smtplib.py", line 607, in login
[Tue May 21 22:11:07.286678 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]     (code, resp) = self.docmd(encode_cram_md5(resp, user, password))
[Tue May 21 22:11:07.286682 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]   File "/usr/lib/python2.7/smtplib.py", line 571, in encode_cram_md5
[Tue May 21 22:11:07.286686 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]     response = user + " " + hmac.HMAC(password, challenge).hexdigest()
[Tue May 21 22:11:07.286690 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]   File "/usr/lib/python2.7/hmac.py", line 75, in __init__
[Tue May 21 22:11:07.286694 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192]     self.outer.update(key.translate(trans_5C))
[Tue May 21 22:11:07.286698 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192] TypeError: character mapping must return integer, None or unicode

I hope I've done no mistake by testing your fix ;)

@fredreichbier

This comment has been minimized.

Copy link
Member

commented May 22, 2019

Thanks for testing! Good news that it's working for one of the SMTP servers.

The traceback looks weird because it contains lines that can't really be part of the traceback (e.g. [Tue May 21 22:11:07.286662 2019] [wsgi:error] [pid 4850] [remote 127.0.0.1:47192] :return: True if the job is sent to the queue, return value of ``SMTPServer.test_). Maybe something went wrong applying the fix, and the *.pyc and *.py files got out of sync. Have you restarted the web server?

If this doesn't fix it, can you retry applying the fix using pip install? That should be more reliable than manually replacing files. As the PR already got merged and the branch deleted, the pip command above doesn't work anymore, it needs to be

/path/to/venv/bin/pip install https://github.com/privacyidea/privacyidea/archive/branch-3.0.zip
@XtraLarge

This comment has been minimized.

Copy link
Author

commented May 22, 2019

Hello, I've updated my system with the command above and it worked perfectly!

Thx a lot for the patch and the nice contact!

best regards
Willi Werres

@XtraLarge XtraLarge closed this May 22, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.