Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UnicdeDecodeError on Ubuntu 16.04 in audit log #1560

Closed
cornelinux opened this Issue Apr 6, 2019 · 7 comments

Comments

Projects
None yet
3 participants
@cornelinux
Copy link
Member

commented Apr 6, 2019

A 3.0dev8 has an LDAP resolver talking to 2012.
A user has a german umlaut.

In this method we get a UnicodeDecodeError:
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/auditmodules/sqlaudit.py#L310

indeed it is due to le.user.

@cornelinux cornelinux added this to the 3.0 Code Cleanup milestone Apr 6, 2019

@cornelinux

This comment has been minimized.

Copy link
Member Author

commented Apr 6, 2019

THe same happens with the packages on Ubuntu 1804. and python 2.7

[2019-04-06 14:50:26,809][4489][140191420528384][ERROR][privacyidea.lib.auditmodules.sqlaudit:253] exception UnicodeDecodeError('ascii', 'k\xf6lbel', 1, 2, 'ordinal not in range(128)')
[2019-04-06 14:50:26,810][4489][140191420528384][ERROR][privacyidea.lib.auditmodules.sqlaudit:254] DATA: {'info': '', 'administrator': u'super', 'realm': u'testfoo', 'policies': '', 'success': True, 'privacyidea_server': '10.0.2.232', 'client_user_agent': 'chrome', 'token_type': u'totp', 'client': '10.1.0.3', 'user': u'k\xf6lbel', 'resolver': u'testfoo', 'action_detail': '', 'action': 'POST /token/init', 'serial': u'TOTP0000FFDB'}
[2019-04-06 14:50:44,383][4489][140191403742976][INFO][privacyidea.lib.user:230] user u'k\xf6lbel' found in resolver u'testfoo'
[2019-04-06 14:50:44,384][4489][140191403742976][INFO][privacyidea.lib.user:231] userid resolved to '30d0b95e-2a32-4a28-9ac9-3e9e0eed828c' 
[2019-04-06 14:50:44,584][4489][140191403742976][ERROR][privacyidea.lib.auditmodules.sqlaudit:253] exception UnicodeDecodeError('ascii', 'k\xf6lbel', 1, 2, 'ordinal not in range(128)')
[2019-04-06 14:50:44,585][4489][140191403742976][ERROR][privacyidea.lib.auditmodules.sqlaudit:254] DATA: {'info': 'wrong otp value', 'realm': u'testfoo', 'tokentype': u'totp', 'success': False, 'privacyidea_server': '10.0.2.232', 'client_user_agent': 'chrome', 'client': '10.1.0.3', 'user': u'k\xf6lbel', 'resolver': u'testfoo', 'action_detail': '', 'action': 'GET /validate/check', 'serial': u'TOTP0000FFDB', 'policies': ''}

This is independent of the charset of the database:

mysql>  SELECT @@character_set_database, @@collation_database;
+--------------------------+----------------------+
| @@character_set_database | @@collation_database |
+--------------------------+----------------------+
| utf8mb4                  | utf8mb4_unicode_ci   |
+--------------------------+----------------------+
@cornelinux

This comment has been minimized.

Copy link
Member Author

commented Apr 6, 2019

Another problem is, that one error in the method _log_to_string will make the view of the complete auditlog fail.
So maybe we should add a try-except anyways.

@cornelinux

This comment has been minimized.

Copy link
Member Author

commented Apr 6, 2019

The user is returned as 'str' here:
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/auditmodules/sqlaudit.py#L412

This then fails with _log_to_string, that tries to create a unicode.

There is no problem with sqlite, but the problem arises with mysql

@cornelinux

This comment has been minimized.

Copy link
Member Author

commented Apr 6, 2019

Interesting. We had a similar error in 2.23.5.

@fredreichbier

This comment has been minimized.

Copy link
Member

commented Apr 8, 2019

I reproduced this on a MySQL setup. In my setup the issue can be fixed by adding a charset to the connect string:

SQLALCHEMY_DATABASE_URI = 'mysql+pymysql://pi:...@localhost/pi?charset=utf8mb4'

But I guess we should adjust the code accordingly anyway.

@plettich

This comment has been minimized.

Copy link
Member

commented Apr 8, 2019

One way to solve this is to set the db.Column to db.Unicode instead of db.String in the Audit() class

@cornelinux

This comment has been minimized.

Copy link
Member Author

commented Apr 8, 2019

I added the charset as @fredreichbier suggested in the ubuntu build respoitory and it works out well - also for already existing databases.
db.Unicode seems only to work, if the db is newly created. Anyways, please create a PR for it.

plettich added a commit that referenced this issue Apr 8, 2019

Fix Unicode error in audit log
When using mysql with the audit log, there is an issue if one of the
columns of an entry contains non ascii symbols.
To avoid writing invalid data to mysql, the column types in the SQLAlchemy
model need to be set to `Unicode`.
Some possible regressions need to be caught when updating the code.

fixes #1560
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.