Allow usernames with non-English characters

[bunnybot]

We currently only allow a subset of ASCII in usernames. It would be nice if speakers of languages other than English were more free in their choice, e.g. allow them to add äßé or a Username in Cyrillic or Chinese script.

See also the website bug 1784693 "Using Unicode literals cause a server error"

---

Imported from Launchpad using lp2gh.

• date created: 2019-05-14T03:38:35Z
• owner: gunchleoc
• the launchpad url was https://bugs.launchpad.net/bugs/1828923

[bunnybot]

(by franku)
This is a restriction of Django in combination python2:

"... Unicode characters have always been accepted when using Python 3. Django 1.10 officially added Unicode support in usernames, keeping the ASCII-only behavior on Python 2."

See: https://docs.djangoproject.com/en/1.11/ref/contrib/auth/#django.contrib.auth.models.User.username

So the switch to python 3 will accept a wide range of additional characters. Tested 'ß' in usernames, also polish characters in the python3 branch.

But we need to change the database encoding to fully support unicode (e.g. chinese characters), which means the length of text columns shrinks and thus there is a possibility to loose some of text in old entries. So this has to be done carefully.

[bunnybot]

(by stonerl)
Before we can enable this on the website, we need to make sure widelands can properly handle this.

Here is a list of things to do first:

• proper username parsing (e.g. spaces in names)
• autocomplete usernames on tab in chats
• proper chinese/japanese/korean etc input
  (at least japanese is broken)

We must be sure that usernames are accessible by others; for wispering, warning & kicking. Otherwise, a user could just choose a name that other can't type and non of the above mentioned would be possible.

That's why I'm currently limiting the possible characters for usernames in widelands.

Enabling this will be much work. We should also make sure that we do not enable support for this on the website without having the code in widelands in place.

[bunnybot]

(by franku)

• proper username parsing (e.g. spaces in names)

FYI: django's default is to permit spaces in usernames.

[bunnybot]

(by franku)
Sorry: djangos default is to forbid spaces in usernames

[bunnybot]

(by stonerl)
Well, then we can add one checkmark to the list :)

[bunnybot]

(by gunchleoc)
I am in favor of reworking the chat UI so that player can click on a user in a list of usernames to add their name to the chat. That would also solve the problem of accidentally posting publicly instead of whispering when you mistype a username.

It would also be really cool if we had a team chat in-game.

So, start this project with a redesign of the chat UI?

[bunnybot]

(by stonerl)

So, start this project with a redesign of the chat UI?

I guess... :)

[bunnybot]

(by gunchleoc)
Issue moved to widelands/widelands-website#300

[gunchleoc]

@frankystone Could our Django handle this now? I don't expect this to happen for Build 21, but it would be nice to know.

[frankystone]

Django should be able to handle this. The problem is the Database encoding.

The Database uses the encoding utf8 (mysql specific), which stores up to 3 bytes for a character. To support all possible unicode code points, we have to convert the columns in the database to use the encoding utf8mb4 (mysql specific), which uses up to 4 bytes for a character.

Characters which uses more than 3 bytes are all below the Basic Multilingual Plane ('0 BMP'), see https://en.wikipedia.org/wiki/Unicode_block. These are e.g. emojis and Caucasian Albanian. May some diacritical signs uses also more than 4 bytes.