Unicode Character Problem

[bilalcorbacioglu]

Describe the bug

We made an arrangement for whitespace characters. While the context looks correct in the Weblate interface, the context from the API is wrong.

Example Context: « Je », pleurant.

I used this site for encoding. (Also, I used Google Chrome.)

from Weblate Interface(Correct):

↑↓	↑↓	↑↓
«	171	0xab
	160	0xa0 <---
J	74	0x4a
e	101	0x65
	160	0xa0 <---
»	187	0xbb
,	44	0x2c
	32	0x20
p	112	0x70
l	108	0x6c
e	101	0x65
u	117	0x75
r	114	0x72
a	97	0x61
n	110	0x6e
t	116	0x74
.	46	0x2e

from Weblate API(Incorrect):

↑↓	↑↓	↑↓
«	171	0xab
	32	0x20 <---
J	74	0x4a
e	101	0x65
	32	0x20 <---
»	187	0xbb
,	44	0x2c
	32	0x20
p	112	0x70
l	108	0x6c
e	101	0x65
u	117	0x75
r	114	0x72
a	97	0x61
n	110	0x6e
t	116	0x74
.	46	0x2e

Expected behavior

All characters appearing in Interface and appearing in API must be the same.

Server configuration and status

 * Weblate: 4.2.1
 * Django: 3.1
 * siphashc: 1.3
 * Whoosh: 2.7.4
 * translate-toolkit: 3.0.0
 * lxml: 4.5.2
 * Pillow: 7.2.0
 * bleach: 3.1.5
 * python-dateutil: 2.8.1
 * social-auth-core: 3.3.3
 * social-auth-app-django: 4.0.0
 * django-crispy-forms: 1.9.2
 * oauthlib: 3.1.0
 * django-compressor: 2.4
 * djangorestframework: 3.11.1
 * django-filter: 2.3.0
 * django-appconf: 1.0.4
 * user-agents: 2.1
 * filelock: 3.0.12
 * setuptools: 40.8.0
 * jellyfish: 0.8.2
 * openpyxl: 3.0.4
 * celery: 4.4.7
 * kombu: 4.6.11
 * translation-finder: 2.1
 * html2text: 2020.1.16
 * pycairo: 1.16.2
 * pygobject: 3.30.4
 * diff-match-patch: 20200713
 * requests: 2.24.0
 * django-redis: 4.12.1
 * hiredis: 1.1.0
 * sentry_sdk: 0.16.5
 * Cython: 0.29.21
 * misaka: 2.1.1
 * GitPython: 3.1.7
 * borgbackup: 1.1.13
 * pyparsing: 2.4.7
 * Python: 3.7.3
 * Git: 2.20.1
 * psycopg2: 2.8.5
 * psycopg2-binary: 2.8.5
 * phply: 1.2.5
 * chardet: 3.0.4
 * ruamel.yaml: 0.16.10
 * tesserocr: 2.5.1
 * akismet: 1.1
 * boto3: 1.14.47
 * zeep: 3.4.0
 * aeidon: 1.7.0
 * iniparse: 0.5
 * mysqlclient: 2.0.1
 * Mercurial: 5.5
 * git-svn: 2.20.1
 * git-review: 1.28.0
 * hub: 2.13.0
 * lab: 0.16
 * Redis server: 5.0.9
 * PostgreSQL server: 11.7
 * Database backends: django.db.backends.postgresql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Email setup: django.core.mail.backends.smtp.EmailBackend: smtp.gmail.com
 * OS encoding: filesystem=utf-8, default=utf-8
 * Celery: redis://cache:6379/1, redis://cache:6379/1, regular
 * Platform: Linux 4.15.0-112-generic (x86_64

[nijel]

@bilalcorbacioglu did you figure out where the issue was? It might be useful to know for others who would hit similar issue.

[github-actions]

The issue you have reported seems to be resolved now.

• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, consider supporting Weblate by donating.

[bilalcorbacioglu]

Actually, I didn't anything special. The problem depends on the browser. Safari showing Non-breaking space a little bit longer. So I didn't realize it.