Emojis break Java Properties (ISO-8859-1) import (file upload and/or GIT)

[afiller]

Describe the issue

When importing Java Properties (ISO-8859-1) files via file upload or GIT it results in an 502 NGINX error. I found out the UWSGI shows the following error:

UnicodeEncodeError: 'utf-8' codec can't encode characters in position 70-71: surrogates not allowed

After some research I found out that the following emojis caused the problem: ☺️👍

The unicode sequence of the same in the file is: \u263A\uFE0F\uD83D\uDC4D

When I remove this sequence from the file everything works fine again... my file is only a bit more boring.

A full line to check it out:

hello.world = Well done! \u263A\uFE0F\uD83D\uDC4D

Maybe it's a Python 3 problem, as I stumbled upon some related issues on the web.

I already tried

See description

To Reproduce the issue

See description

Expected behavior

The file imports normally and all my strings appear as localization strings

Screenshots

Exception traceback

Server configuration and status

• Weblate: 4.5.1
• Django: 3.1.7
• siphashc: 2.1
• translate-toolkit: 3.3.3
• lxml: 4.6.3
• Pillow: 8.1.2
• bleach: 3.3.0
• python-dateutil: 2.8.1
• social-auth-core: 4.1.0
• social-auth-app-django: 4.0.0
• django-crispy-forms: 1.11.2
• oauthlib: 3.1.0
• django-compressor: 2.4
• djangorestframework: 3.12.2
• django-filter: 2.4.0
• django-appconf: 1.0.4
• user-agents: 2.2.0
• filelock: 3.0.12
• setuptools: 44.0.0
• jellyfish: 0.8.2
• openpyxl: 3.0.7
• celery: 5.0.5
• kombu: 5.0.2
• translation-finder: 2.9
• weblate-language-data: 2021.3
• html2text: 2020.1.16
• pycairo: 1.20.0
• pygobject: 3.40.0
• diff-match-patch: 20200713
• requests: 2.22.0
• django-redis: 4.12.1
• hiredis: 1.1.0
• sentry_sdk: 1.0.0
• Cython: 0.29.22
• misaka: 2.1.1
• GitPython: 3.1.14
• borgbackup: 1.1.16
• pyparsing: 2.4.6
• pyahocorasick: 1.4.1
• Python: 3.8.5
• Git: 2.25.1
• psycopg2-binary: 2.8.6
• chardet: 3.0.4
• ruamel.yaml: 0.16.13
• tesserocr: 2.5.1
• Redis server: 5.0.7
• PostgreSQL server: 13.2
• Database backends: django.db.backends.postgresql
• Cache backends: default:RedisCache, avatar:FileBasedCache
• Email setup: django.core.mail.backends.smtp.EmailBackend: localhost
• OS encoding: filesystem=utf-8, default=utf-8
• Celery: redis://localhost:6379, redis://localhost:6379, regular
• Platform: Linux 5.4.0-70-generic (x86_64)

Weblate deploy checks

WARNINGS:
?: (security.W018) You should not have DEBUG set to True in deployment.
?: (weblate.W025.ass) Failure in loading handler for ass file format: aeidon or gaupol package required for Subtitle support
HINT: https://docs.weblate.org/en/weblate-4.5.1/admin/install.html#optional-deps
?: (weblate.W025.ini) Failure in loading handler for ini file format: Missing iniparse library.
HINT: https://docs.weblate.org/en/weblate-4.5.1/admin/install.html#optional-deps
?: (weblate.W025.islu) Failure in loading handler for islu file format: Missing iniparse library.
HINT: https://docs.weblate.org/en/weblate-4.5.1/admin/install.html#optional-deps
?: (weblate.W025.laravel) Failure in loading handler for laravel file format: No module named 'phply'
HINT: https://docs.weblate.org/en/weblate-4.5.1/admin/install.html#optional-deps
?: (weblate.W025.php) Failure in loading handler for php file format: No module named 'phply'
HINT: https://docs.weblate.org/en/weblate-4.5.1/admin/install.html#optional-deps
?: (weblate.W025.srt) Failure in loading handler for srt file format: aeidon or gaupol package required for Subtitle support
HINT: https://docs.weblate.org/en/weblate-4.5.1/admin/install.html#optional-deps
?: (weblate.W025.ssa) Failure in loading handler for ssa file format: aeidon or gaupol package required for Subtitle support
HINT: https://docs.weblate.org/en/weblate-4.5.1/admin/install.html#optional-deps
?: (weblate.W025.sub) Failure in loading handler for sub file format: aeidon or gaupol package required for Subtitle support
HINT: https://docs.weblate.org/en/weblate-4.5.1/admin/install.html#optional-deps
?: (weblate.W033.Gerrit) Failure in loading VCS module for Gerrit: git: 'review' is not a git command. See 'git --help'.
(1)
HINT: https://docs.weblate.org/en/weblate-4.5.1/vcs.html
?: (weblate.W033.Mercurial) Failure in loading VCS module for Mercurial: [Errno 2] No such file or directory: 'hg'
HINT: https://docs.weblate.org/en/weblate-4.5.1/vcs.html
?: (weblate.W033.Subversion) Failure in loading VCS module for Subversion: git: 'svn' is not a git command. See 'git --help'.

The most similar commands are
fsck
mv
show
(1)
HINT: https://docs.weblate.org/en/weblate-4.5.1/vcs.html

INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
HINT: https://docs.weblate.org/en/weblate-4.5.1/admin/install.html#collecting-errors

Additional context

[nijel]

Can you please share full exception traceback?

PS: See also #5041

[github-actions]

Thank you for your report; the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.

[afiller]

Wow, thank you for the fast fix! 👍 It's working in the new release.