Failure while executing task: UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed

[differentlocal]

Hello!

When I trying to add new component to my project, I got an error:

weblate_1   | celery-main stderr | [2020-12-14 18:03:38,787: ERROR/384] Failure while executing task: UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed
weblate_1   | celery-main stderr | [2020-12-14 18:03:38,787: ERROR/ForkPoolWorker-1] Failure while executing task: UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed
weblate_1   | celery-main stderr | [2020-12-14 18:03:38,787: ERROR/384] Failure while executing task: UnicodeEncodeError: {'signal': <Signal: task_failure providing_args={'exception', 'task_id', 'kwargs', 'args', 'einfo', 'traceback'}>, 'sender': <@task: weblate.trans.tasks.component_after_save of weblate at 0x7f2c21964908>, 'task_id': 'bc2cbfd9-0fb3-41dd-bc64-6814deb86fb5', 'args': [28, True, False, False, False], 'kwargs': {'skip_push': True}, 'traceback': <traceback object at 0x7f2c1830d908>, 'einfo': <ExceptionInfo: UnicodeEncodeError('utf-8', '\ud83d\udc68\u200d\ud83d\udcbbАгенты', 0, 2, 'surrogates not allowed')>}
weblate_1   | celery-main stderr | [2020-12-14 18:03:38,787: ERROR/ForkPoolWorker-1] Failure while executing task: UnicodeEncodeError: {'signal': <Signal: task_failure providing_args={'exception', 'task_id', 'kwargs', 'args', 'einfo', 'traceback'}>, 'sender': <@task: weblate.trans.tasks.component_after_save of weblate at 0x7f2c21964908>, 'task_id': 'bc2cbfd9-0fb3-41dd-bc64-6814deb86fb5', 'args': [28, True, False, False, False], 'kwargs': {'skip_push': True}, 'traceback': <traceback object at 0x7f2c1830d908>, 'einfo': <ExceptionInfo: UnicodeEncodeError('utf-8', '\ud83d\udc68\u200d\ud83d\udcbbАгенты', 0, 2, 'surrogates not allowed')>}

(complete log attached to the issue - creationError.log)
I tried changing file_format atttribute to 'properties-utf8' or just 'properties' but it didn't help a bit. Source language is russian, format - java properties.

Weblate version is 4.3.2.

How I can workaround this error?
Thanks!

[nijel]

Normalizing the properties files so that it doesn't use surrogates should help. But still this should be handled more gracefully.

Can you try it with debug enabled in Weblate, so that the log contains traceback as well?

[differentlocal]

Hi!

I didn't plan to use surrogates in properties, I think this is an a mistake or some conversion error.
My weblate already running (in docker) with environmental options:

Weblate setup

WEBLATE_DEBUG=1
WEBLATE_LOGLEVEL=DEBUG

How I can set logging to be more verbose?

[nijel]

That should be good enough .... and it's in the log, sorry for not seeing it there:

weblate_1   | celery-main stderr | Traceback (most recent call last):
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 412, in trace_task
weblate_1   | celery-main stderr |     R = retval = fun(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 704, in __protected_call__
weblate_1   | celery-main stderr |     return self.run(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/autoretry.py", line 35, in run
weblate_1   | celery-main stderr |     return task._orig_run(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/tasks.py", line 279, in component_after_save
weblate_1   | celery-main stderr |     changed_git, changed_setup, changed_template, changed_variant, skip_push
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 2366, in after_save
weblate_1   | celery-main stderr |     was_change = self.create_translations()
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 1741, in create_translations
weblate_1   | celery-main stderr |     force, langs, request, changed_template, from_link
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 1853, in _create_translations
weblate_1   | celery-main stderr |     self, lang, code, path, force, request=request
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/translation.py", line 88, in check_sync
weblate_1   | celery-main stderr |     translation.check_sync(force, request=request)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/translation.py", line 410, in check_sync
weblate_1   | celery-main stderr |     self.sync_unit(dbunits, updated, id_hash, unit, pos + 1)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/translation.py", line 308, in sync_unit
weblate_1   | celery-main stderr |     newunit.update_from_unit(unit, pos, is_new)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/unit.py", line 589, in update_from_unit
weblate_1   | celery-main stderr |     run_checks=not same_source or not same_target or not same_state,
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/unit.py", line 341, in save
weblate_1   | celery-main stderr |     update_fields=update_fields,
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 754, in save
weblate_1   | celery-main stderr |     force_update=force_update, update_fields=update_fields)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 792, in save_base
weblate_1   | celery-main stderr |     force_update, using, update_fields,
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 895, in _save_table
weblate_1   | celery-main stderr |     results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 935, in _do_insert
weblate_1   | celery-main stderr |     using=using, raw=raw,
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/manager.py", line 85, in manager_method
weblate_1   | celery-main stderr |     return getattr(self.get_queryset(), name)(*args, **kwargs)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 1254, in _insert
weblate_1   | celery-main stderr |     return query.get_compiler(using=using).execute_sql(returning_fields)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/compiler.py", line 1397, in execute_sql
weblate_1   | celery-main stderr |     cursor.execute(sql, params)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 66, in execute
weblate_1   | celery-main stderr |     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
weblate_1   | celery-main stderr |     return executor(sql, params, many, context)
weblate_1   | celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 84, in _execute
weblate_1   | celery-main stderr |     return self.cursor.execute(sql, params)
weblate_1   | celery-main stderr | UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed

Not sure if Weblate should try to normalize these or fail on parsing, but it should fail on parsing the file, not when saving it to the database.

See also psycopg/psycopg2#586

[differentlocal]

As a note: previously, in Weblate version 2.17 this file was imported without any errors.

[nijel]

This has been changed in Python 3, see above referenced psycopg issue...

[github-actions]

Thank you for your report, the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.