Weblate throws unhandled error when the entry in glossary has non-Unicode character

[pan93412]

Describe the issue

Users can mistakenly type a control character inside glossary. For example, “[\x08]commit”:

And Weblate will raise exceptions when committing, downloading or (sometimes) saving glossary to tbx files:

User-side sees:

I already tried

• I've read and searched the documentation.
• I've searched for similar issues in this repository.

Steps to reproduce the behavior

1. Add a glossary entry whose source entry has a non-Unicode character.
2. Try to commit this glossary to repository, and the error raises.

Expected behavior

It should strip out the Unicode character when seeing the invalid character instead of raising a unrecoverable error. Besides, it will be better to not letting users insert an entry including such that character.

Screenshots

See Describe the issue.

Exception traceback

ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/contrib/auth/decorators.py", line 23, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "django/views/decorators/http.py", line 43, in inner
    return func(request, *args, **kwargs)
  File "weblate/trans/views/git.py", line 145, in commit_translation
    return perform_commit(request, obj)
  File "weblate/trans/views/git.py", line 54, in perform_commit
    return execute_locked(
  File "weblate/trans/views/git.py", line 35, in execute_locked
    result = call(*args, **kwargs)
  File "contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "weblate/trans/models/translation.py", line 573, in commit_pending
    return self.component.commit_pending(reason, user, skip_push=skip_push)
  File "weblate/trans/models/component.py", line 166, in on_link_wrapper
    return func(self, *args, **kwargs)
  File "weblate/trans/models/component.py", line 1616, in commit_pending
    translation._commit_pending(reason, user)
  File "contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "weblate/trans/models/translation.py", line 613, in _commit_pending
    self.update_units(units, store, author_name, author.id)
  File "weblate/trans/models/translation.py", line 710, in update_units
    pounit = store.new_unit(
  File "weblate/formats/base.py", line 585, in new_unit
    unit = self.create_unit(key, source, target)
  File "weblate/formats/ttkit.py", line 399, in create_unit
    unit = self.construct_unit(context)
  File "weblate/formats/ttkit.py", line 357, in construct_unit
    unit.setsource(source, self.source_language)
  File "translate/storage/lisa.py", line 122, in setsource
    self.source_dom = self.createlanguageNode(sourcelang, text, "source")
  File "translate/storage/tbx.py", line 44, in createlanguageNode
    term.text = text
  File "src/lxml/etree.pyx", line 1041, in lxml.etree._Element.text.__set__
  File "src/lxml/apihelpers.pxi", line 748, in lxml.etree._setNodeText
  File "src/lxml/apihelpers.pxi", line 736, in lxml.etree._createTextNode
  File "src/lxml/apihelpers.pxi", line 1541, in lxml.etree._utf8

How do you run Weblate?

Docker container

Weblate versions

$ docker-compose exec --user weblate weblate weblate list_versions
 * Weblate: 4.14.1
 * Django: 4.1.1
 * siphashc: 2.1
 * translate-toolkit: 3.7.3
 * lxml: 4.9.1
 * Pillow: 9.2.0
 * bleach: 5.0.1
 * python-dateutil: 2.8.2
 * social-auth-core: 4.3.0
 * social-auth-app-django: 5.0.0
 * django-crispy-forms: 1.14.0
 * oauthlib: 3.2.1
 * django-compressor: 4.1
 * djangorestframework: 3.13.1
 * django-filter: 22.1
 * django-appconf: 1.0.5
 * user-agents: 2.2.0
 * filelock: 3.8.0
 * rapidfuzz: 2.8.0
 * openpyxl: 3.0.10
 * celery: 5.2.7
 * kombu: 5.2.4
 * translation-finder: 2.14
 * weblate-language-data: 2022.7
 * html2text: 2020.1.16
 * pycairo: 1.21.0
 * pygobject: 3.42.2
 * diff-match-patch: 20200713
 * requests: 2.28.1
 * django-redis: 5.2.0
 * hiredis: 2.0.0
 * sentry_sdk: 1.9.8
 * Cython: 0.29.32
 * misaka: 2.1.1
 * GitPython: 3.1.27
 * borgbackup: 1.2.2
 * pyparsing: 3.0.9
 * pyahocorasick: 1.4.4
 * python-redis-lock: 3.7.0
 * charset-normalizer: 2.1.1
 * Python: 3.10.7
 * Git: 2.30.2
 * psycopg2: 2.9.3
 * psycopg2-binary: 2.9.3
 * phply: 1.2.5
 * ruamel.yaml: 0.17.21
 * tesserocr: 2.5.2
 * boto3: 1.24.73
 * zeep: 4.1.0
 * aeidon: 1.11
 * iniparse: 0.5
 * mysqlclient: 2.1.1
 * Mercurial: 6.2.2
 * git-svn: 2.30.2
 * git-review: 2.3.1
 * Redis server: 6.2.7
 * PostgreSQL server: 13.8
 * Database backends: django.db.backends.postgresql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Email setup: django.core.mail.backends.smtp.EmailBackend: smtp.mailgun.org
 * OS encoding: filesystem=utf-8, default=utf-8
 * Celery: redis://cache:6379/1, redis://cache:6379/1, regular
 * Platform: Linux 5.10.0-19-amd64 (x86_64)

Weblate deploy checks

$ docker-compose exec --user weblate weblate weblate check --deploy
System check identified some issues:

WARNINGS:
?: (security.W004) You have not set a value for the SECURE_HSTS_SECONDS setting. If your entire site is served only over SSL, you may want to consider setting a value and enabling HTTP Strict Transport Security. Be sure to read the documentation first; enabling HSTS carelessly can cause serious, irreversible problems.
?: (security.W008) Your SECURE_SSL_REDIRECT setting is not set to True. Unless your site should be available over both SSL and non-SSL connections, you may want to either set this setting True or configure a load balancer or reverse-proxy server to redirect all connections to HTTPS.
?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only session cookie makes it more difficult for network traffic sniffers to hijack user sessions.

INFOS:
?: (weblate.I031) New Weblate version is available, please upgrade to 4.14.2.
	HINT: https://docs.weblate.org/en/weblate-4.14.1/admin/upgrade.html

System check identified 4 issues (1 silenced).

Additional context

No response

Workarounds for users

1. Find out the id of the affected string by triggering an error. For example, delete this string and it may navigate to YOUR_INSTANCE_URL/unit/321150/delete – the 321150 is our ID to fix up.
2. Backup your database before running the following operations!
3. Open your frontend of database, and run:

BEGIN;
-- Knowing what column of `id` includes the non-Unicode character.
SELECT * from trans_unit WHERE id = <that id>;
-- Update the column to the correct version.
UPDATE trans_unit SET <column>='<new value>' WHERE id = <that id>;
-- Check again before committing.
SELECT * from trans_unit WHERE id = <that id>;
COMMIT;

ROLLBACK; if any of part gets unexpected result.

[github-actions]

Thank you for your report; the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.