Component language gets deleted after upload new language source file

[adriandiazgar]

Describe the bug

The custom language zh_CN disappears when uploading again the source file with a replacing mechanism.

To Reproduce the bug

Steps to reproduce the behavior:

0. Make sure language zh_CN exists on Weblate, if not create it.
1. Create a project "mobile-project"
2. Add a new component from scratch android-app language English (en), file format Android Resource
3. Add translations for zh_CN XML file based
4. Check that the language is present in the component (for example via de API: api/translations/mobile-project/android-app/zh_CN/)
5. Re-upload the strings.xml file to the Project, so new strings get populated using the replace mechanism.
6. Check that the language is not present anymore in the component (for example via de API: api/translations/mobile-project/android-app/zh_CN/) --> NOT FOUND

Expected behavior

When on point number 4 a new strings.xml get uploaded, only add the new strings to the different languages but not get rid of any of them.

Server configuration and status

Weblate installation: Hosted service

weblate list_versions

On Git checkout:

./manage.py list_versions

` * Weblate: 4.3

• Django: 3.1.2
• siphashc: 2.1
• Whoosh: 2.7.4
• translate-toolkit: 3.1.1
• lxml: 4.5.2
• Pillow: 7.2.0
• bleach: 3.2.1
• python-dateutil: 2.8.1
• social-auth-core: 3.3.3
• social-auth-app-django: 4.0.0
• django-crispy-forms: 1.9.2
• oauthlib: 3.1.0
• django-compressor: 2.4
• djangorestframework: 3.12.1
• django-filter: 2.4.0
• django-appconf: 1.0.4
• user-agents: 2.2.0
• filelock: 3.0.12
• setuptools: 40.8.0
• jellyfish: 0.8.2
• openpyxl: 3.0.5
• celery: 4.4.7
• kombu: 4.6.11
• translation-finder: 2.2
• html2text: 2020.1.16
• pycairo: 1.16.2
• pygobject: 3.30.4
• diff-match-patch: 20200713
• requests: 2.24.0
• django-redis: 4.12.1
• hiredis: 1.1.0
• sentry_sdk: 0.19.0
• Cython: 0.29.21
• misaka: 2.1.1
• GitPython: 3.1.9
• borgbackup: 1.1.14
• pyparsing: 2.4.7
• Python: 3.7.3
• Git: 2.20.1
• psycopg2: 2.8.6
• psycopg2-binary: 2.8.6
• phply: 1.2.5
• chardet: 3.0.4
• ruamel.yaml: 0.16.12
• tesserocr: 2.5.1
• akismet: 1.1
• boto3: 1.15.16
• zeep: 4.0.0
• aeidon: 1.7.0
• iniparse: 0.5
• mysqlclient: 2.0.1
• Mercurial: 5.5.2
• git-svn: 2.20.1
• git-review: 1.28.0
• Redis server: 5.0.7
• PostgreSQL server: 10.12
• Database backends: django.db.backends.postgresql
• Cache backends: default:RedisCache, avatar:FileBasedCache
• Email setup: django.core.mail.backends.smtp.EmailBackend: xxxxxxx
• OS encoding: filesystem=utf-8, default=utf-8
• Celery: amqp://xxxxxx@xxxxxxxx, redis://xxxxxx:6379/2, regular
• Platform: Linux 4.15.0-112-generic (x86_64)`

[nijel]

Do you get any errors in Weblate logs?

Are the files removed from the repository? Check content of DATA_DIR/vcs/mobile-project/android-app on your Weblate server.

[github-actions]

This issue looks more like a support question than an issue. We strive to answer these reasonably fast, but purchasing the support subscription is not only more responsible and faster for your business but also makes Weblate stronger. In case your question is already answered, making a donation is the right way to say thank you!

[adriandiazgar]

Hi, @nijel thanks for your quick response.

Once the new strings.xml file is uploaded. The zn_CN.xml file still remains in the server local repo but Weblate can't retrieve it from the API neither the UI.

Thanks in advance,
Adrián

[nijel]

Does it appear under different name? Or there are no translations shown by Weblate? Anyway the Weblate logs should indicate why it has been removed...

[adriandiazgar]

I think it gets moved to zn_Hans which is an alias for zn_cn (lowercase) however the language disappearing (getting moved) is zn_CN (CN Uppercase).

[nijel]

I think it gets moved to zn_Hans

Can you take a look? And take a look at the actual files?

Before c355570 (included in the 4.3.2 release) there could have been some inconsistencies how newly created files and existing files were parsed and this could lead to different naming of the translations in the UI. Still, the actual files should stay there and should not be removed...

[jojurajan]

Hi Nijel,
We found the actual issue as to why this was happening. Even if we provide the language code for creating a translation, it takes the filepath of generated XML file and due to the conversions happening at

weblate/weblate/lang/models.py

Line 170 in 8f6e12d

def fuzzy_get(self, code, strict=False):

and

weblate/weblate/lang/models.py

Line 149 in 8f6e12d

def aliases_get(self, code, expanded_code=None):

the language code gets set as zh_Hans instead of zh_CN which was provided in the create request.

The issue can be reproduced by adding any chinese locale that has a traditional/simplified version. e.g. zh_HK, zh_CN, zh_SG.

[nijel]

The fuzzy_get first tries exact matches, so in case the language exists it is used. In case no match is found, it falls back to trying aliases and other heuristics. See https://docs.weblate.org/en/latest/admin/languages.html#parsing-language-codes for more detailed explanation.

[jojurajan]

The language provided to fuzzy_get is zh-rCN which fetches zh_Hans, instead of zh_CN which was provided for the translation creation via API.
I suppose this happens because zh_CN and zh_Hans map to the same Android filename zh-rCN.xml.

I hope this clarifies the issue that we are facing after using latest 4.3.2 Docker image which also includes this commit -> c355570.

[nijel]

Thanks, now I can reproduce this.

Anyway, what is wrong on using zh_Hans inside Weblate?

[jojurajan]

We are migrating data from an existing platform and we need to use the old language codes for backward compatibility.

[github-actions]

Thank you for your report, the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.

[nijel]

We are migrating data from an existing platform and we need to use the old language codes for backward compatibility.

You can use anything in the file names, it will be just mapped for Weblate. It's better to stick with Weblate defined languages on Weblate side for correct behaviour of some features (for example machine translation using third-party services).