stringsdict: Plural forms are not loaded correctly from file

[x-sheep]

Continuation of #6368

Describe the issue

I'm doing more tests in the bleeding edge version. I've created another project with an existing file (i.e. the same stringsdict file used in the unit test) and tried to add a new language using only the web interface.

After Weblate parses the file, but before saving the changes, it seems to have parsed 6 plural forms for English instead of 3. This becomes visible when writing the translation for another language (Dutch, in my case).

If I edit the English version in Weblate and save it, and then go to Dutch, it does display the text correctly.

I already tried

Describe the steps you tried to solve the problem yourself.

• I've read and searched the docs and did not find the answer there.
  If you didn’t try already, try to search there what you wrote above.
• I've tried running weblate loadpo --force project/component.
• I've tried manually recreating the component.

To Reproduce the issue

Steps to reproduce the behavior:

1. Create a iOS project with a stringsdict file in English.
2. Create a translation component with the stringsdict format.
3. Add a new language.
4. Edit a plural string.
5. Observe an incorrect number of source texts.

Expected behavior

Three strings should be displayed in the source text:

• English, Zero
• English, Singular
• English, Plural

Screenshots

Server configuration and status

Weblate installation: Docker

 * Weblate: 4.8-9797fd5fe7
 * Django: 3.2.6
 * siphashc: 2.1
 * translate-toolkit: 3.3.6
 * lxml: 4.6.3
 * Pillow: 8.3.1
 * bleach: 3.3.1
 * python-dateutil: 2.8.2
 * social-auth-core: 4.1.0
 * social-auth-app-django: 5.0.0
 * django-crispy-forms: 1.12.0
 * oauthlib: 3.1.1
 * django-compressor: 2.4.1
 * djangorestframework: 3.12.4
 * django-filter: 2.4.0
 * django-appconf: 1.0.4
 * user-agents: 2.2.0
 * filelock: 3.0.12
 * setuptools: 40.8.0
 * jellyfish: 0.8.2
 * openpyxl: 3.0.7
 * celery: 5.1.2
 * kombu: 5.1.0
 * translation-finder: 2.10
 * weblate-language-data: 2021.5
 * html2text: 2020.1.16
 * pycairo: 1.16.2
 * pygobject: 3.30.4
 * diff-match-patch: 20200713
 * requests: 2.26.0
 * django-redis: 5.0.0
 * hiredis: 2.0.0
 * sentry_sdk: 1.3.1
 * Cython: 0.29.24
 * misaka: 2.1.1
 * GitPython: 3.1.18
 * borgbackup: 1.1.17
 * pyparsing: 2.4.7
 * pyahocorasick: 1.4.2
 * python-redis-lock: 3.7.0
 * Python: 3.7.3
 * Git: 2.20.1
 * psycopg2: 2.9.1
 * psycopg2-binary: 2.9.1
 * phply: 1.2.5
 * chardet: 4.0.0
 * ruamel.yaml: 0.17.10
 * tesserocr: 2.5.2
 * akismet: 1.1
 * boto3: 1.18.19
 * zeep: 4.0.0
 * aeidon: 1.9
 * iniparse: 0.5
 * mysqlclient: 2.0.3
 * Mercurial: 5.8.1
 * git-svn: 2.20.1
 * git-review: 2.1.0
 * Redis server: 6.2.5
 * PostgreSQL server: 13.3
 * Database backends: django.db.backends.postgresql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Email setup: django.core.mail.backends.dummy.EmailBackend: 127.0.0.1
 * OS encoding: filesystem=utf-8, default=utf-8
 * Celery: redis://cache:6379/1, redis://cache:6379/1, regular
 * Platform: Linux 5.10.47-linuxkit (x86_64)

Weblate deploy checks

WARNINGS:
?: (security.W004) You have not set a value for the SECURE_HSTS_SECONDS setting. If your entire site is served only over SSL, you may want to consider setting a value and enabling HTTP Strict Transport Security. Be sure to read the documentation first; enabling HSTS carelessly can cause serious, irreversible problems.
?: (security.W008) Your SECURE_SSL_REDIRECT setting is not set to True. Unless your site should be available over both SSL and non-SSL connections, you may want to either set this setting True or configure a load balancer or reverse-proxy server to redirect all connections to HTTPS.
?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only session cookie makes it more difficult for network traffic sniffers to hijack user sessions.
?: (security.W018) You should not have DEBUG set to True in deployment.

INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
        HINT: https://docs.weblate.org/en/latest/admin/install.html#collecting-errors
?: (weblate.I028) Backups are not configured, it is highly recommended for production use
        HINT: https://docs.weblate.org/en/latest/admin/backup.html

Additional context

The stringsdict implementation in Translate Toolkit defaults to using all plural forms when it doesn't know the target language.

[nijel]

Most likely the English file path doesn't contain language code. Am I right? Probably the language detection in the translate-toolkit should be improved here...

[github-actions]

The issue you've reported needs to be addressed in the translate-toolkit. Please file the issue there, and include links to any relevant specifications about the formats (if applicable).

[nijel]

Can you plase try if following patch fixes it?

--- translate/storage/stringsdict.py
+++ translate/storage/stringsdict.py
@@ -97,6 +97,8 @@ class StringsDictFile(base.DictStore):
             match = re.search(r"^(\w*).lproj", parent_dir)
             if match is not None:
                 target_lang = match.group(1)
+                if target_lang == "Base":
+                    target_lang = "en"
             else:
                 target_lang = self.sourcelanguage
 

[nijel]

I've created a PR with this: translate/translate#4411

[github-actions]

Thank you for your report; the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.

[x-sheep]

Hi, sorry for the late response. I've just tried the latest bleeding version again, and it seems to not be fixed. It still displays the same behavior as before.

My paths are:

Project/en.lproj/Localizable.stringsdict
Project/nl.lproj/Localizable.stringsdict

[nijel]

Okay, it will have different cause then.

[x-sheep]

Is it possible that the file is parsed before the target language or filename is set on the StringsDictFile class? The parser will try to fill all 6 plural forms when gettargetlanguage() returns None. Sadly, the multistring class in the Translate Toolkit doesn't save any context for each string...

[nijel]

Yes, that's possible. Other formats return whatever plurals are stored in the file when parsing and they probably don't use language at that stage (they do only on serialization).

[github-actions]

Thank you for your report; the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.