"Remove blank strings" add-on removes only some strings in WebExtension JSON

[avm99963]

Describe the issue

When enabling the "Remove blank strings" add-on in a component with WebExtension JSON format, only half of the empty strings are removed.

I already tried

• I've read and searched the documentation.
• I've searched for similar issues in this repository.

Steps to reproduce the behavior

1. Create a WebExtension JSON component in a Git project.
2. Enable the "Remove blank strings" add-on in the component.
3. Start a translation for a new language.
   • This will create a commit with a new file which has all the strings set to be empty.
4. Translate one string and save it.
   • This will create a commit to update the translated string.
   • Also, another commit will be created by the "Remove blank strings" add-on to remove empty strings (see the "Expected behavior" section).

Expected behavior

I would expect the last commit to remove all strings except the only one I translated.

Instead, only half of the strings are removed by the add-on, as can be seen in the following sample commit (produced after following the reproduction steps):

commit ffd69279ecc6cccf52242e3f0e89239d9a61ae8e (HEAD, weblate/master)
Author: Weblate bot <weblate-bot@avm99963.com>
Date:   Sat Jul 8 10:59:01 2023 +0000

    Update translation files

    Updated by "Remove blank strings" hook in Weblate.

    Translation: TW Power Tools/Extension strings (JSON format)
    Translate-URL: https://i18n.avm99963.com/projects/tw-power-tools/extension-strings-json-format/
    Change-Id: I2f420d0f3200baacbba821597c4718db85e413ca

diff --git src/static/_locales/de/messages.json src/static/_locales/de/messages.json
index ce5a2ac..68f3373 100644
--- src/static/_locales/de/messages.json
+++ src/static/_locales/de/messages.json
@@ -1,604 +1,300 @@
 {
     "appName": {
         "message": "Test",
         "description": "The extension's name"
     },
-    "appNameBeta": {
-        "message": "",
-        "description": "The extension's name for the beta version"
-    },
     "appNameCanary": {
         "message": "",
         "description": "The extension's name for the beta version"
     },
-    "appNameGecko": {
-        "message": "",
-        "description": "The extension's name (with a 50 character limit for Gecko)"
-    },
     "appDescription": {
         "message": "",
         "description": "The extension's description"
     },
-    "options_killswitchwarning": {
-        "message": "",
-        "description": "Warning shown in the options page if a feature has been remotely force disabled via the kill switch mechanism. This happens when a feature doesn't work properly and might be doing dangerous things, and the maintainer or an authorized user remotely force disables the feature."
-    },
     "options_killswitchenabled": {
         "message": "",
         "description": "Warning shown in the options page below an option, when it has been remotely force disabled via the kill switch mechanism."
     },
-    "options_optionalpermissionswarning_header": {
-        "message": "",
-        "description": "Warning shown at the top of the options page if a feature cannot be used because one or more required permissions haven't been granted to the extension."

[...]

Screenshots

N/A

Exception traceback

N/A

How do you run Weblate?

Docker container

Weblate versions

• Weblate: 4.18.2
• Django: 4.2.2
• siphashc: 2.1
• translate-toolkit: 3.9.2
• lxml: 4.9.2
• Pillow: 9.5.0
• nh3: 0.2.13
• python-dateutil: 2.8.2
• social-auth-core: 4.4.2
• social-auth-app-django: 5.2.0
• django-crispy-forms: 2.0
• oauthlib: 3.2.2
• django-compressor: 4.4
• djangorestframework: 3.14.0
• django-filter: 23.2
• django-appconf: 1.0.5
• user-agents: 2.2.0
• filelock: 3.12.2
• rapidfuzz: 3.1.1
• openpyxl: 3.1.2
• celery: 5.3.1
• django-celery-beat: 2.5.0
• kombu: 5.3.1
• translation-finder: 2.15
• weblate-language-data: 2023.5
• html2text: 2020.1.16
• pycairo: 1.24.0
• PyGObject: 3.44.1
• diff-match-patch: 20230430
• requests: 2.31.0
• django-redis: 5.3.0
• hiredis: 2.2.3
• sentry-sdk: 1.26.0
• Cython: 0.29.35
• misaka: 2.1.1
• GitPython: 3.1.31
• borgbackup: 1.2.4
• pyparsing: 3.0.9
• pyahocorasick: 2.0.0
• python-redis-lock: 4.0.0
• charset-normalizer: 3.1.0
• Python: 3.11.4
• Git: 2.30.2
• psycopg2: 2.9.6
• phply: 1.2.6
• ruamel.yaml: 0.17.32
• tesserocr: 2.6.0
• boto3: 1.26.164
• zeep: 4.2.1
• aeidon: 1.12
• iniparse: 0.5
• mysqlclient: 2.2.0
• Mercurial: 6.4.5
• git-svn: 2.30.2
• git-review: 2.3.1
• Redis server: 7.0.11
• PostgreSQL server: 15.3
• Database backends: django.db.backends.postgresql
• Cache backends: default:RedisCache, avatar:FileBasedCache
• Email setup: django.core.mail.backends.smtp.EmailBackend: CENSORED
• OS encoding: filesystem=utf-8, default=utf-8
• Celery: redis://redis:6379/1, redis://redis:6379/1, regular
• Platform: Linux 5.15.0-69-generic (x86_64)

Weblate deploy checks

System check identified some issues:

INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
	HINT: https://docs.weblate.org/en/weblate-4.18.2/admin/install.html#collecting-errors

System check identified 1 issue (1 silenced).

Additional context

No response

[nijel]

Does it remove more in consequent runs, or the other strings are never removed?

[avm99963]

Hey @nijel!

It removes more in consequent runs.

I just added a new translation and from the remaining strings half of them were removed. So, as I assume you were trying to confirm, there might be a bug iterating the translations which makes it skip a translation for each iteration in which it removes a string.

I don't have much experience with Python and its technical intricacies, but maybe removing translations messes up with the iterator here and thus some elements are skipped:

weblate/weblate/addons/base.py

Lines 353 to 357 in 25fc7df

	@staticmethod
	def iterate_translations(component):
	for translation in component.translation_set.iterator():
	if not translation.is_source or component.intermediate:
	yield translation

Here are the 2 commits which have appeared in my last test (after I manually committed the changes from the "Repository maintenance" page):

First commit

commit 8cb5863e9090b6484c41c4c0d6069bb7502bfcec
Author: Adrià <CENSORED_EMAIL>
Date:   Mon Jul 10 16:18:01 2023 +0000

    Weblate: Update German (de) localization

    Currently translated at 1.3% (2 of 143 strings)

    Translation: TW Power Tools/Extension strings (JSON format)
    Translate-URL: https://i18n.avm99963.com/projects/tw-power-tools/extension-strings-json-format/de/
    Change-Id: Ib7aea7070b1cd003a7410f75b58e93594a2cb15c

diff --git a/src/static/_locales/de/messages.json b/src/static/_locales/de/messages.json
index 68f3373..3f7cc07 100644
--- a/src/static/_locales/de/messages.json
+++ b/src/static/_locales/de/messages.json
@@ -4,7 +4,7 @@
         "description": "The extension's name"
     },
     "appNameCanary": {
-        "message": "",
+        "message": "Test 2",
         "description": "The extension's name for the beta version"
     },
     "appDescription": {

Second commit

commit ab9d2d5ad1e3a67f9fffe13c4304d9c7e37f773e (weblate/master)
Author: Weblate bot <weblate-bot@avm99963.com>
Date:   Mon Jul 10 16:18:26 2023 +0000

    Update translation files

    Updated by "Remove blank strings" hook in Weblate.

    Translation: TW Power Tools/Extension strings (JSON format)
    Translate-URL: https://i18n.avm99963.com/projects/tw-power-tools/extension-strings-json-format/
    Change-Id: Ia1af2a162beecc31139d79c2719a760ec9ed2b0a

diff --git src/static/_locales/de/messages.json src/static/_locales/de/messages.json
index 3f7cc07..5e7caf5 100644
--- src/static/_locales/de/messages.json
+++ src/static/_locales/de/messages.json
@@ -1,300 +1,150 @@
 {
     "appName": {
         "message": "Test",
         "description": "The extension's name"
     },
     "appNameCanary": {
         "message": "Test 2",
         "description": "The extension's name for the beta version"
     },
-    "appDescription": {
-        "message": "",
-        "description": "The extension's description"
-    },
     "options_killswitchenabled": {
         "message": "",
         "description": "Warning shown in the options page below an option, when it has been remotely force disabled via the kill switch mechanism."
     },
-    "options_optionalpermissionswarning_label": {
-        "message": "",
-        "description": "Warning shown in the options page below an option, when a feature needs more permissions to work."
-    },
     "options_minorenhancements": {
         "message": "",
         "description": "Heading for several options that can be enabled in the options page."
     },
-    "options_thread": {
-        "message": "",
-        "description": "Feature checkbox in the options page"
-    },
     "options_fixedtoolbar": {
         "message": "",

[...]

[nijel]

The translation iteration you link is iterating over component languages, so that is definitely not the root cause here.

But this code has the issue of iterating over a list that is being changed:

weblate/weblate/formats/base.py

Lines 729 to 751 in 563b2d9

	def cleanup_blank(self) -> List[str]:
	"""
	Removes strings without translations.
	Returning list of additional changed files.
	"""
	changed = False
	result = []
	for ttkit_unit in self.all_store_units:
	target = self.unit_class(self, ttkit_unit, ttkit_unit).target
	if not target or (isinstance(target, list) and not any(target)):
	item = self.delete_unit(ttkit_unit)
	if item is not None:
	result.append(item)
	else:
	changed = True
	if changed:
	self.save()
	self._invalidate_units()
	return result

[github-actions]

Thank you for your report; the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.

[avm99963]

Awesome, thank you so much @nijel :))