Translations lost/overwritten without notification

[swurzinger]

Describe the issue

Translation changes performed in the Weblate UI are visible in the history list in the Weblate-UI, but the translations have never been committed to the weblate repository (checked logs for "committing pending changes" and checked git reflog of repo in /app/data/vcs/... ). An external update vom VCS seems to have overwritten the changes (see logs below). Commit timeout was set to 2 hours.

I already tried

Describe the steps you tried to solve the problem yourself.

• I've searched the weblate-logs for errors or any other useful information, however I haven't been able to find any
• I've walked through git reflog of weblate-internal git repository, it doesn't contain any commit with a change
• I've check the history views in weblate-ui. both history view for the specific translation key and the component do contain the translation change

To Reproduce the issue

unfortunately I haven't been able to find out the steps to reproduce, but below are the events as seen in the logs and history view as they occured first. exactly the same issue occured again on the next day (translation changed in the morning, change lost at the end of the day).

at around 06:22 (UTC) translations for cs_CZ and sk_SK (structured JSON; de_AT is source) were changed:

at around 07:30 (UTC) older translation changes (de_DE and de_AT only) have been merged:
[2021-05-19 07:30:56,041: INFO/443] project/app-ui: repository updated from a7699fc558873447586cc0d3c4905fcb5bdb4b2a to 718c431cfaa16b4947c55d4da46e06217608ecf1

note that nothing changed, especially CZ/SK translations didn't, and still we get in the log:
[2021-05-19 07:30:56,687: INFO/443] project/app-ui/cs: processing server/src/main/resources/static/i18n/cs_CZ.json, content changed

I can't tell exactly when the CZ/SK changes were lost, because the neither the log nor the history view in weblate-UI contain an entry for the change - I assume that the sync from VCS above overwrote the change somehow.

Situation afterwards is that CZ/SK translation have their original values, as before the changes, but the history still shows the change. So that latest change as seen in the history doesn't match the actual translation.

Expected behavior

No changes get lost. In this specific case the Rebase should not have triggered any changes. In general changes should never be lost, but be merged or result in conflicts if merging isn't possible.

Server configuration and status

Weblate installation: openshift

weblate list_versions
 * Weblate: 4.6.2
 * Django: 3.2.2
 * siphashc: 2.1
 * translate-toolkit: 3.3.6
 * lxml: 4.6.3
 * Pillow: 8.2.0
 * bleach: 3.3.0
 * python-dateutil: 2.8.1
 * social-auth-core: 4.1.0
 * social-auth-app-django: 4.0.0
 * django-crispy-forms: 1.11.2
 * oauthlib: 3.1.0
 * django-compressor: 2.4.1
 * djangorestframework: 3.12.4
 * django-filter: 2.4.0
 * django-appconf: 1.0.4
 * user-agents: 2.2.0
 * filelock: 3.0.12
 * setuptools: 40.8.0
 * jellyfish: 0.8.2
 * openpyxl: 3.0.7
 * celery: 5.0.5
 * kombu: 5.0.2
 * translation-finder: 2.9
 * weblate-language-data: 2021.4
 * html2text: 2020.1.16
 * pycairo: 1.16.2
 * pygobject: 3.30.4
 * diff-match-patch: 20200713
 * requests: 2.25.1
 * django-redis: 4.12.1
 * hiredis: 2.0.0
 * sentry_sdk: 1.1.0
 * Cython: 0.29.23
 * misaka: 2.1.1
 * GitPython: 3.1.14
 * borgbackup: 1.1.16
 * pyparsing: 2.4.7
 * pyahocorasick: 1.4.2
 * python-redis-lock: 3.7.0
 * Python: 3.7.3
 * Git: 2.20.1
 * psycopg2: 2.8.6
 * psycopg2-binary: 2.8.6
 * phply: 1.2.5
 * chardet: 4.0.0
 * ruamel.yaml: 0.17.4
 * tesserocr: 2.5.1
 * akismet: 1.1
 * boto3: 1.17.69
 * zeep: 4.0.0
 * aeidon: 1.9
 * iniparse: 0.5
 * mysqlclient: 2.0.3
 * Mercurial: 5.8
 * git-svn: 2.20.1
 * git-review: 2.1.0
 * Redis server: 3.2.13
 * PostgreSQL server: 9.6.10
 * Database backends: django.db.backends.postgresql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Email setup: django.core.mail.backends.smtp.EmailBackend: localhost
 * OS encoding: filesystem=utf-8, default=utf-8
 * Celery: redis://:pJ4mvDTdTjvw4re0@weblate-cache:6379/1, redis://:pJ4mvDTdTjvw4re0@weblate-cache:6379/1, regular
 * Platform: Linux 4.18.0-193.47.1.el8_2.x86_64 (x86_64)

Additional context

log excerpt, showing the update from external VCS

[2021-05-19 07:30:57,820: INFO/ForkPoolWorker-1] project/app-ui/sk: updating completed
[2021-05-19 07:30:57,820: INFO/443] project/app-ui/sk: updating completed
[2021-05-19 07:30:57,261: INFO/443] project/app-ui/sk: processing server/src/main/resources/static/i18n/sk_SK.json, content changed
[2021-05-19 07:30:57,261: INFO/ForkPoolWorker-1] project/app-ui/sk: processing server/src/main/resources/static/i18n/sk_SK.json, content changed
[2021-05-19 07:30:57,241: INFO/443] project/app-ui: checking server/src/main/resources/static/i18n/sk_SK.json (sk_SK) [4/4]
[2021-05-19 07:30:57,241: INFO/ForkPoolWorker-1] project/app-ui: checking server/src/main/resources/static/i18n/sk_SK.json (sk_SK) [4/4]
[2021-05-19 07:30:57,221: INFO/MainProcess] Received task: weblate.memory.tasks.handle_unit_translation_change[0dce0231-1048-4bda-9b85-ddcde0727abe]  
[2021-05-19 07:30:57,220: INFO/MainProcess] Received task: weblate.accounts.tasks.notify_change[9dec7414-ee5b-4c08-aa4d-e1f25e538733]  
[2021-05-19 07:30:57,219: INFO/MainProcess] Received task: weblate.memory.tasks.handle_unit_translation_change[975890be-4d33-4cbc-a9fa-a14c89caef30]  
[2021-05-19 07:30:57,218: INFO/ForkPoolWorker-1] project/app-ui: checking server/src/main/resources/static/i18n/de_DE.json (de_DE) [3/4]
[2021-05-19 07:30:57,218: INFO/443] project/app-ui: checking server/src/main/resources/static/i18n/de_DE.json (de_DE) [3/4]
[2021-05-19 07:30:57,218: INFO/MainProcess] Received task: weblate.memory.tasks.handle_unit_translation_change[7905c430-0c78-4eae-879e-34cf29cde334]  
[2021-05-19 07:30:57,216: INFO/MainProcess] Received task: weblate.memory.tasks.handle_unit_translation_change[2ffbee05-2364-4063-ba53-3431c71c56b3]  
[2021-05-19 07:30:57,213: INFO/MainProcess] Received task: weblate.memory.tasks.handle_unit_translation_change[3e0fa038-a74a-4413-aa81-e7ede8647b55]  
[2021-05-19 07:30:57,209: INFO/MainProcess] Received task: weblate.memory.tasks.handle_unit_translation_change[31a06276-043b-4e5f-ab41-b4852aced212]  
[2021-05-19 07:30:57,180: INFO/ForkPoolWorker-1] project/app-ui/cs: updating completed
[2021-05-19 07:30:57,180: INFO/443] project/app-ui/cs: updating completed
[2021-05-19 07:30:56,687: INFO/443] project/app-ui/cs: processing server/src/main/resources/static/i18n/cs_CZ.json, content changed
[2021-05-19 07:30:56,687: INFO/ForkPoolWorker-1] project/app-ui/cs: processing server/src/main/resources/static/i18n/cs_CZ.json, content changed
[2021-05-19 07:30:56,667: INFO/ForkPoolWorker-1] project/app-ui: checking server/src/main/resources/static/i18n/cs_CZ.json (cs_CZ) [2/4]
[2021-05-19 07:30:56,667: INFO/443] project/app-ui: checking server/src/main/resources/static/i18n/cs_CZ.json (cs_CZ) [2/4]
[2021-05-19 07:30:56,437: INFO/ForkPoolWorker-1] project/app-ui: checking server/src/main/resources/static/i18n/de_AT.json (de_AT) [1/4]
[2021-05-19 07:30:56,437: INFO/443] project/app-ui: checking server/src/main/resources/static/i18n/de_AT.json (de_AT) [1/4]



[2021-05-19 07:30:56,152: INFO/MainProcess] Received task: weblate.accounts.tasks.notify_change[ad603bce-a847-411d-9e92-41b88de69dfa]  
[2021-05-19 07:30:56,141: INFO/ForkPoolWorker-1] project/app-ui: rebase remote into repo 1bbdd02b802f11ec565c8a6bc26e5afff8fa07b3..1bbdd02b802f11ec565c8a6bc26e5afff8fa07b3
[2021-05-19 07:30:56,141: INFO/443] project/app-ui: rebase remote into repo 1bbdd02b802f11ec565c8a6bc26e5afff8fa07b3..1bbdd02b802f11ec565c8a6bc26e5afff8fa07b3
[2021-05-19 07:30:56,041: INFO/443] project/app-ui: repository updated from a7699fc558873447586cc0d3c4905fcb5bdb4b2a to 718c431cfaa16b4947c55d4da46e06217608ecf1
[2021-05-19 07:30:56,041: INFO/ForkPoolWorker-1] project/app-ui: repository updated from a7699fc558873447586cc0d3c4905fcb5bdb4b2a to 718c431cfaa16b4947c55d4da46e06217608ecf1
[2021-05-19 07:30:56,030: INFO/443] project/app-ui: update took 0.61 seconds
[2021-05-19 07:30:56,030: INFO/ForkPoolWorker-1] project/app-ui: update took 0.61 seconds
[2021-05-19 07:30:55,416: INFO/ForkPoolWorker-1] project/app-ui: updating repository
[2021-05-19 07:30:55,416: INFO/443] project/app-ui: updating repository
10.128.0.1 - - [19/May/2021:07:30:55 +0000] "POST /hooks/gitlab/ HTTP/1.1" 200 222 "-" "GitLab/13.11.3"
[pid: 464|app: 0|req: 323/5140] 10.128.0.1 () {48 vars in 764 bytes} [Wed May 19 07:30:55 2021] POST /hooks/gitlab/ => generated 211 bytes in 84 msecs (HTTP/1.1 200) 8 headers in 436 bytes (1 switches on core 0)
[2021-05-19 07:30:55,391: INFO/MainProcess] Received task: weblate.trans.tasks.perform_update[716ee271-5700-4da4-8ef2-0788cb1bc5d7]  
[2021-05-19 07:30:55,388: INFO/MainProcess] Received task: weblate.accounts.tasks.notify_change[88e566e8-84c0-47a8-a42a-b33429d42610]  
[2021-05-19 07:30:55,347: INFO/464] GitLab notification will update Project/app-ui

[Shide]

I think this can happen if on a component, you have activated (it's activated by default) Allow translation propagation.
Unless you have a complete sense of what will happen if you translate something on a component that has the same text than another, I recommend you to deactivate this option.

I have a lot of non wanted auto propagation translations and it's a pain in the ass.

The lack of the merge maybe it's because the changes happens in the Age of changes to commit window

[swurzinger]

Thanks for your hints, but translation propagation has been disabled, so that shouldn't have been the cause. Can I somehow see that in the logs whether the translation has been updated via translation propagation, to be sure that this isn't the issue?

As mentioned above the "Age of changes to commit" was set to 2 hours. But even in case that time didn't fully pass yet and the changes therefore weren't committed yet Weblate should not (and normally does not) lose changes if it detects external changes and rebases its internal repo.

[nijel]

The expected behaviour is that Weblate commits any pending changes before merging upstream changes (it looks at git diff --name-status ..origin/main to figure out whether commit is needed).

What could discard these changes is doing reset either via API or from the repository management page. I don't think this would be visible in the logs, but it would be visible in the component history.

The other reason might a bug in the code - in case you see "disappeared string" entries in the log, that's an indication of that.

[swurzinger]

If I understand that correctly, that means the changes should have been committed to the repository, which most likely did not happen (otherwise I would expect a hint about this in git reflog or in Weblate logs). So the question is why that didn't happen.
If a change is saved (user changes translation and clicks "Save") I guess this is not directly written to disk, is it?

No manual reset has been performed and there is no such entry in the component history.

I didn't see any translation string being contained in the logs (so e.g. no occurrence of "Vlasný výber" or any other translation as far as I saw). If you mean "disappeared string" literally, I couldn't find that either, but unfortunately I don't have full logs anymore by now, so I can't tell for sure.

[nijel]

The translation is first stored in the database only. It is committed later (see https://docs.weblate.org/en/latest/admin/continuous.html#lazy-commits).

I meant "disappeared string" literally, it would be followed by the source string. It is logged here:

weblate/weblate/trans/models/translation.py

Line 663 in 4760786

self.log_error("disappeared string: %s", unit)

[github-actions]

This issue has been automatically marked as stale because there wasn’t any recent activity.

It will be closed soon if no further action occurs.

Thank you for your contributions!