Sources are not in sync in all the languages

[adriandiazgar]

Describe the issue

When adding a new source unit, the added unit is not available for the rest of the languages

while for another language

To Reproduce the issue

Steps to reproduce the behavior:

1. Go to the source language, add a new source string
2. Go to another language (not source), and look for the added string
3. See an error, the string is not found for the language from point nr 2

Expected behavior

Once a unit source is added to the source language, it should be present in all the other languages, or at least the addition of the new source language should give a mechanism to track when the unit will be added, otherwise, we can't know when it is possible to add the translation.

Screenshots

while for another language

Server configuration and status

Weblate installation: weblate.org service / Docker / PyPI / other

 * Weblate: 4.5.2
 * Django: 3.1.7
 * siphashc: 2.1
 * translate-toolkit: 3.3.3
 * lxml: 4.6.3
 * Pillow: 8.1.2
 * bleach: 3.3.0
 * python-dateutil: 2.8.1
 * social-auth-core: 4.1.0
 * social-auth-app-django: 4.0.0
 * django-crispy-forms: 1.11.2
 * oauthlib: 3.1.0
 * django-compressor: 2.4
 * djangorestframework: 3.12.3
 * django-filter: 2.4.0
 * django-appconf: 1.0.4
 * user-agents: 2.2.0
 * filelock: 3.0.12
 * setuptools: 40.8.0
 * jellyfish: 0.8.2
 * openpyxl: 3.0.7
 * celery: 5.0.5
 * kombu: 5.0.2
 * translation-finder: 2.9
 * weblate-language-data: 2021.3
 * html2text: 2020.1.16
 * pycairo: 1.16.2
 * pygobject: 3.30.4
 * diff-match-patch: 20200713
 * requests: 2.25.1
 * django-redis: 4.12.1
 * hiredis: 1.1.0
 * sentry_sdk: 1.0.0
 * Cython: 0.29.22
 * misaka: 2.1.1
 * GitPython: 3.1.14
 * borgbackup: 1.1.16
 * pyparsing: 2.4.7
 * pyahocorasick: 1.4.1
 * Python: 3.7.3
 * Git: 2.20.1
 * psycopg2: 2.8.6
 * psycopg2-binary: 2.8.6
 * phply: 1.2.5
 * chardet: 4.0.0
 * ruamel.yaml: 0.16.13
 * tesserocr: 2.5.1
 * akismet: 1.1
 * boto3: 1.17.38
 * zeep: 4.0.0
 * aeidon: 1.9
 * iniparse: 0.5
 * mysqlclient: 2.0.3
 * Mercurial: 5.7.1
 * git-svn: 2.20.1
 * git-review: 2.0.0
 * Redis server: 3.0.6
 * PostgreSQL server: 10.12
 * Database backends: django.db.backends.postgresql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Email setup: django.core.mail.backends.smtp.EmailBackend: REDACTED
 * OS encoding: filesystem=utf-8, default=utf-8
 * Celery: REDACTED, regular
 * Platform: Linux 4.15.0-123-generic (x86_64)

Weblate deploy checks

SystemCheckError: System check identified some issues:

CRITICALS:
?: (weblate.E019) The Celery does not process tasks, or is too slow in processing them.
	HINT: https://docs.weblate.org/en/weblate-4.5.2/admin/install.html#celery

WARNINGS:
?: (security.W018) You should not have DEBUG set to True in deployment.
?: (security.W019) You have 'django.middleware.clickjacking.XFrameOptionsMiddleware' in your MIDDLEWARE, but X_FRAME_OPTIONS is not set to 'DENY'. Unless there is a good reason for your site to serve other parts of itself in a frame, you should change it to 'DENY'.

INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
	HINT: https://docs.weblate.org/en/weblate-4.5.2/admin/install.html#collecting-errors
?: (weblate.I028) Backups are not configured, it is highly recommended for production use
	HINT: https://docs.weblate.org/en/weblate-4.5.2/admin/backup.html

System check identified 5 issues (1 silenced).

Additional context

Our server is under heavy traffic right now, which would explain why we see the CRITICAL alerts from check --deploy command.

However, this should not interfere with adding translations and source units.

[nijel]

What component setup do you use? Bilingual or monolingual? Which file format?

Also, the debug mode will contribute to increased server load, consider disabling it.

[adriandiazgar]

Hi @nijel thanks for your quick response.

The project is monolingual and the format is PO gettext.

This is an staging environment that's why the debug is enabled, in production that is not enabled.

[nijel]

Unfortunately, I can't reproduce this. For me, the string is properly added to all languages. Do you look for the newly added string immediatelly or once it is commited to the repository? (I tried both and it is always present)

There was bug in committing added strings in some corner cases and that was fixed in b32bb96, but I don't think it applies here. Anyway you would see error in the logs in case it would.

[adriandiazgar]

Hi @nijel

Digging more into the problem we discovered that when adding a new source string the repository wasn't committed automatically (Maybe that's a bug?), that behavior wasn't present in the previous version.

Therefore when adding a new translation (via posting an entire PO file to the endpoint /api/translations/project/component/es/file/) was getting rid of the newly added translations because they weren't present in the repository.

Doing a commit after adding the source string made it work, but still, I think that doesn't seem like proper behavior.

Can you please illustrate more here?

Thanks.

[nijel]

It's expected that it's not commited, but it should be committed when needed... How do you upload the new translation to /file/ API (which method do you use)?

PS: I've just looked into all code paths in the upload, and they seem to trigger the commit when needed.

[adriandiazgar]

So the workflow is:

1- We add the source unit (individually), for instance: phrase 1
2- We upload the entire translation file including the translation for the previously added source string, the call used to upload the file is:

{
	'headers': {
		'user-agent': 'wlc/1.11',
		'Accept': 'application/json',
		'Authorization': 'Token XXXXXX'
	},
	'verify': True,
	'files': {
		'file': ('file.po', '#\nmsgid ""\nmsgstr ""\n"Project-Id-Version: PACKAGE VERSION\\n"\n"Report-Msgid-Bugs-To: \\n"\n"POT-Creation-Date: 2021-03-31 09:00+0000\\n"\n"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\\n"\n"Last-Translator: Automatically generated\\n"\n"Language-Team: none\\n"\n"Language: es\\n"\n"MIME-Version: 1.0\\n"\n"Content-Type: text/plain; charset=UTF-8\\n"\n"Content-Transfer-Encoding: 8bit\\n"\n"X-Generator: Translate Toolkit 3.3.3\\n"\n\nmsgid "phrase 1"\nmsgstr "phrase 1"\n\nmsgid "phrase 2"\nmsgstr "phrase 2 es"\n')
	},
	'data': {
		'method': 'replace',
		'overwrite': 'yes'
	},
}

[nijel]

Is your component really configured as monolingual? Is Monolingual base language file configured?

[adriandiazgar]

Yes, it is configured as monolingual

Translation process | Translations can be made directly.Translation suggestions are turned off.Any authenticated user can contribute.The translation uses monolingual files.The translation base language is editable.
-- | --

Monolingual base language file | dra401_test/en

[nijel]

Thanks, I was able to reproduce the issue now. It will be fixed in 4.5.3 this week.

[github-actions]

Thank you for your report; the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.