Changes in weblate are not written to actuall repository files

[desmax]

Not sure what happened, just yesterday all was working fine and now when I edit something in weblate it is not reflected in repository files, just nothing written to data folder.
I recieve no warnings, logs seem clean. What can I check?
Im on recent master version.

* Weblate weblate-3.1.1-581-ga3e3253
 * Python 2.7.9
 * Django 1.11.4
 * Celery 4.2.1
 * celery-batches 0.2
 * six 1.10.0
 * social-auth-core 1.4.0
 * social-auth-app-django 2.1.0
 * django-appconf 1.0.2
 * translate-toolkit 2.3.0
 * Whoosh 2.7.4
 * defusedxml 0.5.0
 * Git 2.1.4
 * Pillow 4.2.1
 * python-dateutil 2.6.1
 * lxml 3.8.0
 * django-crispy-forms 1.6.1
 * django_compressor 2.1.1
 * djangorestframework 3.8.2
 * user-agents 1.1.0
 * jellyfish 0.6.1
 * pytz 2017.2
 * pyuca 1.1.2
 * python-bidi 0.4.0
 * PyYAML 3.12
 * Mercurial 4.2.2
 * Database backends: django.db.backends.mysql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Platform: Linux 2.6.32-042stab127.2 (x86_64)

./manage.py check --deploy output:

SystemCheckError: System check identified some issues:

ERRORS:
?: (weblate.E005) Celery is configured in the eager mode
	HINT: https://docs.weblate.org/en/latest/admin/install.html#celery
?: (weblate.E014) The cookie secret key has default value
	HINT: https://docs.weblate.org/en/latest/admin/install.html#production-secret

WARNINGS:
?: (security.W004) You have not set a value for the SECURE_HSTS_SECONDS setting. If your entire site is served only over SSL, you may want to consider setting a value and enabling HTTP Strict Transport Security. Be sure to read the documentation first; enabling HSTS carelessly can cause serious, irreversible problems.
?: (security.W006) Your SECURE_CONTENT_TYPE_NOSNIFF setting is not set to True, so your pages will not be served with an 'x-content-type-options: nosniff' header. You should consider enabling this header to prevent the browser from identifying content types incorrectly.
?: (security.W007) Your SECURE_BROWSER_XSS_FILTER setting is not set to True, so your pages will not be served with an 'x-xss-protection: 1; mode=block' header. You should consider enabling this header to activate the browser's XSS filtering and help prevent XSS attacks.
?: (security.W008) Your SECURE_SSL_REDIRECT setting is not set to True. Unless your site should be available over both SSL and non-SSL connections, you may want to either set this setting True or configure a load balancer or reverse-proxy server to redirect all connections to HTTPS.
?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only session cookie makes it more difficult for network traffic sniffers to hijack user sessions.
?: (security.W019) You have 'django.middleware.clickjacking.XFrameOptionsMiddleware' in your MIDDLEWARE, but X_FRAME_OPTIONS is not set to 'DENY'. The default is 'SAMEORIGIN', but unless there is a good reason for your site to serve other parts of itself in a frame, you should change it to 'DENY'.

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[nijel]

The changes are written when needed, not immediately. You can force commit in the repository maintenance page or using ./manage.py commitgit.

[desmax]

Sorry I did not say that I was pushing commit button, which I guess is equal to ./manage.py commitgit.
Anyways just now I figured out that if I delete and add again some component it starts working as expected. I recently updated from 2.12-master to 3.2-master. Any ideas? do not want to readd all of components.. there are quite a few.

[nijel]

Do you have setup celery if you're running master? Missing it might lead to some issues (though the commit button should still work).

Does the repository maintenance tell something about pending changes?

[desmax]

@nijel I did start celery via supervisor like this celery -A weblate worker --loglevel=WARNING -B --concurrency=10 but I put into settings this

CELERY_TASK_ALWAYS_EAGER = True
CELERY_BROKER_URL = 'memory://'

since I had some issues figuring out how to properly set it up.

Repository maintenance says that I have some uncommited changes after edit, but commit button just brings repository back to clean state without actually commiting anything.
If I do commitgit --all command after some change I get this

INFO transport-talent/applicationForwardedEmail/en: committing pending changes (manage commitgit)
ERROR transport-talent/applicationForwardedEmail/en: message Transport Talent/applicationForwardedEmail - English, string 1 disappeared!

[nijel]

The recommended setup is using redis as backend, see https://docs.weblate.org/en/latest/admin/install.html#celery

Looks like the translations are somehow incosistent, can you try manage.py loadpo --force transport-talent/applicationForwardedEmail? What file format do you use?

[desmax]

@nijel I use xliff and the command gave me this

INFO transport-talent/applicationForwardedEmail: checking applicationForwardedEmail/applicationForwardedEmail.da.xliff (da) [1/9]
INFO transport-talent/applicationForwardedEmail/da: processing applicationForwardedEmail/applicationForwardedEmail.da.xliff, check forced
INFO transport-talent/applicationForwardedEmail: checking applicationForwardedEmail/applicationForwardedEmail.de.xliff (de) [2/9]
INFO transport-talent/applicationForwardedEmail/de: processing applicationForwardedEmail/applicationForwardedEmail.de.xliff, check forced
INFO transport-talent/applicationForwardedEmail: checking applicationForwardedEmail/applicationForwardedEmail.el.xliff (el) [3/9]
INFO transport-talent/applicationForwardedEmail/el: processing applicationForwardedEmail/applicationForwardedEmail.el.xliff, check forced
INFO transport-talent/applicationForwardedEmail: checking applicationForwardedEmail/applicationForwardedEmail.en.xliff (en) [4/9]
INFO transport-talent/applicationForwardedEmail/en: processing applicationForwardedEmail/applicationForwardedEmail.en.xliff, check forced
INFO transport-talent/applicationForwardedEmail: checking applicationForwardedEmail/applicationForwardedEmail.es.xliff (es) [5/9]
INFO transport-talent/applicationForwardedEmail/es: processing applicationForwardedEmail/applicationForwardedEmail.es.xliff, check forced
INFO transport-talent/applicationForwardedEmail: checking applicationForwardedEmail/applicationForwardedEmail.fr.xliff (fr) [6/9]
INFO transport-talent/applicationForwardedEmail/fr: processing applicationForwardedEmail/applicationForwardedEmail.fr.xliff, check forced
INFO transport-talent/applicationForwardedEmail: checking applicationForwardedEmail/applicationForwardedEmail.nl.xliff (nl) [7/9]
INFO transport-talent/applicationForwardedEmail/nl: processing applicationForwardedEmail/applicationForwardedEmail.nl.xliff, check forced
INFO transport-talent/applicationForwardedEmail: checking applicationForwardedEmail/applicationForwardedEmail.ru.xliff (ru) [8/9]
INFO transport-talent/applicationForwardedEmail/ru: processing applicationForwardedEmail/applicationForwardedEmail.ru.xliff, check forced
INFO transport-talent/applicationForwardedEmail: checking applicationForwardedEmail/applicationForwardedEmail.sv.xliff (sv) [9/9]
INFO transport-talent/applicationForwardedEmail/sv: processing applicationForwardedEmail/applicationForwardedEmail.sv.xliff, check forced
INFO transport-talent/applicationForwardedEmail: updating completed

After this command everything started to work as expected. Thank you.

another thing is that my context fields started to look like this:
file.ext///fd1c43aaf1225902e33d8493f1143970

My translation file look like this:

<?xml version='1.0' encoding='UTF-8'?>
<xliff xmlns="urn:oasis:names:tc:xliff:document:1.2" version="1.2">
  <file source-language="en-US" target-language="en" datatype="plaintext" original="file.ext">
    <header>
      <tool tool-id="symfony" tool-name="Symfony"/>
    </header>
    <body>
      <trans-unit id="0e6c357a3e2248f788b26d7b803606f8" resname="app.application.salutation" approved="yes">
        <source>app.application.salutation</source>
        <target state="translated">Dear %firstname%</target>
      </trans-unit>
    </body>
  </file>
</xliff>

[nijel]

The handling of xliff files has changed at some point and it leads to this change. That's also why the updates could not be properly written. It seems that this change was not properly documented in the upgrade instructions, I'm really sorry for that.

[desmax]

@nijel How do I see actual key in weblate now? the one I use in my app (ie app.application.salutation). Also earlier I used to search by these keys in weblate, it does not work anymore.

[nijel]

You mean resname or source? The problem with it is that it doesn't have to be unique in the file, while id does have to.

You probably should be creating your translation component as monolingual, are you doing that?

[desmax]

Yes, I use monolingual approach.
I want to see resname in weblate and search by it. (I also put into source tag same value so not sure which weblate has been using earlier).

My IDs are unique inside language file, though same key has equal IDs accross language files inside component.

[nijel]

Looking at the specs it probably makes sense to use resname instead of id if that is available. That has to be fixed in the translate-toolkit though.

I think following patch could do the trick, though I don't have currently time to test it:

diff --git a/translate/storage/xliff.py b/translate/storage/xliff.py
index e3af1432..3e67d580 100644
--- a/translate/storage/xliff.py
+++ b/translate/storage/xliff.py
@@ -505,7 +505,7 @@ class xliffunit(lisa.LISAunit):
             # unit has no proper file ancestor, probably newly created
             pass
         # hide the fact that we sanitize ID_SEPERATOR
-        uid += six.text_type(self.xmlelement.get("id") or u"").replace(ID_SEPARATOR_SAFE, ID_SEPARATOR)
+        uid += six.text_type(self.xmlelement.get("resname") or self.xmlelement.get("id") or u"").replace(ID_SEPARATOR_SAFE, ID_SEPARATOR)
         return uid
 
     def addlocation(self, location):

[desmax]

@nijel That does work, thank you for the great help.

[nijel]

It will certainly work for loading, but I'm not sure if saving will work with this hack properly, please test that before thanking :-).

[desmax]

@nijel It does break saving. Now I noticed.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[nijel]

Reopening as somebody else has ran into this issue as well: #2380

[nijel]

Thank you for your report, the issue you have reported has just been fixed.

• In case you see problem with the fix, please comment on this issue.
• In case you see similar problem, please open separate issue.
• If you are happy with the outcome, consider supporting Weblate by donating.

[mipapo]

I had several suggestions which I accepted and they were not saved to file. Is there a way to force to rewrite everything what is in the database? I have noticed this too late and now are a lot of translations missing.
(updated to 3.4)

[nijel]

There is no simple way for that. What could work is marking all stings as pending, though I really haven't tested that it will work properly:

from weblate.trans.models import Unit
from weblate.utils.state import STATE_TRANSLATED

Unit.objects.filter(
    state=STATE_TRANSLATED, 
    translation__language__code='lang',
    translation__component__slug='component',
    translation__component__project__slug='project'
).update(pending=True)