Automatic Translation using Weblate TM became "very expensive" with v4.3

[ilocit]

Describe the bug

When you pre-translate an update with translations from Weblate Translation Memory, this seems to be very "expensinve" in terms of CPU (and also mem) usage because of postgres processes. Running auto trans from other components seems as fast as before, so does MT. But not from Weblate Translation Memory!

To Reproduce

Update resources with latest sprint data

1. Go to 'Tools'
2. Click on 'Automatic Translation'
3. Select 'Machine Translation' under 'Automatic translation source'
4. Choose "Weblate Translation Memory" from available options

Screenshots

Taking very, very long...

CPU usage:

Server configuration and status

Docker Weblate v 4.3

[nijel]

There is already improvement scheduled for 4.3.1

[nijel]

Should be addresed by 241dd91

[github-actions]

Thank you for your report, the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, consider supporting Weblate by donating.

[ilocit]

There is still something wrong with the performance, I guess. A project I am upadting right now has 28 new strings. Running the "pre-translation from TM" still takes really long. Like 2 minutes.
This is what the CPU usage looks like:

[nijel]

Are you already running 4.3.1? The automatic translation can be responsible only for one of the PostgreSQL connections, the other two must be caused by something else.

On the other side I also think it still performs worse than in 4.2 and I don't see any obvious reason for that.

[ilocit]

Yes, the instance was updated to 4.3.1 already. Maybe another user is running an auto-translation or search at the same time. Especially for long strings, the search really takes a couple seconds, even when translating and using Automatic Suggestions.

[ilocit]

A search for this in MT and TM takes about 5 sec until a result is shown.

That is the time you will see the spinning wheel to indicate that the search is being performed. In some similar cases it takes the same 5 seconds to return the MT results but another 10 seconds after that to display results from the TM.

[nijel]

That's horrible performance. I've just tested that on https://translate.fedoraproject.org/ (which has the biggest translation memory from the publicly available instances) and the TM results take around 300ms there. Did it really change with the 4.3 release for you? There were almost no code changes between 4.2 and 4.3 in terms of translation memory (and definitely no changes in the lookup code).

[ilocit]

I am not sure, actually. But we did not have any complaints from translators recently, only after we switched to 4.3
Maybe it is also the combination of MT and TM? I see that fedora does not use MT. It seems that first MT providers are called and then, once finished, the TM is quried, no? At least that is how the search results come in when you navigate to Automatic Suggestions.

[ilocit]

Now I am trying to pre-translate 410 strings (2122 words) into Finnish from weblate and weblate TM. There is litterally now progress in the past 10min. The process is still running at almost 100%.

There is definitly something wrong. Maybe some migration issue with the DB after upgrading to 4.3? The migration of the db took much longer during the upgrade from 4.2. to 4.3 then it ever took after you moved to postgres.

[nijel]

The migration to 4.3 is expected to take long, see https://docs.weblate.org/en/latest/admin/upgrade.html#upgrade-from-4-2-to-4-3

Let's skip the autotranslate part for now and let's take a look at TM performance. Did that change when upgrading to 4.3? There are no changes in that code between these releases. Maybe the database performance just got worse for some reason. Any changes in hardware or VM setup?

[ilocit]

Hardware and VM setup are exactly the same. I am running some tests at the moment. For instance, I noticed this issue with the wrong source lang tag in export file formats, which you already fixed. Thank you for that! In order to correct the issues I had encountered, I deleted all TMs for that particupar project and imported them again. So that project went from having a TM, to having no TM to having a new TM from imported segments. I believe that the performance for this particular project is now very, very good. In fact better than ever. But this not yet “proven” by real tests. At the moment I have a slight capacity issue and don’t have enough time to continue the tests right now ☹.

[nijel]

Deleting big amount of the TM entries would most likely trigger autovacuum on PostgreSQL, what can quite likely improve the database performance.