Add a CLI command and a cronjob to detect duplicated translations

[amieiro]

Problem

Last week, while I was developing an improvement to be able to make bulk updates to the translations in the consistency tool, I made a mistake and set some translations as "current" for the same original.

I knew we have some translations in the same situation, maybe related with some translations created while the original strings are imported.

I made a script to detect the number of duplicates, and we have more than 42,000 duplicates in 31,600,000 translation sets (around 0.13 % of the translations).

We have different situations:

1. Same date_added and date_modified. We have rows, at least, between 2015 and in 2023:

SELECT * FROM `translate_translations` WHERE `original_id` = 89329 AND `translation_set_id` = 11930

SELECT * FROM `translate_translations` WHERE `original_id` = 14652687 AND `translation_set_id` = 4345261

2. Same date_modified but different date_added:

SELECT * FROM `translate_translations` WHERE `original_id` = 10345042 AND `translation_set_id` = 367588

3. Rows with multiple duplicates:

SELECT * FROM `translate_translations` WHERE `original_id` = 1654345 AND `translation_set_id` = 1814851

4. Translations created by a user and then updated by a script: user_id_last_modified = 0.

SELECT * FROM `translate_translations` WHERE `original_id` = 3750671 AND `translation_set_id` = 6579081

Solution

This PR adds:

• A cronjob, to review daily if we have some duplicated translations in the database with the current status. It will send a notification to a Slack channel.
• A WP-CLI command, to manually update the duplicates.

To detect duplicated translations, the code looks for translations with the same original_id and translation_set_id and the translation status set to current.

To update these duplicated translations, the code updates all the duplicates, setting:

• The status to old.
• The user_id_last_modified to NULL, reflecting a translation updated by a bot.
• The date_modified to the current date time.

Only one duplicated translation remains without been updated:

• The last modified (highest date_modified value).
• If two or more duplicated translations have the same date_modified value, the code selects the one with the highest id (the last introduced in the system).

Cronjob

The cronjob checks once a day if we have some duplicated translations in the database and sends a notification to the #dotorg channel (Slack, "Making WordPress" instance).

These are the screenshots that this cronjob will send:

Command

The command will allow updating the duplicated translations, so we will be able to solve this problem by hand. If this problem persists, we will be able to update these duplicated translations in the cronjob (This has to be done in a different PR).

To use this command, you have to execute in the CLI:

wp wporg-translate duplicate-translations --url=translate.wordpress.org

This command has 4 flags:

• --fix: Updates the duplicates, setting all translations to old except the last one.
• --notify: Notifies the duplicates in Slack.
• --verbose: Shows the results in the CLI.
• --print-sql: Shows the SQL queries to show and to update the duplicate entries.

Some CLI examples:

wp wporg-translate duplicate-translations --url=translate.wordpress.org
wp wporg-translate duplicate-translations --url=translate.wordpress.org --fix
 wp wporg-translate duplicate-translations --url=translate.wordpress.org --notify
wp wporg-translate duplicate-translations --url=translate.wordpress.org --update --verbose --print-sql

[akirk]

I like the approach, I think better naming would make it easier to comprehend

[amieiro]

@akirk I have made the changes you have suggested in the WP-CLI command interface and the improvement to the update SQL query.

[akirk]

Looks great!

[amieiro]

I close this PR, as it has been deployed. Finally, the notifications are sent to Matrix.

[NekoJonez]

@amieiro , why set them on old right away? What if the wrong one is set to old. Maybe set them fuzzy so PTE/GTE's can decide ;)
That's safer and will avoid mistakes and bugs. Otherwise one duplicate can cause issues.

[amieiro]

@NekoJonez We had a lot of translations with duplications: around 42,000, so if we put this as fuzzy, the work would be very big for the GTE.
We had a lot of old duplications, some from 2015, and most of them had the same text, so I decided to put them as old, because I considered this was not a risk.
In future executions, with small sets of duplicated translations, the script will set them as fuzzy instead of old, and the GTE will review them with low effort.