Remove or move entries with non normalized codes in mongodb

[alexgarel]

We have entries on the file system or in mongodb with a non normalized code, (see Products.pm::normalize_code). This leads to confusion for people using the data dump (see #7244)

Run a script to clean those entries. For each entry with a non normalized code:

• always remove the non normalized version in mongodb
• for the sto file:
  • if an entry with the normalized code exists: remove the non normalize one
  • else move the entry to its normalized code

Note: be sure to double search the non normalized codes : in mongodb and on the filesystem

[alexgarel]

First dry-run on preprod, it was really really slow. I must add some progression indicators to understand what is the slowest.

real    3608m29.575s
user    11m10.066s
sys     34m11.942s

Difference user / real indicates it spends a lot of time waiting for something (file system). I suspect we have a problem with zfs mount on preprod. (see also #6164). Maybe it's just that we need to avoid being on an old version of the ZFS, and for that we need to do something ! (maybe mount / unmount the file system, I'm not sure).

Finally it died on mongodb stuff.

MongoDB::DatabaseError: Executor error during find command :: caused by :: operation exceeded t
ime limit

I attach the log of advertised changes from the first phase (it's a dry run, so it's not done yet).

I will:

• add logs to get progression (with a specific option)
• try to see if I can remount zfs and see if it brings a change
• add options to only run part of the script (that make sense) to be able to jump to one part of the script, it will help debugging

[github-actions]

This issue has been open 90 days with no activity. Can you give it a little love by linking it to a parent issue, adding relevant labels and projets, creating a mockup if applicable, adding code pointers from https://github.com/openfoodfacts/openfoodfacts-server/blob/main/.github/labeler.yml, giving it a priority, editing the original issue to have a more comprehensive description… Thank you very much for your contribution to 🍊 Open Food Facts