Categories - Pâtés et Pates are merged together

[teolemon]

What

fr:pâtés
fr:pates
are amalgamated when typing the category from world.off

Part of

• 🧽 Data quality (tracker) #5538

[hangy]

Let's maybe simply remove the unconditional deaccenting of tags when they are canonicalized (ProductOpener::Store::unac_string_perl or ProductOpener::Store::get_fileid)? We had other reports where this yieled similar unintended results in German.

[hangy]

We should probably also use Unicode::Casing to support different languages properly (ie. Turkish I problem)?

[teolemon]

@hangy probably. @maddingue knowledgeable about this ?

[stephanegigandet]

For French we should keep the unaccenting, it helps in many cases, a lot of people type "boeuf" (I have no idea how to type the oe char in fact ;-) ). There are a few conflicts where 2 words that deaccent to the same string mean 2 different things, but they are very rare. One example is pâte and pâté.

[hangy]

One problem is that get_fileid does not have a language/country for context. äöü shouldn't be replaced for a de locale, for sure. There's just too much potential for conflict, and noone with a German keyboard layout writes "Doener" instead of "Döner".

Unconditional unaccenting of é to e for other languages than French might still cause conflicts. I honestly don't know enough about all languages to know how ie. a native Hungarian speaker would handle that.

[teolemon]

We can close this one, right ?

[hangy]

We can close this one, right ?

Depends. https://world.openfoodfacts.org/category/fr:p%C3%A2t%C3%A9s and https://world.openfoodfacts.org/category/fr:pates both redirect to https://world.openfoodfacts.org/category/pastas, as unaccenting is intentionally enabled for French:

openfoodfacts-server/lib/ProductOpener/Config_off.pm

Lines 124 to 127 in e736687

	fr => {
	unaccent => 1,
	lowercase => 1,
	},