Differences in transliteration compared to behat/transliterator

[kdambekalns]

Symfony version(s) affected: 5.0.2

Description

We used behat/transliterator until now, but because of issues with PHP 7.4 look into replacing it. I switched our codebase to use symfony/string and run into some differences with the result of transliteration to ASCII.

Input	behat/transliterator	symfony/strings
汉语	yi-yu	han-yu
ភាសាខ្មែរ	bhaasaakhmaer
ภาษาไทย	phaasaaaithy	phas-a-thiy
العَرَبِية	l-arabiy	al-arabit
한국어	hangugeo	hangug-eo
မြန်မာဘာသာ	m-n-maabhaasaa
हिन्दी	hindii	hindi

Notes:

• The result of symfony/strings is backed by http://dzcpy.github.io/transliteration for 汉语 (han-yu)
• The result of behat/transliterator is backed by http://dzcpy.github.io/transliteration for ភាសាខ្មែរ (bhaasaakhmaer), 한국어 (hangugeo) and हिन्दी (hindii)
• For العَرَبِية and မြန်မာဘာသာ http://dzcpy.github.io/transliteration disagrees with both libraries
• If an empty string was returned from the transliteration, that may be due to missing support for a language and/or wrong detection.

How to reproduce

class TransliterationTest extends UnitTestCase
{
    public function sourcesAndResults(): array
    {
        return [
            ['Überlandstraßen; adé', 'uberlandstrassen-ade'],
            ['TEST DRIVE: INFINITI Q50S 3.7', 'test-drive-infiniti-q50s-3-7'],
            ['汉语', 'yi-yu'],
            ['日本語', 'ri-ben-yu'],
            ['Việt', 'viet'],
            ['ភាសាខ្មែរ', 'bhaasaakhmaer'],
            ['ภาษาไทย', 'phaasaaaithy'],
            ['العَرَبِية', 'l-arabiy'],
            ['עברית', 'bryt'],
            ['한국어', 'hangugeo'],
            ['ελληνικά', 'ellenika'],
            ['မြန်မာဘာသာ', 'm-n-maabhaasaa'],
            [' हिन्दी', 'hindii'],
            ['Иван Иванович', 'ivan-ivanovic'],
            ['Löic & René', 'loic-rene'],
            ['Châteauneuf du Pape', 'chateauneuf-du-pape'],
            ['Žluťoučký kůň', 'zlutoucky-kun'],
            [' x- ', 'x'],
        ];
    }

    /**
     * @test
     * @dataProvider sourcesAndResults
     */
    public function transliterationWorksAsExpected($source, $result): void
    {
        $result = strtolower((new AsciiSlugger())->slug($source)->toString());
        self::assertEquals($result, $result);
    }
}

[nicolas-grekas]

Thanks for the report and the links that's interesting. A core design principle of the Symfony slugger is that we want to reuse standard data as a source: we're not linguists and we have no authority to maintain translit maps.

The source we use is from Unicode, the CLDR sub-project maintains maps here:
https://github.com/unicode-org/cldr/tree/master/common/transforms

E.g. for ភាសាខ្មែរ, there are no maps for Khmer (km), that's why it's not translated correctly.

This raises two questions to me:

• is the empty string the best possible return value for the situation? (shouldn't we throw an exception instead when this happens? How do we detect the situation?)
• what's the authority of https://github.com/dzcpy/transliteration?

Maybe the missing maps should be reported to the CLDR project?

[javiereguiluz]

What Nicolas said is true. Symfony's rules should be correct because they aren't our rules but CLDR rules. Also, in some cases (but not in all of them) Google also agrees with Symfony/CLDR. Example:

About the "return empty string vs throw exception" I think it'd be better to throw the exception. An empty slug will probably break the app anyway ... so better warn the developer.

[bwaidelich]

is the empty string the best possible return value for the situation?

That's a tricky one..
I'd say exception would definitely be better than an empty string. And I suppose it's an exceptional case if either the input or the output string is empty (trimmed).

That way the invoking side would be forced to deal with the situation (and generate a random slug for example)

[stof]

Well, I would say that if the input is the empty string, we should still return the empty string as output. That's the sane transliteration for it. The exceptional case would be a non-empty input producing an empty output.

[bwaidelich]

But trying to create a slug for an empty string.. Does that make sense?

[nicolas-grekas]

The issue is how do we handle Khmer chars found in the middle of non-Khmer ones.
A fallback character is another option, turning unmapped chars to eg. dots.

[bwaidelich]

Just a thought on future backwards compatibility:
Being defensive and throwing an exception for strings/characters that can't be converted would probably make it easier to extend the functionality without breaking possibly expected behavior. It would then just accept additional inputs rather than changing the resulting output.

[garak]

I would add another important difference: Symfony is not lowercasing string.

[stof]

@garak we transliterate to ASCII. ASCII has uppercase letters.

[garak]

@garak we transliterate to ASCII. ASCII has uppercase letters.

Well, it also has a lot of other characters that I would not put in a slug.

[nicolas-grekas]

slug($foo)->toLower() and done

[garak]

slug($foo)->toLower() and done

I guess you meant ->lower(), but anyway it's fine. Thanks