Error message when comparing same string in NFC and NFD forms is not clear on Python 3

[pekkaklarck]

If I compare same string in different Unicode normal forms the error message doesn't show any differences between them when using Python 3. For example, if I have a test like

def test_nfc_nfd():
    nfc = 'hyv\xe4'
    nfd = 'hyva\u0308'
    assert nfc == nfd

the result I got is

>       assert nfc == nfd
E       AssertionError: assert 'hyvä' == 'hyvä'
E         - hyvä
E         + hyvä

which isn't very informative.

The problem is caused by repr('hyva\u0308') being 'hyva\u0308' which is rendered as 'hyvä'. I submitted an issue about repr() escaping combining characters (i.e. turning the result to 'hyva\\u0308') but it was closed as invalid.

In PyTest it would be possible to use ascii() instead of repr(), but that would make all non-ASCII strings unreadable and that would be worse. I guess the best solution would be using ascii() if strings look the same, but I have no idea how to detect that.

[pekkaklarck]

Cool bot but I don't think any of the referenced issues is related. os: mac label is somewhat appropriate as the OSX file system uses NFD and is probably the most common source of Unicode strings in NFD form. The problem itself is OS agnostic.

[RonnyPfannschmidt]

its not clear what to do here, but i do indeed wonder if we should have an heuristics that adds a diff comment line when the difference character is equal when normalized

[Zac-HD]

I guess the best solution would be using ascii() if strings look the same, but I have no idea how to detect that.

Check if the strings are different, but the NFKD forms are the same - see wikipedia on Unicode equivalence for details and unicodedata.normalize() for the function to use.

[dnstone]

Hi @Zac-HD, I talked with you about this issue as part of pycon mentored sprint. Is it okay if I take over this issue?

[Zac-HD]

Go for it!

[itxasos23]

Hey all!
Is it ok if I try to tackle this as part of Hacktoberfest?

[Zac-HD]

Absolutely!

[pekkaklarck]

Great to see this fixed! 🎉