canonicalize unicode identifiers

[stevengj]

As discussed on the mailing list, It is very confusing that

const μ = 3
µ + 1

throws a µ not defined exception (because unicode codepoints 0x00b5 and 0x03bc are rendered almost identically). This could easily be encountered in real usage because option-m on a Mac produces 0x00b5 ("micro sign"), which is different from 0x03bc ("Greek small letter mu").

It would be good if Julia internally stored a table of easily confused Unicode codepoints, i.e. homoglyphs, and used them to help prevent these sorts of confusions. Three possibilities are:

• foo not defined exceptions could check whether a homograph of foo is defined and let the user know if so.
• Julia could issue a warning if a non-canonical homoglyph is used in an identifier.
• Simply canonicalize all homoglyphs in identifiers (so the users can type them any way they want, but they are treated as equivalent identifiers).

My preference would be for the third option. I don't see any useful purpose being served by treating μ and µ as distinct identifiers.

[johnmyleswhite]

+100 for this. Any strategy for ensuring that homoglyphs are merged seems like a big improvement to me.

[toivoh]

+1 for canonicalizing everything.

[stevengj]

(We should probably also normalize the Unicode identifiers, in addition to canonicalizing homoglyphs.)

[stevengj]

On possible software package that we could adapt for this might be utf8proc, which is MIT-licensed and fairly compact (600 lines of code plus a 1M data file). It looks like it does Unicode normalization, but not homograph canonicalization (except for a small number of special cases?). Looks like it handles homoglyphs for us.

[jiahao]

+1 for canonicalization and normalization.

We certainly don't want the same disambiguation issues with combining diacritics and nonprinting control characters (like the right to left specifier). The Unicode list contains quite a few characters with combining diacritics already; not sure if it's exhaustive though.

[stevengj]

Actually, it looks like the utf8proc library completely solves this problem, because it implements (among other things) the standard "KC" Unicode normalization which canonicalizes homoglyphs.

I just compiled the utf8proc library and called it from Julia via:

function snorm(s::ByteString, options=0)
       r = Ptr{Uint8}[C_NULL]
       e = ccall((:utf8proc_map,:libutf8proc), Int, (Ptr{Uint8},Csize_t,Ptr{Ptr{Uint8}},Cint), s, sizeof(s), r, options)
       e < 0 && error(bytestring(ccall((:utf8proc_errmsg,:libutf8proc), Ptr{Uint8}, (Int,), e)))
       return bytestring(r[1])
end

and then

julia> s = "µ"
julia> uint16(snorm(s)[1])
0x00b5
julia> uint16(snorm(s, (1<<1) | (1<<2) | (1<<3) | (1<<5) | (1<<12))[1])
0x03bc

works (the second argument is various canonicalization flags copied from the utf8proc.h header file).

Moreover, the utf8proc canonicalization functions (including Unicode-aware case-folding and diacritical-stripping) would be useful to have in Julia anyway. I vote that we just put the whole utf8proc into deps and export some version of this functionality in Base, in addition to canonicalizing identifiers.

[jiahao]

Awesome, thanks for doing the legwork on this.

[JeffBezanson]

That sounds like a really good idea to me.

[pao]

KC has one case that we probably don't care about but seems worth mentioning: superscript numerals will be normalized to normal numerals. (We probably don't care because why would you have superscript numerals in a numeric literal, but this seems like the sort of thing to be abused in a future International Obfuscated Julia Coding Contest.)

[JeffBezanson]

That's not totally ideal; N² is a cute variable name :)

[jiahao]

I've actually used χ² somewhere.

[JeffBezanson]

We also have to avoid normalizing out different styled letters that represent different symbols in mathematics.

[nalimilan]

The problem with ² is that it seems to mean ^2, so maybe it's better not to encourage it.

[jiahao]

@JeffBezanson may be referring to what UAX #15 calls font variants (see Fig. 2). They give as an example \mathfrak H vs \bbold H, but I suspect regular \phi vs script \varphi is the one that would come up fairly often. (Ironically, Github won't let me enter the characters...)

So it seems that we are learning toward canonical equivalence, as opposed to full compatibility equivalence, in which case NFD may be sufficient rather than NFKC.

[pao]

For variable names, I don't see the superscript/subscript being as much of a problem, other than i.e., χ² will be the same identifier as χ2; if you are distinguishing these I might think you were mad.