Support Unicode identifiers

[phdoerfler]

I want to cythonize this Python 3 code:

def say_hello_to_λ(name):
    print("Hello λ %s!" % name)

which fails:

cythoning src/main/cython/hellocython.pyx to src/main/cython/hellocython.cpp

Error compiling Cython file:
------------------------------------------------------------
...
def say_hello_to_λ(name):
                ^
------------------------------------------------------------

src/main/cython/hellocython.pyx:1:17: Unrecognized character
building 'cignificance' extension
clang -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c++11 -I/usr/local/miniconda3/envs/spiht/include/python3.6m -c src/main/cython/cignificance.cpp -o build/temp.macosx-10.7-x86_64-3.6/src/main/cython/cignificance.o -std=c++11 -fextended-identifiers
src/main/cython/cignificance.cpp:1:2: error: Do not use this file, it is the result of a failed Cython compilation.
#error Do not use this file, it is the result of a failed Cython compilation.
 ^
1 error generated.
error: command 'clang' failed with exit status 1

I am not the only one with a desire to use unicode in my identifiers:

https://stackoverflow.com/questions/47462127/avoid-unrecognized-character-when-compiling-pyx-to-c-without-deleting-the-nord

And apparently certain Unicode characters are permitted in identifiers according to this https://stackoverflow.com/a/12693346/969122 since C11.

So I don't see a reason cython should not allow these characters.

[scoder]

Enough of PEP-489 should now be supported in the latest master to consider this a realistic feature. PR welcome.

Note that it's worth not requiring C11. The C code that Cython generates usually satisfies itself with C89. Punycode can be used to encode Unicode identifier names to ASCII identifiers.

[ghost]

Note that Punycode uses the hyphen as separator. As the separator is not allowed in C or C++ identifiers, we'd have to use a custom variant of Punycode which uses the underscore as separator.

Edit: There is already a note 'with hyphens ("-") replaced by underscores ("_")' in the PEP-489 which you linked to.