Support python 2's almost-raw ur'' literals

[habnabit]

Python 2 allows ur'' literals to contain \u and \U escapes which are actually parsed as unicode escapes instead of literally '\' 'u'.

I was going to write test cases but I'm not sure where to put them! This does actually fix the problem I was having where code relied on this, though.

[robertwb]

Tests?

[habnabit]

Still not sure where to put them!

…

On Mon, Feb 6, 2017 at 11:11 PM, Robert Bradshaw ***@***.***> wrote: Tests? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1594 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABSsDyix7HZXIrkihP6rmapm9ByjDqoks5raBklgaJpZM4LvBIs> .

[robertwb]

Sorry, should have answered your question. tests/run//unicodeliterals.pyx might be a good spot to add some examples.

[scoder]

I'm not sure if we should really support this. It is a legacy feature and has the potential to break existing code in subtle ways.

[scoder]

The change for latest master would be this, BTW:

            if is_raw and (is_python3_source or kind != 'u' or systr[1] not in u'Uu'):

[habnabit]

It currently is breaking existing python 2 code when run translated with Cython instead of natively as python. Is full compatibility not a goal? This seems like oversight of an obscure feature that nobody (but me) was using rather than an intentional decision.

[scoder]

The deviation from Python behaviour is (almost) always a bug and this one was definitely an oversight and not a deliberate decision. I agree that it's a trap that users might run into when compiling or converting Python 2 code to Cython code. But changing the current behaviour might break existing Cython code. And people have the same problem when migrating code from Python 2 to Python 3 already. Thus, it does not seem an obvious choice to implement this.

[habnabit]

The deviation from Python behaviour is (almost) always a bug and this one was definitely an oversight and not a deliberate decision. I agree that it's a trap that users might run into when compiling or converting Python 2 code to Cython code. But changing the current behaviour might break existing Cython code.

Sure. But given how long it took for this to be discovered, it seems unlikely.

And people have the same problem when migrating code from Python 2 to Python 3 already.

Sure. This code would require many more changes to run under python 3. I don't think this code would ever be ported (certainly not by hand) until the automated tooling situation improves.

Thus, it does not seem an obvious choice to implement this.

Okay. I can change this to a documentation patch instead, as nobody knew this was omitted.

[robertwb]

I would find it really surprising (a bug) if someone was relying on ur"\uXXXX" not following Python 2 semantics. Thus I think it makes a lot of sense to fix this bug (despite it being a bit legacy, at least it doesn't add non-trivial complexity to support).