mixing py3 str/bytes with classic unicode/str

[faassen]

I need some help in my understanding. Perhaps it's a documentation request (there's some documentation in "What else you need to know" already, but I'd like it in a table, I think, with a bit more motivation and talking about consequences). Maybe at the end of writing this I'll understand your reasoning better. :)

python-future could be used to incrementally upgrade a larger codebase to Python 3, on a per-module basis. Is this a valid use case? If not, it's at least likely that futurized code can call into non-futurized python 2 code at some point. So I think that this is a use case in any case on an inter-package level, so inter-module should be okay too.

In this case it's possible for backported Python 3 str and bytes objects to interact with Python 2 unicode/str objects. For convenience I'll call them py3str, py3bytes, py2unicode, py2str.

I expected the system to behave as follows:

py3str + py2unicode -> py2unicode

py3str + py2str -> py2unicode (or unicode error if py2str is not ascii)

py3bytes + py2unicode -> py2unicode (or unicode error if py3bytes is not ascii)

py3bytes + py2str -> py2str

i.e. the system falls back to Python 2 behavior when mixing Python 2 objects with Python 3 objects, for ensuring compatibility with non-Python 3 code.

Instead the behavior is as follows

py3str + py2unicode -> py3str

py3str + py2str -> py3str

py3bytes + py2unicode -> illegal

py3bytes + py2str -> py3bytes

i.e. the system upcasts combinations to the Py3 versions and makes one interaction illegal.

I can see why py3bytes interaction should not allow interacting with py2unicode even in Python 2 code in practice, so can be made illegal to enforce it: if you know you're dealing with a byte str in Python 2, there's no sane way to add unicode to it, so could shouldn't do it anyway.

I can also see that for the reason of working with literals, it makes sense to upcast to py3str.

And then, as the documentation describes, py3str may be combined with py2str to support backwards compatibility. So instead of downcasting things, upcasting is needed to make it possible to work sanely with literals, and there's a concession to backward compatibility by allowing mixing with py2str.

It could still lead to problems down the line in Python 2 code, as it might not do a decode before writing to a file, for instance. But in that case the situation is the same as passing unicode into code that isn't unicode-safe, so the bug exists in Python 2 already.

Is my reasoning correct? Could I help writing docs on this?

[edschofield]

Hi Martijn,

Thanks very much for your thoughtful comments. Yes, it is definitely a valid and important use-case for future to support module-by-module upgrades to Python 2/3. And yes, your reasoning exactly reflects the design decisions. I would be very grateful for your help with writing some docs on this. Perhaps this could live in a new .rst file in the docs/ directory.

The only additional point I would like to make is that we do not yet have a lot of real-world porting stories to either support or reject the current casting behaviour. If some compelling evidence is brought in about difficulties caused by the current behaviour, and a better alternative exists, I would be open to changing the design.

However, I would like to lock down the API of future by version 1.0 to avoid any backward incompatibilities until a hypothetical version 2.0 (which would have a different namespace so it could be installed side-by-side with version 1.x if needed).

Dr. Edward Schofield
(M) +61 (0)405 676 229
Python Charmers
http://pythoncharmers.com

On Wednesday, 8 January 2014 at 3:39 am, Martijn Faassen wrote:

I need some help in my understanding. Perhaps it's a documentation request (there's some documentation in "What else you need to know" already, but I'd like it in a table, I think, with a bit more motivation and talking about consequences). Maybe at the end of writing this I'll understand your reasoning better. :)
python-future could be used to incrementally upgrade a larger codebase to Python 3, on a per-module basis. Is this a valid use case? If not, it's at least likely that futurized code can call into non-futurized python 2 code at some point. So I think that this is a use case in any case on an inter-package level, so inter-module should be okay too.
In this case it's possible for backported Python 3 str and bytes objects to interact with Python 2 unicode/str objects. For convenience I'll call them py3str, py3bytes, py2unicode, py2str.
I expected the system to behave as follows:
py3str + py2unicode -> py2unicode py3str + py2str -> py2unicode (or unicode error if py2str is not ascii) py3bytes + py2unicode -> py2unicode (or unicode error if py3bytes is not ascii) py3bytes + py2str -> py2str
i.e. the system falls back to Python 2 behavior when mixing Python 2 objects with Python 3 objects, for ensuring compatibility with non-Python 3 code.
Instead the behavior is as follows
py3str + py2unicode -> py3str py3str + py2str -> py3str py3bytes + py2unicode -> illegal py3bytes + py2str -> py3bytes
i.e. the system upcasts combinations to the Py3 versions and makes one interaction illegal.
I can see why py3bytes interaction should not allow interacting with py2unicode even in Python 2 code in practice, so can be made illegal to enforce it: if you know you're dealing with a byte str in Python 2, there's no sane way to add unicode to it, so could shouldn't do it anyway.
I can also see that for the reason of working with literals, it makes sense to upcast to py3str.
And then, as the documentation describes, py3str may be combined with py2str to support backwards compatibility. So instead of downcasting things, upcasting is needed to make it possible to work sanely with literals, and there's a concession to backward compatibility by allowing mixing with py2str.
It could still lead to problems down the line in Python 2 code, as it might not do a decode before writing to a file, for instance. But in that case the situation is the same as passing unicode into code that isn't unicode-safe, so the bug exists in Python 2 already.
Is my reasoning correct? Could I help writing docs on this?

—
Reply to this email directly or view it on GitHub (#27).

[faassen]

When discussing Future there was some concern from a Twisted developer that Future is not good for libraries (as opposed to applications), because it can change the API. I was hoping this person would comment on the issue himself, but I just thought I'd drop in a note for future reference as this would be worthwhile to explore.

I think the issue centers around the behavior of 'bytes'. If use future for a Python 2 library that works with bytes, suddenly the new bytes object would be returned back into the code that uses that library. The code would have a problem this bytes is not compatible with its expectations.

So is the bytes object incompatible with the expectations of code that expects a Python 2 str? Or is it clever enough with it's @no decorator to prevent this incompatibility? What about __getitem__, though? That returns numbers, which is totally not what Python 2 code would expect...

[itamarst]

Right. Imagine you're a Python 2 Twisted user who hasn't upgraded to a Python 3 compatible code base. One day Twisted starts giving you bytes that behave completely differently (__getitem__ is the fun one, yes). This breaks your code. Twisted cannot therefore use future.builtins.bytes. And for similar reasons I would guess it's not possible to upgrade module by module using something like future. Bytes at least are irrelevant for many libraries, though.

Probably what you would want for a real module by module upgrade is past, which is a package that makes Python 3 behave like Python 2. Then you add from past import * to all your modules, which is noop on Python 2 and makes the code work on Python 3 (modulo minor syntax tweaking). After that you could go the "port to intersection of 2 and 3" route on a module by module basis, removing the past import as you go. You'd still not be able to use the library on Python 3 without also doing import past thing, but at least you could start writing new code in the intersection language.

Perhaps future should grow a future.past? :)

[edschofield]

@faassen, @itamarst: thanks very much for your comments.

I agree that a past module would be useful for module-by-module upgrades. I have committed an initial version to a new branch here. The tests in past/tests are probably the best documentation at this stage. I'd be grateful for any feedback.

I agree that changing the APIs to accept and return Py3 / future data types would be backwards-incompatible change.

I think of past as more of a temporary migration tool and future as more of a long-term solution. Ultimately I expect Twisted would want to change its APIs to accept and return standard Python 3 data types. Providing backwards compatibility with Py2 after making this change is where future would shine.

[itamarst]

For bytes, Twisted has already gone the route of intersection language, which is what you're forced to do if you need to support both versions. The main attraction for me in something like past is the ability to upgrade an application module by module, not a library, especially if combined with a "auto-wrap all literals" transformation. Unlike 2to3 this would be simple enough that you could almost certainly be sure it wouldn't break your code, although it would perhaps slow it down a bunch. Also it wouldn't break Python 2 support again unlike 2to3.

For libraries I presume telling Python 3 programmers "you can use Twisted but bytes will act like Python 2 in half the API" won't go over that well, although perhaps it's better than nothing. So using native bytes is probably what we'd want to do.

That being said, being able to shadow map/filter/etc. would make me happy, that was one of the annoying parts of porting Twisted to Python 3 (a process which is only a 1/3 finished) because it's easy to miss one and end up with the unexpected lazy semantics breaking an end user's code.

Another thought: for people going the intersection language route, a dict that is unmodified except that it has viewitems etc. added on Python 3 might be useful. list(d.keys()) is a terrible idiom. If there was viewitems etc. on both Python 2 and Python 3 you could just say "this is the API that works on both". Doesn't help you with dicts input from external callers, though. In an ideal world that would be in Python 3, but on bad days I suspect the Python 3 core developers deliberately tried to making porting harder than it needs to be.

[edschofield]

@itamarst: Thanks very much for your feedback and ideas! They have definitely helped to clarify the direction of this.

[edschofield]

Update: I have released v0.11 of python-future, which includes a top-level past package with past.builtins. There's an overview here and some example code here. There's also an early-stage translation tool for importing Py2 modules into Py3.