Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3 support #1506

Closed
spooning opened this issue Jun 30, 2014 · 39 comments

Comments

@spooning
Copy link
Contributor

commented Jun 30, 2014

Originally submitted to Google Code by sabbir.shafi on 18 Aug 2013

Later versions of Python 2.7 does not support robot framework...

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by @pekkaklarck on 20 Aug 2013

It is a known limitation that Robot Framework does not support Python 3. This is due to Python 2 and 3 being incompatible and Robot being developed for the former.

With some effort it is relatively easy to write Python code that is compatible with Python 2.7 and 3.3. Supporting also Python 2.6 wouldn't be that big a task but other releases would be either require lot of extra work (2.5) or not be worth the effort due to limited installation base (3.0-3.2).

With Robot Framework the main problem is that the latest stable Jython release is 2.5.3 and we cannot drop Jython support. Jython 2.7 is under development, beta1 was released already in Feb 2013, but unfortunately the development is rather slow. If/when Jython 2.7 is released, or gets at least to the release candidate phase, dropping support from older releases is not an issue. People that need to keep using Python 2.5 can also keep on using older Robot versions.

Another main problem regarding to this issue is that the current core development team doesn't have resources to look at this during 2013.

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by zimmermann.code on 30 Sep 2013

Hi. I've also wanted to use Robot Framework with Python 3 for a while. Last week I found some time and just took a look at how hard it could get. And then I just couldn't stop... ;)
I created a robotframework-python3 fork on Bitbucket:

https://bitbucket.org/userzimmermann/robotframework-python3

It's already working really well. Only 109/3111 critical atests still failing, mainly because some std libs like OS and some atests themselves need some further changes. It remains fully compatible with Python 2.5+. I use a combination of 2to3 and manual changes and tried to make the latter as small and unugly as possible.

I prepended some further explanations to the README.txt. For a complete diff look at:

https://bitbucket.org/userzimmermann/robotframework-python3/compare/default..c148e32#diff

If you like my approach I would be happy about some discussion regarding the missing changes. Especially the _Process communication bytes/str issues in OS lib. Better run Popen in text mode or better decode/encode stdin/stdout on demand?...

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by zimmermann.code on 30 Sep 2013

Dealing with issue 1531 I realized that with my current approach you can only run the acceptance tests for Python 3 if you also run the run_atests.py script with Python 3. I need to add more flexibility for that...

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by @pekkaklarck on 31 Oct 2013

Great to hear you have got that far with Python 3 support rather easily. Have you continued the work from 109 failing tests?

I have always been thinking that we should implement Python 3 support by having a common code base. Based on your work it seems that 2to3 would be a viable option too. Did you do manual changes to the original code or to the converted? In the latter case keeping them in sync with future changes to the original code would probably be complicated.

Did you try that or are you interested to try the common code base option too? In my opinion only Python 2.7 and 3.3 are must versions to support, but it would be nice to support also 2.6. Supporting 2.5 would be hard because it doesn't support except Exception as e, and 3.0-3.2 do not support u'string with u prefix'.

The problem with dropping support from Python 2.5 is that it also drops Jython 2.5 which is the latest stable Jython release. Luckily Jython 2.7 development has been picking up some steam lately. There should be Jython 2.7 beta 2 in November with pip support and other such goodies. If this beta or future beta/rc releases are stable enough, we might be able to drop Python/Jython 2.5 support in the next RF major release.

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by @pekkaklarck on 1 Nov 2013

Issue 1298 has been merged into this issue.

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by user.stefanz on 5 Nov 2013

yes. i am still working on it :) and doing regular merges.
currently 59/3146 atests fail.
but they only need further workarounds in the atests themselves.
i did not care about the utests so far.
i also have to add the 2to3 stuff to run_utests.py.
but the code is working fine with the atests... so that should be no problem.

i only made manual changes in the original code.
2to3 is used on demand in setup.py and run_atests.py.
the changes to the atests are also partly manual and partly automated.
i created Run on python 2.x and Run on python 3.x switches
i made the Evaluate code compatible
(mostly adding ()s to print statements).
i wrote automated on-demand-conversions for log/output checking
like removing the u prefixes from unicode literals,
removing the L suffixes from long literals,
and substituting \x.. and \u.... with the actual unicode chars
(python 3 always prints them auto-encoded).

i think of 2to3 as an intermediate solution.
my goal is also common code which doesn't need 2to3 anymore.
i just want to get the tests completely working first.
then i want to add further manual compatibility changes
by going through the complete 2to3 diff step by step...

limiting support to 2.7+ and 3.3+ is also useful in my opinion.
and using six for the cases where really no common syntax exists.
like metaclasses...

most of the current manual changes are of course related to bytes/str/unicode/encode/decode issues.
i also added some new features to the standard test libraries to deal with that:

BuiltIn.Create Bytes \xabyte\x57ring

to explicity create python 3 bytes objects.
is it a good keyword name for that?

OperatingSystem.Start Process textio=True

to open the process io streams in text mode with Popen(unverasal_newlines=True)
to make them accept str instead of bytes in python 3.
or better make this default and add a bytesio= switch?

i would really like to get in some further discussion about the whole approach
if you see it as a suitable base for further official development...

you can view the current complete fork diff at:

https://bitbucket.org/userzimmermann/robotframework-python3/compare/default..523e395#diff

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by @pekkaklarck on 5 Nov 2013

Great progress! Having Python 3 compatible version based on 2to3 would be great. I agree we can work towards common codebase in the future. Comments to your questions below:

  1. Is BuiltIn.Create Bytes needed when we now support \x escapes ( issue 1524 ) and there is String.Encode String To Bytes?

  2. I would like to touch OperatingSystem.Start Process on Python 2 because it is in practice deprecated by Process.Start Process. Python 3 specific changes obviously can be done.

Would you like to discuss this stuff further on #robotframework IRC channel?

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by zimmermann.code on 5 Nov 2013

  1. Encode String To Bytes also works but always needs an explicit encoding=iso8859
    to result in the same byte values as defined in the input argument.
    and you get UnicodeEncodeErrors.
    i think that is not very intuitive if you just want to deal with bytes.

what do you think about changing Create Bytes
to acutally do an iso8859 encoding of the input string
so you don't need double \ before x?
maybe rename it to Convert To Bytes.
maybe make it also accept an integer sequence
or other input supported by the python 3 bytes constructor?
maybe additionally provide an Evaluate To Bytes
which behaves like the current Create Bytes?

irc. yes. i am always somehow present in #robotframework.
and mostly active during working hours :) ~ 9-18 cet

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by @pekkaklarck on 5 Nov 2013

I don't think we need separate "Create Bytes" or "Convert To Bytes" when we already have "Encode String To Bytes". That keyword, and its decoding counterpart, definitely could get default encoding, though. I was initially thinking the default encoding should be UTF-8 that we use also elsewhere, but in order to be able to create bytes 0-255 by default, it apparently needs to be ISO-8859-1. If you think this is a good change, please submit a separate enhancement about it.

What kind of use cases you would have for other keywords for creating bytes?

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by zimmermann.code on 6 Nov 2013

initially i implemented Create Bytes as a workaround for
atest/testdata/standard_libraries/string/encode_decode.txt.
it uses Evaluate "...\x..." to create bytes
and Decode Bytes To String with a string argument.
both don't work in python 3.
i also thought of using Encode String To Bytes.
but it needs the explicit latin encoding.
and as a default encoding utf8 really makes more sense.
in that Decode... case i also thought it looks a bit strange
to first encode something and then decode when you actually only want to decode.
i thought about adding explicit internal str-->bytes argument conversions
to keywords if actually bytes are needed.
but finally i decided that special Bytes Keywords are a cleaner way
to deal with the missing implicit bytes<-->str convertibility in python 3.
and also a cleaner way to deal with pure binary data
for writing stuff to or checking input from binary file, device or process streams.
for me encoding/decoding always implies working with textual data.
and Create Bytes could provide these additional python 3 bytes() like features
like creation from number sequences.

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by @pekkaklarck on 8 Nov 2013

I can see Create Bytes could be useful in some cases, especially when not dealing with textual data. It probably should work both with string (Unicode) and integer inputs and look something like below:

>>> def create_bytes(*args):
... return ''.join(''.join(chr(ord(c)) for c in arg) if isinstance(arg, basestring) else chr(arg) for arg in args)
...
>>> create_bytes(0, 1, 80, 228)
'\x00\x01P\xe4'
>>> create_bytes(u'hyv\xe4', 32, '\xe4iti', '!')
'hyv\xe4 \xe4iti!'

This should be submitted as a separate issue, though.

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by zimmermann.code on 27 Nov 2013

I merged with 2.8.2, made some additional changes, and everything still works fine.
In fact even better :)

Only 36/3249 critical atests still failing.

I removed my original Create Bytes keyword
and replaced it with (now Python 3 compatible) Convert To Bytes in the atests.

The Get Dictionary * keywords caused a lot of problems in the atests,
because the keys are sorted() and Python 3 doesn't support comparison
of values with different basic types like int and str.
I just removed the sorting in Python 3 as a first workaround.
But this now causes the fail of 10 other critical atests (10 out of the 36 above)
due to different order in output checks.
Maybe always sort the keys the Python 2 way with some custom robot.utils.sorted?

Current complete diff:

https://bitbucket.org/userzimmermann/robotframework-python3/compare/default..1fd297b#diff

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by @pekkaklarck on 28 Nov 2013

Where's this Get Dictionary? We probably could change it to use custom sorter that handles also different values somehow.

Do you think it would make sense to make a separate Python 3 compatible release at some point? I hope we can drop Jython 2.5 support and make the core project Python 3 compatible before next summer, but it would be awesome to have a separate release already before that.

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by zimmermann.code on 28 Nov 2013

I mean Collections.Get Dictionary Keys.
(And Get Dictionary Values/Items which rely on Keys)

I also thought about a separate robotframework-python3 package as a temporary solution.
(But not unofficial...)
I really would like to drop the 2.5 support asap.
To start fully porting to 2/3 compatible code without the need for 2to3.
And to remove some (for 2.6+) unnecessary version checks and try...excepts :)

1 design question is still open:
How to handle the different Python 3 Popen invocations needed for opening process streams
as either BytesIO or TextIO (universal_newlines=False/True)?
As described in comment #6 I added the boolean textio= argument to OperatingSystem.Start Process,
using TextIO on request instead of default BytesIO.
Is this a useful approach?
Change the default?
Handle it completely different? :)

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by zimmermann.code on 28 Nov 2013

Just a little addition:
Releasing for Python 3 is technically already possible.
I also made setup.py (using 2to3) fully work with Python 3.
And the installation result works too ;)
I only need to handle the few remaining atest fails and add the 2to3 stuff to run_utests.py.
And maybe make some manual changes in the utests code.
Just to make it complete in its current approach.

And the docs could need some updates for things that behave a bit different in Python 3 (mainly bytes/str related stuff).

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by @pekkaklarck on 29 Nov 2013

Unfortunately we cannot yet drop Jython 2.5 support in the main line. I hope that things look different already in few months when we start thinking about next Robot major release. I hope we can drop Python/Jython 2.5 and also 2.6 then and make the framework officially Python 3(.3) compatible.

Until that happens, having a separate project and release targeting Python 3 support would be awesome. The code could be translated with 2to3 and manually edited where needed. The project would also help targeting official Python 3 support later by exposing compatibility issues with str/unicode. As long as those issues are in libraries they are easy to handle one way or other.

With OperatingSystem.Start Proces I would say that it's best to just leave it. I would like to deprecate it in favor of Process.Start Process sooner or later anyway. If further configuration is needed, it's much easier to add to Process.Start Process that takes free **config.

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by zimmermann.code on 29 Nov 2013

I reverted OperatingSystem.Start Process, keeping Popen(..., universal_newlines=True),
to always use text mode, to keep the atests passing.

Process.Start Process already uses text mode as default.
I added a boolean bytesio= argument to switch to binary mode.
Better rename it somehow?

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by @pekkaklarck on 30 Nov 2013

I think binary_mode would be more describing that bytesio and might even consider super describing no_universal_newlines.

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by zimmermann.code on 16 Dec 2013

Some time passed... :)

I changed bytesio to binary_mode,
merged with 2.8.3,
fixed some remaining issues in src/,
added more workarounds to the atests,
added 2to3 to run_utests.py,
and made the utests ERROR-free in Python 3.

Only some utest FAILS left to fix before I release robotframework-python3...

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by zimmermann.code on 4 Jan 2014

Finally, and with even more fixes and (almost) FAIL-free utests, the first release:

https://pypi.python.org/pypi/robotframework-python3/2.8.3

The "almost" above is due to a strange behavior of the importer tests.
When the import files are dynamically created just before they should be imported,
they often can't be found.
It only happens in Python 3, and randomly.
I added some additional checks to see if the files exist and are readable.
Everything fine, but the import functions can't find them.

Complete diff to 2.8.3:

https://bitbucket.org/userzimmermann/robotframework-python3/branches/compare/python3-2.8.3..2.8.3#diff

I further merged with the commits after 2.8.3
and will now start with statically applying the (necessary) 2to3 diffs,
keeping the code Python 2.7 compatible.

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by zimmermann.code on 11 Feb 2014

Merged with 2.8.4 and made the whole robot code directly Python 2.7/3.3+ compatible,
by applying backwards compatible 2to3 diffs and integrating six.
I also simplified and/or extended some of the existing workarounds.

I used six for the following cases:

  • all version queries with six.PY2/PY3
  • basestring --> six.string_types
  • (int, long) --> six.integer_types
  • to keep 'unicode' --> from six import text_type as unicode
  • metaclass --> @⁠six.add_metaclass

Diff of direct compatibility step:
https://bitbucket.org/userzimmermann/robotframework-python3/branches/compare/3da31c1..939ab93#diff

Complete diff to 2.8.4:
https://bitbucket.org/userzimmermann/robotframework-python3/branches/compare/3da31c1..2.8.4#diff

I didn't test the code with Jython/IronPython 2.7 so far.

I will continue with the utest/atest py code.

I still do dynamic PY3 conversions of the atest data in run_atests:

  • Remove u prefixes and replace \u and \x codes with chars for matching string reprs
  • Remove L suffixes for evaluating number literals and matching reprs

This needs a static solution... maybe someone interested in helping? :)
I also need to implement a generic way to check for PY2/3 in testdata/

And finally... there are some remaining PY3 bytes/str issues in Remote library,
which let some atests fail...

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by zimmermann.code on 9 May 2014

robotframework-python3 2.8.4 :)

https://pypi.python.org/pypi/robotframework-python3

  • No 2to3 left.
  • Still dynamic atest data conversions.
  • Working Remote Library.

Complete diff to official 2.8.4:
https://bitbucket.org/userzimmermann/robotframework-python3/branches/compare/python3-2.8.4..2.8.4#diff

There is a strange atest issue with keywords/return_values.txt
In PY3 I get "Incompatible XML element 'errors'.".
But there don't seem to be any other problems with XML generating/parsing.
I didn't have the time to find the root cause yet.

@userzimmermann

This comment has been minimized.

Copy link
Contributor

commented Sep 8, 2014

I merged with 2.8.5, added new (and improved some old) compatibility changes and made a new release:

https://pypi.python.org/pypi/robotframework-python3

Complete diff to official 2.8.5:
https://bitbucket.org/userzimmermann/robotframework-python3/compare/python3-2.8.5..2.8.5#diff

The fork is still based on the old mercurial repo from https://robotframework.googlecode.com
I will migrate it soon...

@pekkaklarck

This comment has been minimized.

Copy link
Member

commented Sep 8, 2014

It would be great if you could migrate to GitHub. Would be easier to track changes.

By the way, I've been working with moving installation instructions to INSTALL.rst (#1779). My idea is to use same instructions that in User Guide and there I already mention your fork. Probably should add a note about Python versions and Py3 fork also to README.rst.

@userzimmermann

This comment has been minimized.

Copy link
Contributor

commented Oct 2, 2014

I migrated all my commits to a python3 branch in my GitHub fork:

https://github.com/userzimmermann/robotframework/tree/python3

And I'm working on merging it up to date. You can always get a complete diff at this URL:

userzimmermann/robotframework@master...python3

@uSpike

This comment has been minimized.

Copy link

commented Feb 10, 2015

Word on the street (#1912 (comment)) is that RF 2.9 will drop j/python 2.5 support. Does that mean that the python3 support can be / will be integrated in RF 2.9?

@pekkaklarck

This comment has been minimized.

Copy link
Member

commented Feb 10, 2015

I doubt we have time for that. The main goal of dropping 2.5 support is making it easier to add 3.x support later. If required changes are not too big, we can consider that already in RF 2.9, though.

@userzimmermann

This comment has been minimized.

Copy link
Contributor

commented Feb 26, 2015

Status update :)

The latest release of my Python 3 compatibility port is 2.8.7:

https://github.com/userzimmermann/robotframework/releases/tag/python3-2.8.7
https://pypi.python.org/pypi/robotframework-python3/2.8.7

Complete diff to original 2.8.7:

userzimmermann/robotframework@2.8.7...python3-2.8.7

Further I merged with recent commit 3a7571d which already contains many Python 2.5 incompatible code updates as discussed in #1928 and lots of code restructuring.
I got a lot of merge conflicts :D but they are solved.
There are only a few new utests and atests which need some more compatibility adaptions.

You can always get the latest complete code diff at:

userzimmermann/robotframework@master...python3

I also started to create issues for atests FAILing on Python 3:

https://github.com/userzimmermann/robotframework/issues

So anyone can participate in solving them. They generally only need workarounds in the atests themselves.

And the best thing is... actually not the best but very close ;)
It is still running on Python 2.6!!! (At least with Python 2.6.8)
I remember there were some issues with Python 2.6 in the beginning of my fork. I can't remember exactly. But I declared my fork incompatible with Python 2.6 because of them and defined 2.7 as the only compatible Python 2 version. I was just curious and ran the tests of the current code with Python 2.6 and it worked. Maybe the earlier issues were only related to older Python 2.6.x versions or somehow resolved by using six.
I would appreciate if someone could run the tests with some older Python 2.6 version.

@userzimmermann

This comment has been minimized.

Copy link
Contributor

commented Mar 3, 2015

I basically agree that most of my six use can be handled with some simple custom compatibility layer.
I also used only a small fraction of all the six features.
However, there are some features like six.moves,with_metaclass and reraise which need some more effort to reimplement and which end up in reinventing the wheel to some degree.
Anyway, I started adding PYTHON2/3 vars and is_<type>/<type>_like functions and some common type definitions and imports to robot.utils and opened pull request #1937 for this.

@pekkaklarck

This comment has been minimized.

Copy link
Member

commented Mar 3, 2015

I know there are cases where six would really help us, but I'm not yet convinced we need it so much that having an additional dependency is worth it. Handling differences between str/unicode/int/long types ought to be easy as PR #1937 indicates. Do you have an idea how many places we have where six.moves, with_metaclass or reraise are needed? If there are only few, special casing them shouldn't be a big problem.

@userzimmermann

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2015

It took a while... I was quite busy the last weeks... but: 2.9a1 !

https://github.com/userzimmermann/robotframework/releases/tag/python3-2.9a1
https://pypi.python.org/pypi/robotframework-python3/2.9a1

Complete diff to original 2.9a1:

userzimmermann/robotframework@2.9a1...python3-2.9a1

Now I will continue with #1937

@userzimmermann

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2015

It's been a while... again ;)

I merged with 2.9rc1 and things are still working very well!
Lots of merging conflicts due to #1990 but nice to see how the diff is slowly shrinking... ;)

userzimmermann/robotframework@master...python3

I will do some some further cleanup and resolve an additional atest issue and then make a robotframework-python3-2.9rc1 release... trying to get it done this evening

@userzimmermann

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2015

@userzimmermann

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2015

@pekkaklarck After applying is_<type> checks from #1990 to the test code, I would like to introduce __future__.print_function to the standard code base as next migration step. What do you think?

@pekkaklarck

This comment has been minimized.

Copy link
Member

commented Jul 31, 2015

print_function sounds fine. In many places it would be a good idea to avoid print in favor of robot.api.logger or sys.stdout.write, depending on the context.

@pekkaklarck

This comment has been minimized.

Copy link
Member

commented Nov 10, 2015

Official Python 3 support is very close. All acceptance tests pass on Linux, OSX and Windows, and installation with Python 3 seems to work fine. Final tasks to do before this issue can be considered done:

  • setup.py must be updated to list Python 3 as a supported version.
  • Go through TODOs and FIXMEs in code and see is there anything Python 3 related.
  • Document supported Python versions (2.6, 2.7, 3.3->) in User Guide, README.rst, and INSTALL.rst. UPDATE: Partly done. Remaining tasks tracked by #2222.
  • Decide do we want to create pybot3 script to avoid overwriting pybot script created by Python 2. Alternatively we could start using generic robot script with all interpreters. UPDATE: We decoded to go with the generic robot script. Issue #2216 covers that.

@pekkaklarck pekkaklarck self-assigned this Nov 18, 2015

@pekkaklarck pekkaklarck referenced this issue Nov 18, 2015
4 of 4 tasks complete
@pekkaklarck

This comment has been minimized.

Copy link
Member

commented Nov 18, 2015

Final tasks listed in my previous comment are all either done or handed by other issues. I guess we can consider this issued finally done! Go Python 3!!!!!!!!!!!!!!!!!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.