Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLN/ENH/BLD: Remove need for 2to3 for Python 3. #4384

Merged
merged 11 commits into from Jul 29, 2013

Conversation

@jtratner
Copy link
Contributor

commented Jul 27, 2013

Fixes #4375 and #4372.

Many changes to make codebase compatible in 2 and 3. For range, zip, map, etc. tried to favor iterators over needing to use lists.

Changes:

  • No more 2to3 in setup.py
  • merges util/compat and util/py3compat into pandas/compat
  • incorporates useful parts of the six library into compat (+ adds SIX to LICENSES)
  • defaults to using iterators in both Python 2 and Python 3 for range, zip, map, and filter
  • Adds lrange, lzip, lmap, and lfilter, which wrap corresponding iterator methods with lists
  • Improved type checks (where appropriate --> many need to be rigid to work with pandas)
  • various other utilities to be Py2/3 compatible (iteritems, iterkeys, itervalues, etc.)
  • deprecates iterkv with a warning (no longer necessary because library no longer uses 2to3)
  • compatibility wrapper around dateutil for handling unicode when version <= 2.0 + 1.5 dateutil build in Travis

Switches everything to use special iteritems, so can deprecate as discussed in #4372.

This iteritems use "iteritems" method if available and otherwise uses items.

Checklist of packages to run through 2to3 to check:

  • compat
  • core
  • io
  • rpy
  • sandbox
  • sparse
  • src (python files only)
  • stats
  • tests
  • tools
  • tseries
  • util

Other tasks:

  • alias __bool__ to __nonzero__
@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 27, 2013

I think GitHub might explode if I click on the files changed tab 💥

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 27, 2013

I think filter is also an iterator in Python 3 IIRC

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 27, 2013

@cpcloud it is...still coming up on my list :P

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 27, 2013

@cpcloud actually, only used in one place in library and it's used as an iterator anyways. still need to check for itertools changes.

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 27, 2013

great. i think i might use filter in eval-3393, i'll check

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 27, 2013

@cpcloud just put from six.moves import filter at the top of the file and you'll be fine. It just uses itertools.ifilter under the hood, so slap a list on it otherwise. Very simple

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 27, 2013

cool thanks! pandas is going to rule the world soon.

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 27, 2013

i hope python 3 testing is going to be faster after this .... those 2to3 runs seem like they take FOREVER

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 27, 2013

@cpcloud I think it makes the entire test suite a little faster...not sure.

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 27, 2013

nice looks like travis build times are 3-4 mins faster that is cool

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 27, 2013

@cpcloud does Cython have any changes for Python 3?

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 27, 2013

I believe all of those issues are taken care of during translation from the Cython to C. I don't see why any Cython would need to be changed. Are you getting compilation errors?

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 27, 2013

@cpcloud guess I can empirically see that that's not true...just gone through so many changes (and I can't get py3 pandas to work on my mac).

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 27, 2013

@cpcloud I'm getting all these weird stata reader errors. Do you have any ideas on what might be causing them?

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 28, 2013

@jtratner can u link me to a build with those errors?

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 28, 2013

@cpcloud check out the first few errors on here: https://travis-ci.org/jtratner/pandas/jobs/9563671

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 28, 2013

have you tried stepping with a debugger?

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 28, 2013

@cpcloud I've been having trouble getting pandas with Python 3 to install on my mac, which is why I haven't been able to step through.

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 28, 2013

@cpcloud but eventually I'll figure out how to get it to work

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 28, 2013

ah ok... it's weird because i think you actually have to setup.py install it for 2to3 to run which is annoying whenever you make a change...you have to keep doing it

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 28, 2013

@cpcloud well, the current build has 2to3 disabled, so you don't actually need to do that. I'm trying a separate build with 2to3 enabled to see if it resolves the problem.

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 28, 2013

indeed, i hadn't pulled down your PR in my py3.2 venv yet 😄

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 28, 2013

anyway my dev version works so just keep at it i think you'll get it ... i would offer to clone my env but i'm not using mac

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 28, 2013

@cpcloud okay, pretty sure I can figure it out. It doesn't fail with any of the stata errors with 2to3, so that means I just have to apply some of those fixes and it will work :).

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 28, 2013

for some reason the path_or_buf.read() calls are not reading anything...

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 28, 2013

(in _read_header)

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 28, 2013

@cpcloud thanks. And I found all the non-stata errors are resolved by 2to3'ing pandas/core.

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 28, 2013

ahh so the problem is actually not in _read_header you need to convert the calls to np.where(map(...)) to np.where(list(map())) then you're golden...i just tried it, it works

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 28, 2013

that's in the data method

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 28, 2013

i love getting to the bottom of things!

@jreback

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2013

@jtratner

rebases all worked (with a little TLC..) except for 1 thing:

FAIL: test_iterkv_deprecation (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/home/jreback/pandas/build/lib.linux-x86_64-3.3/pandas/tests/test_frame.py", line 10316, in test_iterkv_deprecation
    self.mixed_float.iterkv()
  File "/usr/local/lib/python3.3/contextlib.py", line 55, in __exit__
    next(self.gen)
  File "/mnt/home/jreback/pandas/build/lib.linux-x86_64-3.3/pandas/util/testing.py", line 1058, in assert_produces_warning
    % expected_warning.__name__)
AssertionError: Did not see expected warning of class 'DeprecationWarning'.
``
which works when I run only tests/test_frame.py but not all tests...
?
@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2013

Maybe I put it in as future warning by accident? Or did we lose the
commit/change that added the deprecation?
On Jul 29, 2013 9:39 PM, "jreback" notifications@github.com wrote:

@jtratner https://github.com/jtratner

rebases all worked (with a little TLC..) except for 1 thing:

FAIL: test_iterkv_deprecation (pandas.tests.test_frame.TestDataFrame)

Traceback (most recent call last):
File "/mnt/home/jreback/pandas/build/lib.linux-x86_64-3.3/pandas/tests/test_frame.py", line 10316, in test_iterkv_deprecation
self.mixed_float.iterkv()
File "/usr/local/lib/python3.3/contextlib.py", line 55, in exit
next(self.gen)
File "/mnt/home/jreback/pandas/build/lib.linux-x86_64-3.3/pandas/util/testing.py", line 1058, in assert_produces_warning
% expected_warning.name)
AssertionError: Did not see expected warning of class 'DeprecationWarning'.
``
which works when I run only tests/test_frame.py but not all tests...
?


Reply to this email directly or view it on GitHubhttps://github.com//pull/4384#issuecomment-21764382
.

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2013

@jreback oh didn't see your last line, I have an idea...let me look.

@jtratner jtratner deleted the jtratner:better-python3-compat branch Jul 30, 2013

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2013

@jreback which branch?

@jreback

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2013

msgpack3 (simpler one)

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2013

Why does it require pytest?

======================================================================
ERROR: Failure: ImportError (No module named 'pytest')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/failure.py", line 38, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/loader.py", line 413, in loadTestsFromName
    addr.filename, addr.module)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/imp.py", line 160, in load_module
    return load_source(name, filename, file)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/imp.py", line 109, in load_source
    return _LoadSourceCompatibility(name, pathname, file).load_module(name)
  File "<frozen importlib._bootstrap>", line 586, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1023, in load_module
  File "<frozen importlib._bootstrap>", line 1004, in load_module
  File "<frozen importlib._bootstrap>", line 562, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 869, in _load_module
  File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
  File "/Users/jtratner/projects/python/pandas3/pandas/tests/test_msgpack/test_except.py", line 4, in <module>
    from pytest import raises
ImportError: No module named 'pytest'

======================================================================
ERROR: Failure: ImportError (No module named 'pytest')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/failure.py", line 38, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/loader.py", line 413, in loadTestsFromName
    addr.filename, addr.module)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/imp.py", line 160, in load_module
    return load_source(name, filename, file)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/imp.py", line 109, in load_source
    return _LoadSourceCompatibility(name, pathname, file).load_module(name)
  File "<frozen importlib._bootstrap>", line 586, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1023, in load_module
  File "<frozen importlib._bootstrap>", line 1004, in load_module
  File "<frozen importlib._bootstrap>", line 562, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 869, in _load_module
  File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
  File "/Users/jtratner/projects/python/pandas3/pandas/tests/test_msgpack/test_obj.py", line 4, in <module>
    from pytest import raises
ImportError: No module named 'pytest'

======================================================================
ERROR: Failure: ImportError (No module named 'pytest')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/failure.py", line 38, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/loader.py", line 413, in loadTestsFromName
    addr.filename, addr.module)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/imp.py", line 160, in load_module
    return load_source(name, filename, file)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/imp.py", line 109, in load_source
    return _LoadSourceCompatibility(name, pathname, file).load_module(name)
  File "<frozen importlib._bootstrap>", line 586, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1023, in load_module
  File "<frozen importlib._bootstrap>", line 1004, in load_module
  File "<frozen importlib._bootstrap>", line 562, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 869, in _load_module
  File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
  File "/Users/jtratner/projects/python/pandas3/pandas/tests/test_msgpack/test_pack.py", line 6, in <module>
    from pytest import raises, xfail
ImportError: No module named 'pytest'

======================================================================
ERROR: Failure: ImportError (No module named 'pytest')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/failure.py", line 38, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/loader.py", line 413, in loadTestsFromName
    addr.filename, addr.module)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/imp.py", line 160, in load_module
    return load_source(name, filename, file)
  File "/Users/jtratner/.virtualenvs/pandas3/lib/python3.3/imp.py", line 109, in load_source
    return _LoadSourceCompatibility(name, pathname, file).load_module(name)
  File "<frozen importlib._bootstrap>", line 586, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1023, in load_module
  File "<frozen importlib._bootstrap>", line 1004, in load_module
  File "<frozen importlib._bootstrap>", line 562, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 869, in _load_module
  File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
  File "/Users/jtratner/projects/python/pandas3/pandas/tests/test_msgpack/test_sequnpack.py", line 7, in <module>
    from pytest import raises
ImportError: No module named 'pytest'

======================================================================
FAIL: test_iterkv_deprecation (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jtratner/projects/python/pandas3/pandas/tests/test_frame.py", line 10316, in test_iterkv_deprecation
    self.mixed_float.iterkv()
  File "/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/lib/python3.3/contextlib.py", line 55, in __exit__
    next(self.gen)
  File "/Users/jtratner/projects/python/pandas3/pandas/util/testing.py", line 1058, in assert_produces_warning
    % expected_warning.__name__)
AssertionError: Did not see expected warning of class 'DeprecationWarning'.

----------------------------------------------------------------------
Ran 3537 tests in 166.243s
@jreback

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2013

its there from where when msgpack tests originally created; been meaning to take it out

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2013

@jreback easy to put raises-equivalent into util/testing

Also, you should change this line:

    unpacker = Unpacker(six.BytesIO(b'foobar'), read_size=3)

to the BytesIO from pandas.compat

@jreback

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2013

what about the rest of them that directly import from six?

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2013

hold on, I have a diff for you.

@jreback

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2013

I fixed them all....how do I do raw bytes? b'foo' ?

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2013

yeah, that's cross-compatible. I just posted a commit on jtratner/msgpack3 that fixes everything. Main thing is you need to pass string encoding in test_seq.py and wrap around chr

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2013

(hadn't seen your note before that...)

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2013

binarydata = [bytes(chr(i), 'utf-8') for i in range(256)]

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2013

nvm, my example doesn't work for that

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 30, 2013

actually assert_raises is in nose.tools

@jreback

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2013

jreback@d6c8279

look at test_msgpack/test_seq.py

I had to hack a bit, encoding in py3, but passing str in py2...seems to work, but never could figure out why

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2013

@jreback well, in Python 3, everything is unicode, whereas in py2 everything is encoded by default, so using bytes requires setting explicit coding. In python 2 bytes is exactly the same as str.

@jreback

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2013

@jtratner I resolved that issue with assert_produces_warning....

I was calling iterkv (in the msgpack ).....which was working correctly. I suspect that the warnings filter somehow was not turned back on correctly though in assert_produces_warning....because this happened before the test failure (of the test for calling iterkv)....

IOW I was calling iterkv accidently, which produced a warning (but I think globally warnings are somehow turned off for DeprecationWarning).....

then when it hit the specific test, warnings were still turned off!

weird

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2013

warnings are incredibly finicky. I'm -1 on checking for things like
DeprecationWarnings, especially because I think they don't appear in a
standard interpreter.

@jreback

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2013

agreed.....I in fact usually when I test something like this I make it raise while I am testing (which in this case would have found the cases where I didn't fix the deprecated routine)....

so

maybe need something that is a sub-class of DeprecationWarning that assert_produces_warning can really 'catch' (but in normal usage just spews a warning)?

just thinking out loud here

@cpcloud

This comment has been minimized.

Copy link
Member

commented Jul 30, 2013

there's no runtime switch to turn on deprecation warnings?

@jreback

This comment has been minimized.

Copy link
Contributor

commented Aug 5, 2013

@jtratner

This comment has been minimized.

Copy link
Contributor Author

commented Aug 6, 2013

python-modernize was an amazingly helpful tool in this - it wraps 2to3 in some useful ways. Running it once gets you 90% of the way there. The big issue for me was that I wanted to use iterators as much as possible so 2.X wouldn't have unnecessary list calls - that took the majority of the time (as well as integrating the six library). I don't know whether it had a performance boost, but if I hadn't cared about that, it probably would've been easier.

There were two particularly hard parts:

  1. Cross-compatible urllib/httplib imports and exceptions.
  2. That weird unicode error we found and fixed with StringIO.
@nburns

This comment has been minimized.

Copy link
Contributor

commented on pandas/core/frame.py in 6c304c7 Apr 7, 2015

Out of curiosity why did you leave the conditional in here? (when data.next() became next(data))

This comment has been minimized.

Copy link
Member

replied Apr 7, 2015

probably just an oversight. It is still in the code, so if you want to a PR to clean this up, very welcome!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.