Get rid of 2to3 #397

merged 43 commits into from Jan 20, 2013


None yet

8 participants

pv commented Jan 5, 2013

This PR converts the code base so that it runs both on Python >= 2.6 and 3.x, without any conversions needed.

The size of the diff is smaller than what I expected, and is maybe possible to review as a single chunk... Some more testing is probably needed, but the test suite passes and I didn't see in the first pass anything too suspicious remaining. Discussion on the ML is probably required before merging.

The majority of changes are relative imports, changing print statements to print(), inserting list() to correct locations, and dealing with dissappearing unicode and unicode literals. The conversion was done by first taking the 2to3 results, and then back-porting that to Python 2, trimming out unnecessary changes.

The six compatibility module is also now bundled to deal with some compatibility issues.

The impact for further development would be that new code written should run as-is both on Python 2.6 and 3.x. The main annoyance for scipy code are probably the changed semantics of map(), zip(), keys() et al., which requires using list() in correct places. The other Python 3 issues are probably of not much consequence.

The upside of this change is un-ghettoization of Python 3, and getting rid of the build-time conversion process (which has its own annoyances).

pv added some commits Jan 5, 2013
@pv pv MAINT: Automated 2to3 conversion of the code base 3add620
@pv pv BLD: Remove 2to3 conversion step from the build 74e42cf
@pv pv ENH: lib: bundle six Python 2/3 compatibility module 410f27e
@pv pv PY23: tools: single-sourcify 8765151
@pv pv PY23: special: remove unnecessary list()'s + remove unused file 57410d5
@pv pv PY23: cluster: remove unnecessary list()'s af33e9a
@pv pv PY23: cluster: fix string type handling e730a5a
@pv pv PY23: constants: remove unnecessary list()'s 3d89473
@pv pv PY23: fftpack: print -> print_ e211e2c
@pv pv PY23: integrate: print -> print_ e965867
@pv pv PY23: interpolate/rbf: fix function code handling + callable() 5131c12
@pv pv PY23: interpolate: remove unnecessary list()'s + minor cleanup b220321
@pv pv PY23: io: remove unnecessary list()'s 4c1497e
@pv pv PY23: io/arff: import next() on Py2 7a9ff05
@pv pv PY23: io: fix string handling + add some missing imports + fix int ty…
…pe handling
@pv pv PY23: lib: port 68362ad
@pv pv PY23: linalg: callable() c6298be
@pv pv PY23: linalg: print -> print_ 16bfcf9
@pv pv PY23: linalg: remove unnecessary list()'s + minor fixes 1fc7594
@pv pv PY23: misc: remove unnecessary list()'s cacf46d
@pv pv PY23: ndimage: fix type handling a01536a
@pv pv PY23: ndimage: remove unnecessary list()'s + minor fixes 6ad855d
@pv pv PY23: odr: print -> print_ + remove unnecessary list()'s 155cd94
@pv pv PY23: optimize: callable() + exec + builtins c1cbb7d
@pv pv PY23: optimize: remove unnecessary list()'s 787b8d0
@pv pv PY23: optimize: print -> print_ 2f14a14
@pv pv PY23: signal: callable() + remove unnecessary list()'s f8190b2
@pv pv PY23: linalg: print -> print_ 86b09ae
@pv pv PY23: sparse.linalg: iteritems + remove unnecessary list()'s + other …
@pv pv PY23: sparse: __bool__/__nonzero__ + remove unnecessary list()'s + it…
…eritems + izip
@pv pv PY23: spatial: print -> print_ d44cccc
@pv pv PY23: spatial: callable + unicode literals + string type handling adf9d24
@pv pv PY23: spatial: remove unnecessary list()'s a2af3a1
@pv pv PY23: stats: remove unnecessary list()'s d413935
@pv pv PY23: stats: iteritems 387a5c3
@pv pv PY23: stats: string type handling + callable + method function + func…
…tion code
@pv pv PY23: restore xrange()
Restores the range() vs. xrange() situation before 2to3
@pv pv TST: interpolate: fix list comprehension Py2/3 scope difference intro…
…duced by 2to3
@pv pv PY23: signal: convert 61b8136
@pv pv MAINT: replace asbytes(literal) by bytes literals ce73f19
rgommers commented Jan 5, 2013

Haven't tested yet, but +10 for getting rid of 2to3.

rgommers commented Jan 5, 2013

One question after a quick browse: why the mix of print() and print_()? If the former works it's probably preferable, saves an import and is better for code highlighting in editors.

pv commented Jan 5, 2013

print() works on Python 2.6 only when there is a single argument. In other cases, I switched whole files to print_(), for uniformity.

Carreau commented Jan 5, 2013

We had a similar discussion on IPython ,
We found that for review and slow transition, one could apply the 2to3 fixes independently and disable them one by one in so that you don't "introduce" new incompatible fixes.

It is much more easier to deal with as some fixes are widely accepted, and other less.

Thomas Pointed out today the uprefix project that seems nice and allow u prefix on 3.2, which is much better that having six everywhere.

Hope that will help.

jakevdp commented Jan 5, 2013

@pv - this is great! Thanks for putting in the work. I'm very much in favor of this sort of change: let's hope it will inspire similar changes in upstream packages!

pv commented Jan 5, 2013

@Carreau: thanks for the Ipython discussion link.

I think what differs in Scipy from Ipython in that (i) Scipy's Python code is much simpler and less intraconnected, and (ii) we use little Unicode. So, six.u() is not too large an annoyance, and I think the other semantic differences are tolerable as they are.

We probably should use from __future__ import print_function instead of six.print_().

@dengemann dengemann referenced this pull request in mne-tools/mne-python Jan 15, 2013

ENH: support py3k #388

dwf commented Jan 16, 2013

Just wanted to chime in and thank you for the hard work, @pv. It's unglamorous yak shaving but it's very valuable indeed.


Last commit looks good, and still passes all tests for me. Time to merge this.

@rgommers rgommers closed this Jan 20, 2013

And thank you to Pauli, this is extremely helpful!

@rgommers rgommers reopened this Jan 20, 2013
@rgommers rgommers merged commit 0e9a7db into scipy:master Jan 20, 2013

Is there a similar idea for NumPy in the works?

charris commented Jan 20, 2013

There is such an idea, but no action yet ;)

charris commented Jan 21, 2013

I've made a start, but it will take a while...


this breaks line 93-94
npos = ((pos + _BLOCKSIZE - 1) / _BLOCKSIZE) * _BLOCKSIZE
npos is now a float

though this file should probably be replaced with the dumbdbm from python
probably only _addval was patched for weave pre 2006, possible its not needed anymore

pv replied Feb 5, 2013

Yes, the __future__ import brings it in line with the behavior on Python 3.

I don't think this is actually used by weave, as it uses this only as a fallback to Python's shelf module.
It's used as a back-end for save_as_module, but I don't think that is very commonly used.


its used when python falls back to dumbdbm, I hit this in such an environment today


there erros here

TypeError: can't multiply sequence by non-int of type 'float'

the fix is simply make it an integer division again, but I think it would be better to just remove the file.

pv replied Feb 5, 2013

Python's shelve with dumbdbm backend doesn't seem still to work with scipy.weave.catalog, so we probably need to stick with the patched version. (I don't know what was patched and why --- this stuff dates from pre-2002 --- only that some tests start to fail otherwise)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment