[WIP] backport from master to the 1.7.x release branch #439

Merged
merged 35 commits into from Sep 16, 2012

Projects

None yet

10 participants

@certik
Contributor
certik commented Sep 12, 2012

I have backported the patches from master. There is a lot of patches, so I still need to carefully go through them and determine whether we need to apply all of them or only a subset.

I will be rebasing this branch, dropping or adding commits until it is polished and ready to merge.

If you see any problems, let me know.

njsmith and others added some commits Jul 18, 2012
@njsmith @certik njsmith [FIX] Add missing header so separate compilation works again
(Broken by PR #350.)

Should be applied to maintenance/1.7.x as well.
a1561c2
@cgohlke @certik cgohlke Fix tests for empty shape, strides and suboffsets on Python 3.3 d72280f
@rlamy @certik rlamy Copy bytes object when unpickling an array 470486b
@rlamy @certik rlamy Re-enable unpickling optimization for large py3k bytes objects.
Mutating a bytes object is theoretically unsafe, but doesn't cause any
problem in any existing version of CPython.
a022015
@certik @certik certik FIX: Fixes the PyUnicodeObject problem in py-3.3
Previously NumPy did not compile in Python 3.3 due to the changes in PEP 393.
This patch simply calls PyUnicode_FromKindAndData() from the new Python 3.3 API
and possibly swaps the data before calling it if needed. The data in NumPy is always UCS4 and the PyUnicode_FromKindAndData() internally converts it to UCS1, UCS2 or UCS4 depending on the maximum code point.

The following tests now fail, because they produce invalid unicode in the
process (will be fixed in the next patch):

======================================================================
ERROR: Check byteorder of 0-dimensional objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ondrej/py33/lib/python3.3/site-packages/numpy/core/tests/test_unicode.py", line 277, in test_values0D
    self.assertTrue(ua[()] != ua2[()])
SystemError: invalid maximum character passed to PyUnicode_New

======================================================================
ERROR: Check byteorder of multi-dimensional objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ondrej/py33/lib/python3.3/site-packages/numpy/core/tests/test_unicode.py", line 297, in test_valuesMD
    self.assertTrue(ua[0,0,0] != ua2[0,0,0])
SystemError: invalid maximum character passed to PyUnicode_New

======================================================================
ERROR: Check byteorder of single-dimensional objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ondrej/py33/lib/python3.3/site-packages/numpy/core/tests/test_unicode.py", line 286, in test_valuesSD
    self.assertTrue(ua[0] != ua2[0])
SystemError: invalid maximum character passed to PyUnicode_New

======================================================================
ERROR: Check byteorder of 0-dimensional objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ondrej/py33/lib/python3.3/site-packages/numpy/core/tests/test_unicode.py", line 277, in test_values0D
    self.assertTrue(ua[()] != ua2[()])
SystemError: invalid maximum character passed to PyUnicode_New

======================================================================
ERROR: Check byteorder of multi-dimensional objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ondrej/py33/lib/python3.3/site-packages/numpy/core/tests/test_unicode.py", line 297, in test_valuesMD
    self.assertTrue(ua[0,0,0] != ua2[0,0,0])
SystemError: invalid maximum character passed to PyUnicode_New

======================================================================
ERROR: Check byteorder of single-dimensional objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ondrej/py33/lib/python3.3/site-packages/numpy/core/tests/test_unicode.py", line 286, in test_valuesSD
    self.assertTrue(ua[0] != ua2[0])
SystemError: invalid maximum character passed to PyUnicode_New
d67785b
@certik Chris Fix memory leak in concatenate.
Temporary array was not being freed after use.
d5d6894
@certik @certik certik FIX: Make sure the tests produce valid unicode
The tests are testing byte order for unicode, so we can only use such unicode
data, so that both versions (swapped and unswapped) are valid unicode.
8141e1e
@certik @certik certik Follow the C guidelines 31660d0
@certik @certik certik Use PyUnicode_DecodeUTF32()
This function handles the swapping automatically and it returns a unicode
object in one of: UCS1, UCS2 or UCS4 internal Python format.
99c9397
@cgohlke @certik cgohlke Allow long numbers in numpy.rec.array formats string 77da3f8
@cgohlke @certik cgohlke Add test for long numbers in numpy.rec.array formats string 24da7f6
@cgohlke @certik cgohlke Add test for long number in shape specifier of dtype string 5cb80f8
@cgohlke @certik cgohlke Remove test_recarray_from_long_formats fbad4a7
@cournape @certik cournape BUG: fix bento build. 3930881
@certik @certik certik pavement.py: rename "yop" to "atlas" 416af9a
@rgommers @certik rgommers MAINT: silence DeprecationWarning in np.safe_eval().
It comes from the Python compiler package, which isn't available on Python 3.x.
We already handle that issue by instead importing the ast module.
7583adc
@certik certik FIX: bug in np.where and recarray swapping
Fixes #369 and the Ticket 2185

Thanks to @seberg for providing the patch.
996a9fb
@jayvius @certik jayvius Fix returned copy
Fix returned copy so that copy of view with offsets copies only fields in view, not all the fields from original array. Also add unit tests to make sure this doesn't break when copy is fully deprecated in favor of returning a view.
124a608
@certik certik FIX: simplify the import statement f65ff87
@stefanv @certik stefanv Improve ndindex execution speed. 5a471b5
@certik certik BUG: Add a test for Ticket #2189 ca29849
@mwiebe @certik mwiebe BUG: Fix ticket #1588/gh issue #398, refcount error in clip
This patch enforces a strict dichotomy for the variables 'indescr'
and 'newdescr', so they are either NULL, or they own a reference.
Following the consequences of this allowed the reference error
to be tracked down.
7b5dba0
@certik certik BUG: Add a test for Ticket #1588
The bug was fixed by the previous patch.
1ee4a00
@certik certik FIX: Add a test for Ticket #2066 2f28db6
@stefanv @certik stefanv Retain backward compatibility. Enforce C order. 3dc3b1b
@certik certik FIX: Regenerate mtrand.c with Cython 0.17
Fixes #416
6e3e289
@njsmith
Member
njsmith commented Sep 13, 2012

I would just merge them as you confirm each one is correct, instead of trying to review a giant pile together...

Anyway skimming the list above: 5a471b5 is a new optimization, not a bugfix, so technically shouldn't be merged into the stable branch, but it's small enough that it probably won't matter.

56596a0fa04c7474967747d21a9df61a95edca62 was an error and immediately reverted in master, so it shouldn't be here.

cournape and others added some commits Aug 26, 2012
@cournape @certik cournape REF: remove unused imports in bscript. cb0de72
@cournape @certik cournape BUG: forgot to build _dotblas in bento build. 095a3ab
@cournape @certik cournape BUG: fix custom post_check. 6da087e
@cournape @certik cournape BUG: fix npymath install location. 07e02d0
@87 @certik 87 BUG: Fix for issues #378 and #392
This should fix the problems with numpy.insert(), where the input values
were not checked for all scalar types and where values did not get inserted
properly, but got duplicated by default.
8460514
@87 @certik 87 TST: Add extra test for multidimensional inserts. 9ee9984
@87 @certik 87 TST: Add test for boolean insert 65ec87e
@njsmith @certik njsmith FIX: loosen numerical tolerance in test_pareto()
The problem was that in 32bit Ubuntu 12.04, one gets the following:

>
/home/njs/numpy/.tox/py27/local/lib/python2.7/site-packages/numpy/random/tests/test_random.py(363)test_pareto()
-> np.testing.assert_array_almost_equal(actual, desired, decimal=15)
(Pdb) actual[1, 0]
52828779.702948704
(Pdb) desired[1, 0]
52828779.702948518

and the test was comparing the numbers to 1e-14, which obviously
failed.

Fixes #424.
d01354e
@njsmith @certik njsmith FIX: use py24-compatible version of virtualenv on Travis
Travis recently upgraded to virtualenv 1.8, which has dropped support
for Python 2.4. So, in our Python 2.4 setup script, we need to
explicitly fetch and use virtualenv 1.7. Likewise for pip 1.1.

File imported from the already-fixed version for patsy:
  https://github.com/pydata/patsy/blob/0316d2901f4195db06e8091c15f37d9fe4ad09de/.travis-make-py24-virtualenv.sh
fcacdcc
@certik
Contributor
certik commented Sep 13, 2012

Thanks! I've removed the 56596a0 as well as your commit fixing it (which added the keyword arguments, but that can just stay in master). As far as Stefan's commit 5a471b5, I tried to remove it, but then his other commit "Retain backward compatibility. Enforce C order." does not apply cleanly. So I would have to manually fix the conflicts, so I think it's better to just keep both patches in, so that it's clear where they come from.

@certik
Contributor
certik commented Sep 13, 2012

P.S. I want to run Travis tests on this, as well as keep it here for other people to see, rather than just pushing it in.

@certik
Contributor
certik commented Sep 16, 2012

Here is the list of PRs from #396 contained in this "squash" (manual inspection): #395, #397, #401, #405. There are more which I will put in subsequent comments.

@certik
Contributor
certik commented Sep 16, 2012

Also Included: #417, #376, #404, #390, eebd7b2, #429, #431, #430

Missing: #432

@certik
Contributor
certik commented Sep 16, 2012

Ok, this looks good. One PR is missing, but I will simply note it at the TODO list and merge it in the next batch. All tests pass and as far as I can see, this looks good. So I am merging this.

@certik certik merged commit fcacdcc into numpy:maintenance/1.7.x Sep 16, 2012

1 check passed

default The Travis build passed
Details
@certik
Contributor
certik commented Sep 16, 2012

I just cherry-picked the #432 manually. So all is good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment