Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

fix&test push/pull recarrays #2034

Merged
merged 4 commits into from

3 participants

@minrk
Owner

We were only pushing the dtype str for some reason,
which is totally wrong for recarrays.

closes #2033

@wkerzendorf

man you're fast!! This only seems to happen on the pull. I can access things on the engines once they're pushed.

@minrk
Owner

This is less efficient that what we used to do for simple arrays, as sending the array is ~200B instead of ~3, but that's not very much for sending a real array, and it should be much more general.

@minrk
Owner

Test results for commit d8f2b33 merged into master
Platform: darwin

  • python2.6: Failed, log at https://gist.github.com/2986419 (libraries not available: cython matplotlib oct2py pygments pymongo qt rpy2 tornado wx wx.aui)
  • python2.7: OK (libraries not available: oct2py wx wx.aui)
  • python3.2: OK (libraries not available: cython matplotlib oct2py pymongo qt rpy2 wx wx.aui)

Not available for testing: python3.1

@minrk
Owner

Hm, unpickling dtypes doesn't seem to work in numpy 1.5. Need to work on it for a moment more.

@minrk
Owner

Okay, I think I got it for real this time. testing again...

@fperez
Owner

Code looks clean, running test_pr right now. Thanks for catching this one, @wkerzendorf, and as usual record-setting response time, @minrk :)

@wkerzendorf

No need to thank me, @minrk did all the work (and really fast), well if now I only have to get my own code working ;-)

@minrk minrk explicit dtype for str in recarray test
'str' would fail with zero-size in numpy < 1.6
b54e738
@fperez
Owner

Test results for commit fc2de03 merged into master
Platform: linux2

Not available for testing: python2.6, python3.1

@fperez
Owner

rerunning test_pr in case b54e738 helps with the failures reported above...

@minrk
Owner

slow down - there's one more commit after b54.

@fperez
Owner

ok, restarted run with bdf7984.

@minrk
Owner

Hm, while I'm pretty sure bdf should work, I might take one more stab at sending not the dtype object itself.

@minrk
Owner

Nevermind - I'll stick with the full dtype object since I'm not super confident in my understanding of dtype objects. I'll take a stab at cleaning it up when I do my general serialization improvement pass for 0.14.

@fperez
Owner

Sounds good. Let's be conservative here, and we can ping Travis O. or Mark Wiebe in Austin for a bit of wisdom on possible improvements we could make here while remaining 100% correct.

@minrk
Owner

Test results for commit bdf7984 merged into master
Platform: darwin

  • python2.6: OK (libraries not available: cython matplotlib oct2py pygments pymongo qt rpy2 tornado wx wx.aui)
  • python2.7: OK (libraries not available: oct2py wx wx.aui)
  • python3.2: OK (libraries not available: cython matplotlib oct2py pymongo qt rpy2 wx wx.aui)

Not available for testing: python3.1

@minrk
Owner

It's relatively minor - with this change, we are sending a few hundred bytes of metadata when ~99% of cases are covered with less than five.

I'm pretty sure the right answer is:

dtype = array.dtype.descr if array.dtype.shape else array.dtype.str

But I don't want to make a change like that immediately before release.

@fperez
Owner

Test results for commit bdf7984 merged into master
Platform: linux2

  • python2.7: OK
  • python3.2: OK (libraries not available: cython matplotlib oct2py pygments pymongo rpy2 wx wx.aui)

Not available for testing: python2.6, python3.1

@fperez
Owner

Conservative but safe, and code looks clean; thanks for the added tests! Merging now, glad we caught this before 0.13!

@fperez fperez merged commit feac402 into from
@fperez
Owner

Perhaps we should leave an issue open on this, or is it 'in cache' enough that you want to just open the 0.14 PR with that change?

@minrk
Owner

I'll add a note to #1504, which is for the more general serialization adjustments (and for which I already have a months-old branch in-progress)

@fperez
Owner

Perfect, thanks.

@minrk minrk deleted the branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jun 25, 2012
  1. @minrk
  2. @minrk

    include dtype itself in metadata, not str representation

    minrk authored
    which is totally wrong for recarrays
  3. @minrk

    explicit dtype for str in recarray test

    minrk authored
    'str' would fail with zero-size in numpy < 1.6
  4. @minrk
This page is out of date. Refresh to see the latest.
View
4 IPython/parallel/tests/test_newserialized.py
@@ -90,14 +90,14 @@ def test_ndarray_serialized(self):
self.assertEquals(td, 'ndarray')
md = ser1.getMetadata()
self.assertEquals(md['shape'], a.shape)
- self.assertEquals(md['dtype'], a.dtype.str)
+ self.assertEquals(md['dtype'], a.dtype)
buff = ser1.getData()
self.assertEquals(buff, buffer(a))
s = ns.Serialized(buff, td, md)
final = ns.unserialize(s)
self.assertEquals(buffer(a), buffer(final))
self.assertTrue((a==final).all())
- self.assertEquals(a.dtype.str, final.dtype.str)
+ self.assertEquals(a.dtype, final.dtype)
self.assertEquals(a.shape, final.shape)
# test non-copying:
a[2] = 1e9
View
24 IPython/parallel/tests/test_view.py
@@ -275,6 +275,30 @@ def test_apply_numpy(self):
C = view.apply_sync(lambda x:x, B)
assert_array_equal(B,C)
+ @skip_without('numpy')
+ def test_push_pull_recarray(self):
+ """push/pull recarrays"""
+ import numpy
+ from numpy.testing.utils import assert_array_equal
+
+ view = self.client[-1]
+
+ R = numpy.array([
+ (1, 'hi', 0.),
+ (2**30, 'there', 2.5),
+ (-99999, 'world', -12345.6789),
+ ], [('n', int), ('s', '|S10'), ('f', float)])
+
+ view['RR'] = R
+ R2 = view['RR']
+
+ r_dtype, r_shape = view.apply_sync(interactive(lambda : (RR.dtype, RR.shape)))
+ self.assertEquals(r_dtype, R.dtype)
+ self.assertEquals(r_shape, R.shape)
+ self.assertEquals(R2.dtype, R.dtype)
+ self.assertEquals(R2.shape, R.shape)
+ assert_array_equal(R2, R)
+
def test_map(self):
view = self.client[:]
def f(x):
View
2  IPython/utils/newserialized.py
@@ -110,7 +110,7 @@ def __init__(self, unSerialized):
self.obj = numpy.ascontiguousarray(self.obj, dtype=None)
self.typeDescriptor = 'ndarray'
self.metadata = {'shape':self.obj.shape,
- 'dtype':self.obj.dtype.str}
+ 'dtype':self.obj.dtype}
elif isinstance(self.obj, bytes):
self.typeDescriptor = 'bytes'
self.metadata = {}
Something went wrong with that request. Please try again.