Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

BUG: Fix Ticket #2187 #401

Merged
merged 1 commit into from Aug 31, 2012

Conversation

Projects
None yet
3 participants
Contributor

jayvius commented Aug 31, 2012

Fix for ticket #2187.

When indexing with subset of fields, returning a copy is being deprecated in favor of returning a view. For now a copy is still returned, but it's a copy of the view that will be eventually returned. Fix returned copy so that copy of view with offsets copies only fields in view, not all the fields from original array. Also add unit tests to make sure this doesn't break when copy is fully deprecated in favor of returning a view.

Additional changes will need to be made when returning a copy is fully deprecated.

Fix returned copy
Fix returned copy so that copy of view with offsets copies only fields in view, not all the fields from original array. Also add unit tests to make sure this doesn't break when copy is fully deprecated in favor of returning a view.

This pull request passes (merged 93100c9 into 68320a1).

Contributor

certik commented Aug 31, 2012

Thanks a lot for this! Let me test it.

Contributor

certik commented Aug 31, 2012

After this PR, this is what I get:

In [1]: import statsmodels.api as sm


In [2]: 

In [2]: dta = sm.datasets.macrodata.load()

In [3]: dta.data[['infl', 'realgdp']].view((float,2))
Out[3]: 
array([[     0.   ,   2710.349],
       [     2.34 ,   2778.801],
       [     2.74 ,   2775.488],
       [     0.27 ,   2785.204],
       [     2.31 ,   2847.699],
       [     0.14 ,   2834.39 ],
       [     2.7  ,   2839.022],
       [     1.21 ,   2802.616],
       [    -0.4  ,   2819.264],
       [     1.47 ,   2872.005],
       [     0.8  ,   2918.419],
       [     0.8  ,   2977.83 ],
       [     2.26 ,   3031.241],
       [     0.13 ,   3064.709],
       [     2.11 ,   3093.047],
       [     0.79 ,   3100.563],
       [     0.53 ,   3141.087],
       [     2.75 ,   3180.447],
       [     0.78 ,   3240.332],
       [     2.46 ,   3264.967],
       [     0.13 ,   3338.246],
       [     0.9  ,   3376.587],
       [     1.29 ,   3422.469],
       [     2.05 ,   3431.957],
       [     1.28 ,   3516.251],
       [     2.54 ,   3563.96 ],
       [     0.89 ,   3636.285],
       [     2.9  ,   3724.014],
       [     4.99 ,   3815.423],
       [     2.1  ,   3828.124],
       [     4.9  ,   3853.301],
       [     0.61 ,   3884.52 ],
       [     2.42 ,   3918.74 ],
       [     3.61 ,   3919.556],
       [     3.58 ,   3950.826],
       [     4.72 ,   3980.97 ],
       [     3.5  ,   4063.013],
       [     5.77 ,   4131.998],
       [     4.56 ,   4160.267],
       [     4.51 ,   4178.293],
       [     6.67 ,   4244.1  ],
       [     5.47 ,   4256.46 ],
       [     5.4  ,   4283.378],
       [     6.38 ,   4263.261],
       [     6.28 ,   4256.573],
       [     4.13 ,   4264.289],
       [     5.11 ,   4302.259],
       [     5.04 ,   4256.637],
       [     2.   ,   4374.016],
       [     4.96 ,   4398.829],
       [     2.94 ,   4433.943],
       [     2.92 ,   4446.264],
       [     2.9  ,   4525.769],
       [     2.88 ,   4633.101],
       [     3.81 ,   4677.503],
       [     4.71 ,   4754.546],
       [     9.26 ,   4876.166],
       [     4.55 ,   4932.571],
       [    12.47 ,   4906.252],
       [    10.39 ,   4953.05 ],
       [    10.96 ,   4909.617],
       [     9.86 ,   4922.188],
       [    13.56 ,   4873.52 ],
       [    10.07 ,   4854.34 ],
       [     5.32 ,   4795.295],
       [     7.48 ,   4831.942],
       [     6.61 ,   4913.328],
       [     6.5  ,   4977.511],
       [     2.14 ,   5090.663],
       [     6.37 ,   5128.947],
       [     6.27 ,   5154.072],
       [     5.49 ,   5191.499],
       [     8.76 ,   5251.762],
       [     5.3  ,   5356.131],
       [     5.23 ,   5451.921],
       [     7.08 ,   5450.793],
       [     7.58 ,   5469.405],
       [     9.89 ,   5684.569],
       [     9.65 ,   5740.3  ],
       [     8.26 ,   5816.222],
       [    12.08 ,   5825.949],
       [    13.37 ,   5831.418],
       [    11.88 ,   5873.335],
       [    14.62 ,   5889.495],
       [    14.6  ,   5908.467],
       [     8.32 ,   5787.373],
       [    10.04 ,   5776.617],
       [    11.64 ,   5883.46 ],
       [     8.62 ,   6005.717],
       [    10.63 ,   5957.795],
       [     8.22 ,   6030.184],
       [     4.26 ,   5955.062],
       [     2.53 ,   5857.333],
       [    10.39 ,   5889.074],
       [     2.45 ,   5866.37 ],
       [    -0.82 ,   5871.001],
       [     3.66 ,   5944.02 ],
       [     4.03 ,   6077.619],
       [     3.99 ,   6197.468],
       [     5.13 ,   6325.574],
       [     4.67 ,   6448.264],
       [     3.09 ,   6559.594],
       [     3.82 ,   6623.343],
       [     2.28 ,   6677.264],
       [     4.89 ,   6740.275],
       [     2.61 ,   6797.344],
       [     2.96 ,   6903.523],
       [     5.13 ,   6955.918],
       [    -4.39 ,   7022.757],
       [     2.93 ,   7050.969],
       [     2.55 ,   7118.95 ],
       [     4.33 ,   7153.359],
       [     4.64 ,   7193.019],
       [     3.89 ,   7269.51 ],
       [     4.2  ,   7332.558],
       [     3.46 ,   7458.022],
       [     4.12 ,   7496.6  ],
       [     4.41 ,   7592.881],
       [     4.7  ,   7632.082],
       [     4.31 ,   7733.991],
       [     6.22 ,   7806.603],
       [     4.52 ,   7865.016],
       [     2.88 ,   7927.393],
       [     6.64 ,   7944.697],
       [     4.37 ,   8027.693],
       [     4.93 ,   8059.598],
       [     8.79 ,   8059.476],
       [     3.88 ,   7988.864],
       [     1.19 ,   7950.164],
       [     3.24 ,   8003.822],
       [     2.93 ,   8037.538],
       [     3.19 ,   8069.046],
       [     3.17 ,   8157.616],
       [     3.14 ,   8244.294],
       [     3.4  ,   8329.361],
       [     3.09 ,   8417.016],
       [     2.79 ,   8432.485],
       [     1.94 ,   8486.435],
       [     3.03 ,   8531.108],
       [     1.92 ,   8643.769],
       [     2.45 ,   8727.919],
       [     3.25 ,   8847.303],
       [     2.69 ,   8904.289],
       [     2.93 ,   9003.18 ],
       [     3.44 ,   9025.267],
       [     2.1  ,   9044.668],
       [     2.35 ,   9120.684],
       [     3.11 ,   9184.275],
       [     3.6  ,   9247.188],
       [     2.3  ,   9407.052],
       [     3.05 ,   9488.879],
       [     3.02 ,   9592.458],
       [     1.25 ,   9666.235],
       [     1.25 ,   9809.551],
       [     2.73 ,   9932.672],
       [     1.24 ,  10008.874],
       [     0.49 ,  10103.425],
       [     2.46 ,  10194.277],
       [     1.71 ,  10328.787],
       [     1.95 ,  10507.575],
       [     2.9  ,  10601.179],
       [     1.92 ,  10684.049],
       [     3.35 ,  10819.914],
       [     2.85 ,  11014.254],
       [     3.76 ,  11043.044],
       [     4.19 ,  11258.454],
       [     2.77 ,  11267.867],
       [     3.89 ,  11334.544],
       [     1.82 ,  11297.171],
       [     2.26 ,  11371.251],
       [     0.45 ,  11340.075],
       [     0.23 ,  11380.128],
       [     3.59 ,  11477.868],
       [     1.56 ,  11538.77 ],
       [     2.66 ,  11596.43 ],
       [     3.08 ,  11598.824],
       [     1.31 ,  11645.819],
       [     1.09 ,  11738.706],
       [     2.6  ,  11935.461],
       [     3.02 ,  12042.817],
       [     2.35 ,  12127.623],
       [     3.61 ,  12213.818],
       [     3.58 ,  12303.533],
       [     2.09 ,  12410.282],
       [     4.15 ,  12534.113],
       [     1.85 ,  12587.535],
       [     9.14 ,  12683.153],
       [     0.4  ,  12748.699],
       [     2.6  ,  12915.938],
       [     3.97 ,  12962.462],
       [    -1.58 ,  12965.916],
       [     3.3  ,  13060.679],
       [     4.58 ,  13099.901],
       [     2.75 ,  13203.977],
       [     3.45 ,  13321.109],
       [     6.38 ,  13391.249],
       [     2.82 ,  13366.865],
       [     8.53 ,  13415.266],
       [    -3.16 ,  13324.6  ],
       [    -8.79 ,  13141.92 ],
       [     0.94 ,  12925.41 ],
       [     3.37 ,  12901.504],
       [     3.56 ,  12990.341]])

(See the original ticket: http://projects.scipy.org/numpy/ticket/2187), so that's good.

Contributor

certik commented Aug 31, 2012

And before the PR, I get:


In [1]: import statsmodels.api as sm

In [2]: dta = sm.datasets.macrodata.load()

In [3]: dta.data[['infl', 'realgdp']].view((float,2))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-0229df340daa> in <module>()
----> 1 dta.data[['infl', 'realgdp']].view((float,2))

/home/ondrej/repos/numpy/py27/lib/python2.7/site-packages/numpy/core/records.pyc in view(self, dtype, type)
    494             dtype = sb.dtype(dtype)
    495             if dtype.fields is None:
--> 496                 return self.__array__().view(dtype)
    497             return ndarray.view(self, dtype)
    498         else:

ValueError: new type not compatible with array.

So I think that this fixes it!

@certik certik commented on the diff Aug 31, 2012

numpy/core/_internal.py
@@ -295,7 +295,11 @@ def _index_fields(ary, fields):
view_dtype = {'names':names, 'formats':formats, 'offsets':offsets, 'itemsize':dt.itemsize}
view = ary.view(dtype=view_dtype)
- return view.copy()
+ # Return a copy for now until behavior is fully deprecated
+ # in favor of returning view
+ copy_dtype = {'names':view_dtype['names'], 'formats':view_dtype['formats']}
+ from numpy import array
@certik

certik Aug 31, 2012

Contributor

This "array" should be imported from the line from multiarray import empty, dtype, array in the same function. I'll fix it in the next patch.

@certik

certik Aug 31, 2012

Contributor

See #404.

certik added a commit that referenced this pull request Aug 31, 2012

@certik certik merged commit c6d8734 into numpy:master Aug 31, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment