npyio.py --> recfromcsv seems to have a bug that ignores any dtypes provided #311

Closed
kobejohn opened this Issue Jun 16, 2012 · 5 comments

Projects

None yet

4 participants

@kobejohn

I found this due to someone's problem loading large integers with recfromcsv. After playing with it, I think it is simply ignoring all the dtype information provided.

Line 1859 in npyio.py, part of recfromcsv() tries to either bring in the dtypes passed or set None. However, it actually brings in a kwarg that I couldn't find anywhere else called 'update' which looks suspiciously like a copied mistype from earlier in the line. If I change 'update' --> 'dtype', then the types get picked up as expected. More explicitly, in recfromcsv, change:

kwargs.update(dtype=kwargs.get('update', None),

to this:

kwargs.update(dtype=kwargs.get('dtype', None),

I found this in the github repo (although I may have been looking at the wrong branch) as well as the 1.6.2 release that I am using.

_(apologies for double posting this. I posted in trac first and then heard that maybe that won't be looked at. I haven't found a way to delete or close that ticket.)_

@jarondl jarondl pushed a commit to jarondl/numpy that referenced this issue Sep 29, 2013
Yaron de Leeuw MAINT (API?): organize npyio.recfromcsv defaults
Organizes the default kwargs in recfromcsv. Changes two undocumented
kwargs behaviors:

*  previously, if a user set `names=None`, it was ignored and replaced
   with `names=True`
*  the `dtype` kwarg was ignored. If `update` was given, it was used as
   `dtype`, and if not, None was used. We can retain the `update` behavior
 by using `kwargs.setdefault("dtype",kwargs.get('update', None))`.
  This Closes #311 .
fffb8df
@hpaulj
hpaulj commented Nov 25, 2013

I came across this issue via a Stackoverflow question:
http://stackoverflow.com/questions/19944408/getting-indices-in-numpy/20172164#20172164

The questioner was loading a file with text fields in Python 3, and having problems testing fields. By default recfromcsv giving these a bytes dtype. Trying to load the file with unicode dtype did not work.

@kobejohn

did changing the line to use the (I think) correct kwarg of 'dtype' instead
of 'update' work?

On Mon, Nov 25, 2013 at 10:14 AM, Paul Jacobson notifications@github.comwrote:

I came across this issue via a Stackoverflow question:

http://stackoverflow.com/questions/19944408/getting-indices-in-numpy/20172164#20172164

The questioner was loading a file with text fields in Python 3, and having
problems testing fields. By default recfromcsv giving these a bytesdtype. Trying to load the file with
unicode dtype did not work.


Reply to this email directly or view it on GitHubhttps://github.com/numpy/numpy/issues/311#issuecomment-29172012
.

@hpaulj
hpaulj commented Nov 25, 2013

kwargs.update(dtype=kwargs.get('dtype', None), solves the problem, as does kwargs.setdefault("dtype", None) (jarondl's patch).

@jarondl
jarondl commented Nov 26, 2013

I've added a regression test to #3830 for this bug.

@charris
Member
charris commented Feb 17, 2014

@juliantaylor This might fit into your bytes/str work.

@jarondl jarondl pushed a commit to jarondl/numpy that referenced this issue Feb 28, 2014
Yaron de Leeuw MAINT (API?): organize npyio.recfromcsv defaults
Organizes the default kwargs in recfromcsv. Changes two undocumented
kwargs behaviors:

*  previously, if a user set `names=None`, it was ignored and replaced
   with `names=True`
*  the `dtype` kwarg was ignored. If `update` was given, it was used as
   `dtype`, and if not, None was used. We can retain the `update` behavior
 by using `kwargs.setdefault("dtype",kwargs.get('update', None))`.
  This Closes #311 .
803a75f
@charris charris closed this in #3830 Mar 29, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment