BUG: SparseDataFrame from scipy.sparse.dok_matrix returns incorrect dataframe in python 2.7 #16179

Closed
keitakurita opened this Issue Apr 30, 2017 · 3 comments

Comments

Projects
None yet
3 participants
@keitakurita
Contributor

keitakurita commented Apr 30, 2017

Code Sample, a copy-pastable example if possible

Python 2.7

>>> import sys
>>> sys.version
'2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:05:08) \n[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]'
>>> import numpy as np
>>> import scipy.sparse
>>> a = np.arange(1, 5).reshape(2,2)
>>> a
array([[1, 2],
       [3, 4]])
>>> spm = scipy.sparse.dok_matrix(a)
>>> spm
<2x2 sparse matrix of type '<type 'numpy.int64'>'
	with 4 stored elements in Dictionary Of Keys format>
>>> pd.SparseDataFrame(a)
   0  1
0  1  2
1  3  4
>>> pd.SparseDataFrame(spm)
   0  1
0  3  2
1  1  4

Python 3.6

>>> import sys
>>> sys.version
'3.6.1 |Continuum Analytics, Inc.| (default, Mar 22 2017, 19:25:17) \n[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]'
>>> a = np.arange(1, 5).reshape(2,2)
>>> spm = scipy.sparse.dok_matrix(a)
>>> pd.SparseDataFrame(spm)
   0  1
0  1  2
1  3  4

Problem description

Initialization of SparseDataFrame with scipy.sparse.dok_matrix returns a different result from direct initialization from np.ndarray only in Python 2.7. This is inconsistent and unestimated behavior.
The scipy.sparse.dia_matrix shows similar buggy behavior.

Expected Output

>>> spm = scipy.sparse.dok_matrix(np.arange(1, 5).reshape(2,2))
>>> pd.SparseDataFrame(spm)
   0  1
0  1  2
1  3  4

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: 075eca1 python: 2.7.13.final.0 python-bits: 64 OS: Darwin OS-release: 16.5.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: ja_JP.UTF-8 LOCALE: None.None

pandas: 0.20.0rc1+29.g075eca1
pytest: 3.0.7
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.18.1
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
pandas_gbq: None
pandas_datareader: None

@TomAugspurger

This comment has been minimized.

Show comment
Hide comment
@TomAugspurger

TomAugspurger Apr 30, 2017

Contributor

Do you have the same pandas and scipy versions on both?

Contributor

TomAugspurger commented Apr 30, 2017

Do you have the same pandas and scipy versions on both?

@keitakurita

This comment has been minimized.

Show comment
Hide comment
@keitakurita

keitakurita Apr 30, 2017

Contributor

Yes, I have scipy version 0.18.1 installed on both environments (Python 3.6 and Python 2.7)

Contributor

keitakurita commented Apr 30, 2017

Yes, I have scipy version 0.18.1 installed on both environments (Python 3.6 and Python 2.7)

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback May 1, 2017

Contributor

cc @kernc

Contributor

jreback commented May 1, 2017

cc @kernc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment