Reindex broken

[boldt]

MWE

from __future__ import print_function

import pandas as pd
import numpy as np

print("Panda version:", pd.__version__)
print("+++++++++++++++++++++++++++++++++++")
print(pd.show_versions())
print("+++++++++++++++++++++++++++++++++++")

####################################################
# Config
####################################################

pd.options.display.max_rows = 10
pd.options.display.float_format = '{:.1f}'.format

####################################################
# Read data
####################################################

file = "/tmp/california_housing_train.csv"
if(np.DataSource().exists(file)):
	dataset = file
else:
	dataset = "https://download.mlcc.google.com/mledu-datasets/california_housing_train.csv"

sep=","
california_housing_dataframe = pd.read_csv(dataset, sep)

####################################################
# Reorder
####################################################

newOrder = np.random.permutation(california_housing_dataframe.index)
california_housing_dataframe_reordered = california_housing_dataframe.reindex(newOrder)

####################################################
# Merge and show diff of the heads
####################################################

# Let's take the heads of both datasetstand compare them
# They should be different in (mostly) all elements 

head1 = california_housing_dataframe.head(10)
head2 = california_housing_dataframe_reordered.head(10)

# @see https://stackoverflow.com/a/36893675/605890
merged = head1.merge(head2, indicator=True, how='outer')
print(merged)

Run on colab

I created a colab for the MWE, which is based on pandas 0.22.0:

https://colab.research.google.com/drive/19uDE_H4AtpLaEL6INrRrDMXkdANsNr69#scrollTo=CzxuGppV26Rt

If you run it, you see at the output (if non is doubled randomly):

• 10x left_only
• 10x right_only

Run with docker containers

Now, run the same MWE (located under /tmp/tf/Bug.py) in a two different docker containers, which uses pandas 0.23.4,:

Both return:

• 10x both

This means, both heads are the same, which means that reindex does not have any effect.

Python docker container (python 3.6.6)

docker run --rm -it -v /tmp/tf/:/tmp/ python:3.6.6 /bin/bash -c "pip install pandas && python /tmp/Bug.py"

tensorflow docker container (tensorsflow 1.11.0)

docker run --rm -it -v /tmp/tf/:/tmp/ tensorflow/tensorflow:1.11.0-py3 python /tmp/Bug.py 

TLDR

The following code does not have any effect in pandas 0.23.4:

california_housing_dataframe_reordered = california_housing_dataframe.reindex(newOrder)

[jorisvandenbossche]

Can you indicate the version for which this is not working for you? (or for which it is working, and for which not) Use, pd.show_versions() (as was mentioned in the template)

I cannot reproduce it locally with pandas master

[jorisvandenbossche]

Can you also include the small code snippet to reproduce in the top post here? That is much easier for us, than to click to the other issue / to colab.

[boldt]

I hope, my updated bug report helps?

[jorisvandenbossche]

Can you add the result of pd.show_versions() ?

[jreback]

likely this is the result if inplace modification by permutation. I think we have a bug about this already

[boldt]

I reproduced the same issue in the default python 3.6.6 docker container.

[boldt]

I missed pd.show_versions(). Here they are:

Colab

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.6.final.0
python-bits: 64
OS: Linux
OS-release: 4.14.65+
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.22.0
pytest: None
pip: 18.1
setuptools: 39.1.0
Cython: None
numpy: 1.14.6
scipy: 0.19.1
pyarrow: None
xarray: None
IPython: 5.5.0
sphinx: None
patsy: 0.5.0
dateutil: 2.5.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.1.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.3
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: 0.4.1
pandas_datareader: None
None

Tensorflow docker container

INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-139-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.23.4
pytest: None
pip: 18.0
setuptools: 39.1.0
Cython: None
numpy: 1.15.2
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.5.0
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 3.0.0
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 1.0.1
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
None

Python docker container

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.6.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-139-generic
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: C.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: None
pip: 18.1
setuptools: 40.4.3
Cython: None
numpy: 1.15.2
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
None

Local installation

TODO (Bad wifi atm)

[boldt]

@jreback Because of this, all examples on the Machine Learning Crash Course are broken (if not executed on colab)

[boldt]

@jreback Probably this issue? The comment from @ceciliassis seems to be exactly my issue. She is having the issue in Anaconda3 (I observed the issue there are well)

[jorisvandenbossche]

@jreback np.random.permutation should not do any inplace modification. Do you remember numpy bugs about that?

Regarding #17132, that comment of @ceciliassis is indeed about the same, but the issue itself is completely unrelated (that's why I asked to open a new issue)

@boldt so I cannot reproduce this (also on linux, python 3.5, pandas 0.23.0, numpy 1.14.3 / or pandas 0.23.4 + numpy 1.13).
Could you try:

• if you change the numpy permutation with something else that generates the random numbers, do you still see it?
• try to make a small dummy example (with / without using numpy permutation) creating a small dataframe, and see if you can reproduce it?

[boldt]

@jorisvandenbossche Thanks for your reponse.

Have you tried is with the docker containers? With them, it should be easy to reproduce.
I was able to reproduce it with the docker containers on an Ubuntu 14.04 and on a WIndows 10.

Can you recommend another permutation package to replace the numpy permutation?

[jorisvandenbossche]

Sorry, I currently cannot try with docker containers.

Can you try the following small example:

In [80]: df = pd.DataFrame({'a': range(5), 'b': np.arange(0, 0.5, 0.1)})

In [81]: df
Out[81]: 
   a    b
0  0  0.0
1  1  0.1
2  2  0.2
3  3  0.3
4  4  0.4

In [82]: new_order = np.random.permutation(df.index)

In [83]: df.reindex(new_order)
Out[83]: 
   a    b
3  3  0.3
0  0  0.0
4  4  0.4
1  1  0.1
2  2  0.2