BUG: Inconsistent replacement of values

[darck-neos]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Problem description

Value at a given position don't change using "values" indexing when the column in cuestion has object data type. However, the value change using "at".

I known that values return numpy array, then I test the same behaviour on numpy and everything is ok.

Normal Behaviour:

import pandas as pd

#Create DataFrame
columns = ["A", "B"]

data = [[0, 1], [2, 3]]

dataset = pd.DataFrame(data=data, columns=columns)

print(dataset)
print("\n", dataset.dtypes)
print("\n", dataset.values.flags)
print("="*25)

#Modify value
dataset.values[0, 1] = "4"

print(dataset)
print("\n", dataset.dtypes)
print("\n", dataset.values.flags)
print("="*25)

#Modify value
dataset.values[0, 1] = 4

print(dataset)
print("\n", dataset.dtypes)
print("\n", dataset.values.flags)

Output:

   A  B
0  0  1
1  2  3

 A    int64
B    int64
dtype: object

   C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

=========================
   A  B
0  0  4
1  2  3

 A    int64
B    int64
dtype: object

   C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

=========================
   A  B
0  0  4
1  2  3

 A    int64
B    int64
dtype: object

   C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

Inconsistent Behaviour:

import pandas as pd

#Create DataFrame
columns = ["A", "B"]

data = [[0, 1], [2, "3"]]

dataset = pd.DataFrame(data=data, columns=columns)

print(dataset)
print("\n", dataset.dtypes)
print("\n", dataset.values.flags)
print("="*25)

#Modify value
dataset.values[0, 1] = "4"

print(dataset)
print("\n", dataset.dtypes)
print("\n", dataset.values.flags)
print("="*25)

#Modify value
dataset.values[0, 1] = 4

print(dataset)
print("\n", dataset.dtypes)
print("\n", dataset.values.flags)
print("="*25)

#Modify value
dataset.at[0, "B"] = "4"

print(dataset)
print("\n", dataset.dtypes)
print("\n", dataset.values.flags)

Output:

   A  B
0  0  1
1  2  3

 A     int64
B    object
dtype: object

   C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

=========================
   A  B
0  0  1
1  2  3

 A     int64
B    object
dtype: object

   C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

=========================
   A  B
0  0  1
1  2  3

 A     int64
B    object
dtype: object

   C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

=========================
   A  B
0  0  4
1  2  3

 A     int64
B    object
dtype: object

   C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

Normal Behaviour on Numpy:

import numpy as np

data = [[0, 1], [2, "3"]]

#Create array
dataset = np.array(data)

print(dataset)
print("\n", dataset.dtype)
print("\n", dataset.flags)
print("="*25)

#Modify value
dataset[0, 1] = 4

print(dataset)
print("\n", dataset.dtype)
print("\n", dataset.flags)
print("="*25)

#Create array
dataset = np.array(data, dtype=object)

print(dataset)
print("\n", dataset.dtype)
print("\n", dataset.flags)
print("="*25)

#Modify value
dataset[0, 1] = 4

print(dataset)
print("\n", dataset.dtype)
print("\n", dataset.flags)

Output:

[['0' '1']
 ['2' '3']]

 <U21

   C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

=========================
[['0' '4']
 ['2' '3']]

 <U21

   C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

=========================
[[0 1]
 [2 '3']]

 object

   C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

=========================
[[0 4]
 [2 '3']]

 object

   C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

Output of pd.__version__

'1.0.3'

Output of pd.show_versions()

/usr/local/lib/python3.6/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
/usr/local/lib/python3.6/dist-packages/pandas_datareader/compat/__init__.py:7: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  from pandas.util.testing import assert_frame_equal

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.9.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.19.104+
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.0.3
numpy            : 1.18.2
pytz             : 2018.9
dateutil         : 2.8.1
pip              : 19.3.1
setuptools       : 46.0.0
Cython           : 0.29.16
pytest           : 3.6.4
hypothesis       : None
sphinx           : 1.8.5
blosc            : None
feather          : 0.4.0
xlsxwriter       : None
lxml.etree       : 4.2.6
html5lib         : 1.0.1
pymysql          : None
psycopg2         : 2.7.6.1 (dt dec pq3 ext lo64)
jinja2           : 2.11.1
IPython          : 5.5.0
pandas_datareader: 0.8.1
bs4              : 4.6.3
bottleneck       : 1.3.2
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.2.6
matplotlib       : 3.2.1
numexpr          : 2.7.1
odfpy            : None
openpyxl         : 2.5.9
pandas_gbq       : 0.11.0
pyarrow          : 0.14.1
pytables         : None
pytest           : 3.6.4
pyxlsb           : None
s3fs             : 0.4.2
scipy            : 1.4.1
sqlalchemy       : 1.3.15
tables           : 3.4.4
tabulate         : 0.8.7
xarray           : 0.15.1
xlrd             : 1.1.0
xlwt             : 1.3.0
xlsxwriter       : None
numba            : 0.47.0

[jbrockmendel]

@dark-neos can you cut this down to the minimal material i need to copy/paste to see the issue? e.g. are the flags really relevant?

[darck-neos]

The flags are to show the state of numpy arrays from DataFrame and directly from numpy, for example that WRITEABLE is True.

The minimal material to replicate would be:

import pandas as pd

#Create DataFrame
columns = ["A", "B"]

data = [[0, 1], [2, "3"]]

dataset = pd.DataFrame(data=data, columns=columns)

print(dataset)
print("="*25)

#Modify value
dataset.values[0, 1] = 4 #No matter if it is 4 or "4"

print(dataset)

Output:

   A  B
0  0  1
1  2  3
=========================
   A  B
0  0  1
1  2  3

[jbrockmendel]

Thank you, that clarifies the issue.

In general setting on frame.values is not equivalent to setting directly on frame.iloc. In particular, when we have mixed dtypes, frame.values is not a view on the underlying arrays.

[darck-neos]

In that case, the recommendation would be use iloc instead values and avoid it unless you want to display some values or don't use values for anything?

[jbrockmendel]

In that case, the recommendation would be use iloc instead values and avoid it unless you want to display some values or don't use values for anything?

use .iloc for setting, yes. .values really should only be used if you need to pass a numpy array to something outside of pandas

[jbrockmendel]

Closing since this isnt a bug, but happy to discuss this further.