BUG: Fillna doesn't work on large dataframes

[PranayAnchuri]

• [X ] I have checked that this issue has not already been reported.

• [X ] I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

I'm using the fillna function as follows

ax = data.xtrain.fillna({"A": 123}, inplace=False)

Here data.xtrain is pandas dataframe

ipdb> type(data.xtrain)                                                                                                                    
<class 'pandas.core.frame.DataFrame'>
ipdb> data.xtrain.shape                                                                                                                    
(1317, 2728)

I noticed that the fillna method doesn't on my dataframe. It works okay on the samples provided in documentation. By digging into the codebase, I was able to isolate the error to the following line

pandas/pandas/core/generic.py

Line 6424 in bbffcb9

obj = result[k]

Here result[k] seems to return a new copy of the column k

ipdb> obj is result[k]                                                                                                                     
False

fillna wouldn't work if the above attribute access returns a copy of the column.

I don't see a similar effect when the dataframe is a small 4*5 matrix.

Code Sample, a copy-pastable example

Unfortunately, the dataframe contains sensitive information which I can't share.

Problem description

fillna method uses attribute access when passed a dict of na values. In some cases, attribute access returns a new copy of the column which leads to the method not working as expected.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : d9fff27
python : 3.7.6.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-58-generic
Version : #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.1.0
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 20.0.2
setuptools : 45.1.0.post20200127
Cython : None
pytest : 5.4.3
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.1
IPython : 7.16.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : 0.7.4
fastparquet : None
gcsfs : None
matplotlib : 3.3.0
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 1.0.0
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[paste the output of pd.show_versions() here leaving a blank line after the details tag]

[jreback]

pls show a copy paste able example that reproduces

show pandas versions

[PranayAnchuri]

@jreback Here is a reproducible example :

import pandas as pd
import numpy as np

print(f"Pandas version {pd.__version__}")
print(f"Numpy version {np.__version__}")

df = pd.DataFrame({"a": [1, 2, np.nan], "b": [1, 3, np.nan]})

print("df")
print(df)

df2 = df[["a", "a"]]

print("df2")
print(df2)

df2.fillna({"a": 5}, inplace=True)

print("df2 after fillna on a")
print(df2)

df.fillna({"b": 6}, inplace=True)

print(df)

The result on my machine is

Pandas version 1.1.0
Numpy version 1.19.1
df
     a    b
0  1.0  1.0
1  2.0  3.0
2  NaN  NaN
df2
     a    a
0  1.0  1.0
1  2.0  2.0
2  NaN  NaN
df2 after fillna on a
     a    a
0  1.0  1.0
1  2.0  2.0
2  NaN  NaN
     a    b
0  1.0  1.0
1  2.0  3.0
2  NaN  6.0

I realized that the error occurs when the dataframe has duplicate column names. In the above example, the first fillna doesn't replace the values neither in df2 nor the original df even though inplace is set to True. I also didn't get SettingWithCopyWarning. In the current implementation of fillna, the line

pandas/pandas/core/generic.py

Line 6424 in bbffcb9

obj = result[k]

assumes that result[k] always returns a reference to column k. I think this implementation fails if this invariant is not satisfied.