read_csv clobbers values of columns with duplicate names

[stevenmanton]

xref #10577 (has test for duplicates with empty data)

I don't expect this is the correct behavior, although it's always possible I'm doing something wrong. Importing data using the names keyword will clobber the values of columns where the name is duplicated. For example:

from StringIO import StringIO
import pandas as pd

data = """a,1
b,2
c,3"""
names = ['field', 'field']

print pd.read_csv(StringIO(data), names=names, mangle_dupe_cols=True)
print pd.read_csv(StringIO(data), names=names, mangle_dupe_cols=False)

returns

   field  field
0      1      1
1      2      2
2      3      3
   field  field
0      1      1
1      2      2
2      3      3

However, this produces the correct result:

df = pd.read_csv(StringIO(data), header=None)
df.columns = names
print df

   field  field
0      a      1
1      b      2
2      c      3

Interestingly, it works if the field names are in the header:

data_with_header = "field,field\n" + data
print pd.read_csv(StringIO(data_with_header))

  field  field.1
0     a        1
1     b        2
2     c        3

Is this a bug or am I doing something wrong?

[yoavram]

I've came across something similar. When using mangle_dupe_cols=False I get duplicate column names but the column data is all the same, although it's not so in the data I read. So it seems that the last column of the same name overrides all other columns.

import pandas as pd
from StringIO import StringIO

data = """A,A,B,B,B
    1,2,3,4,5
    6,7,8,9,10
    11,12,13,14,15
    """

df1 = pd.read_table(StringIO(data), sep=',', mangle_dupe_cols=True)
df2 = pd.read_table(StringIO(data), sep=',', mangle_dupe_cols=False)

Now df1 is:

	A	A.1	B	B.1	B.2
0	1	2	3	4	5
1	6	7	8	9	10
2	11	12	13	14	15

which has the original data but non-duplicate column names;

and df2 is:

	A	A	B	B	B
0	2	2	5	5	5
1	7	7	10	10	10
2	12	12	15	15	15

which has duplicate column names but their respecrive data has been overriden.

Reproducible bug in IPython notebook: http://nbviewer.ipython.org/github/yoavram/ipython-notebooks/blob/master/pandas%20duplicate%20column%20bug.ipynb

Pandas version 0.16.0. Python 2.7.

[shoyer]

This seems very strange to me -- I don't think there's any good reason for this behavior. I'm going to label it as a bug.

[gfyoung]

xref #7160