ERR: usecols fails to raise error if column doesn't exist but is the same length as headers

[GGordonGordon]

A small, complete example of the issue

read_csv doesn't raise a value error if the use cols contains the same number of values as the headers

simple test_csv.txt file contains:
A,B
1,2
3,4

It currently will set the other column as NaN values:
A B
0 1 NaN
1 3 NaN

import pandas as pd
file = pd.read_csv('data/test.csv',usecols=['A','C']) # Column C doesn't exist
file

Expected Output

ValueError("Usecols do not match names.") exception

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.5.2.final.0 python-bits: 64 OS: Darwin OS-release: 16.1.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8

pandas: 0.18.1
nose: 1.3.7
pip: 8.1.2
setuptools: 27.2.0
Cython: 0.24.1
numpy: 1.11.1
scipy: 0.18.1
statsmodels: 0.6.1
xarray: None
IPython: 5.1.0
sphinx: 1.4.6
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.6.1
blosc: None
bottleneck: 1.1.0
tables: 3.2.3.1
numexpr: 2.6.1
matplotlib: 1.5.3
openpyxl: 2.3.2
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.3
lxml: 3.6.4
bs4: 4.5.1
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.13
pymysql: None
psycopg2: None
jinja2: 2.8
boto: 2.42.0
pandas_datareader: 0.2.1

[jreback]

yeah this seems reasonable.

[aileronajay]

@GGordonGordon will you be creating a PR for this?

[GGordonGordon]

@aileronajay yeah fairly soon

[GGordonGordon]

Pull request: #14674

[GGordonGordon]

For clarification should if usecols is a list of integers should it refer to the columns within the original data table (appears this is what the current documentation states) or should they refer as an index to the names list (appears to be how the code currently operates)?

[jreback]

@GGordonGordon the documentation is correctly. you would have to show an example where this is not true.

[GGordonGordon]

This appears to be resolved by #14234 and this is no longer needed.

[jorisvandenbossche]

Using master (so with #14234 merged), I still see the buggy behaviour:

In [12]: s = """A,B
    ...: 1,2
    ...: 3,4"""

In [13]: pd.read_csv(StringIO(s))
Out[13]: 
   A  B
0  1  2
1  3  4

In [15]: pd.read_csv(StringIO(s), usecols=["A", "C"])
Out[15]: 
   A    B
0  1  NaN
1  3  NaN

So it would be welcome if you could update the PR.

[bpraggastis]

Is anyone working on this now. Just picked it up at #pandas sprint at #PyCon2017 ?

[jorisvandenbossche]

@bpraggastis Note there is a closed PR #14674 that started this work, but never was finished. You can have a look at that, and see whether it would be useful to start from that work (use the changes in that PR)

[bpraggastis]

The approach originally taken (#14674) caused the test suite to fail. I used a different approach focusing only on the case when usecols_dtype == 'string'. In this case it was sufficient to check that usecols is a subset of names. I commited to branch gh-14671.

There is another case that is not covered that I would like clarification on. If header=0 and names are provided to replace the column headers in the data, then usecols should be able to reference individual columns by index or by the name assigned in names. I verified this works in practice:

When I wrote tests for this the tests failed due to a conditional in parsers.py: _infer_columns(self): on code lines:

if ((self.usecols is not None and
                     len(names) != len(self.usecols)) or
                    (self.usecols is None and
                     len(names) != len(columns[0]))):
                    raise ValueError('Number of passed names did not match '
                                     'number of header fields in the file')

The implication from this code is that if usecols is provided then its length must match len(names).
I am not sure if my BUG fix is adequate as the additional tests I wrote won't pass.

[jorisvandenbossche]

cc @gfyoung

[gfyoung]

@jorisvandenbossche : I too didn't realize this issue wasn't handled in full. Thanks for the CC!

@bpraggastis : Mind opening up a PR for this? Would make it a little easier to view changes and see whether they are merge-able.

Also, FYI, whenever you write "gh-xxxx" in the comment box, GitHub creates a link to the corresponding issue or PR in the repository. If you meant to point to your branch, then you will need to do:

<a href="URL-to-your-branch">gh-xxxx</a>

[bpraggastis]

@jorisvandenbossche @gfyoung : I created a PR and left the tests referencing the problem I mentioned commented out for future use (not sure if that is the correct protocol). Leaving PyCon now but would like to continue working on this if you open another issue addressing the new bug.

[TomAugspurger]

I mentioned commented out for future use (not sure if that is the correct protocol).

we typically mark it with an xfail for expected failure, so that it gets reported

@pytest.mark.xfail('this will fail')
def test_fail():
    assert False

[gfyoung]

we typically mark it with an xfail for expected failure, so that it gets reported

Right, but let's see if we can patch them first before marking. What you just found @bpraggastis is a bug with the Python engine (what you were demonstrating was with the C engine). So let's patch that first before addressing your PR.

[gfyoung]

Actually, this problem is bigger than I thought. Filing issue for this.