csv.Sniffer().has_header() Returns False for a CSV with a Header

[mouchen626]

Bug report

Bug description:

I encountered an issue while using csv.Sniffer().has_header() in Python's built-in csv module. Despite providing a CSV file with a clear header, the function returns False, whereas the expected result is True.

import csv

with open('1.csv', mode='r', encoding='utf-8') as file:
    sample = file.read(1024)
    has_header = csv.Sniffer().has_header(sample)  
    print(f"Has header: {has_header}")

1.csv

filename,path
c.txt,/tmp/c.txt
python.txt,/tmp/python.txt

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

[donBarbos]

According to the documentation, has_header method is rough heuristic and may produce both false positives and negatives.

Also two criterias are listed and neither of them seem to fit your case:

• the second through n-th rows contain numeric values
• the second through n-th rows contain strings where at least one value’s length differs from that of the putative header of that column.

[AffanShaikhsurab]

Hi I would like to work on this issue , can someone please assign me on this .and any things I should know before I start to solve this issue as I am kinda new here . Thanks

[gaogaotiantian]

In this case, I don't see a way to detect that the file has a header. The has_header detection is always heuristic, it's impossible to know for sure. Also the module is kind of inactive - I had a PR #104385 from years ago and there's no one that can review it. I don't suggest anyone working on this specific case, mostly because it does not seem solvable.

[tomasr8]

This is what's written in the has_header method:

cpython/Lib/csv.py

Lines 452 to 460 in 2bef8ea

	def has_header(self, sample):
	# Creates a dictionary of types of data in each column. If any
	# column is of a single type (say, integers), *except* for the first
	# row, then the first row is presumed to be labels. If the type
	# can't be determined, it is assumed to be a string in which case
	# the length of the string is the determining factor: if all of the
	# rows except for the first are the same length, it's a header.
	# Finally, a 'vote' is taken at the end for each column, adding or
	# subtracting from the likelihood of the first row being a header.

In the case of string values, it wants the strings in each column to have the same lengths which is not the case with the CSV file you gave (not saying this cannot be improved, just wanted to explain why the heuristic fails in this case)

[donBarbos]

I don't think it's a bug because this behavior is documented. It's look like improvement, part of issue #104380

[ericvsmith]

I think practically the current algorithm can't be changed. If we improve it for this case, we'll probably make it fail for an existing case where someone is using it, and we don't want to break existing code.

But if someone wants to have a try at it, we can look at the changes.

[rhettinger]

I concur with Eric. It is hard to change the sniffer heuristics because code that currently works likely relies on the current behavior. In the past, we've closed reports like this as won't fix.

AFAICT, the only reasonable ways forward are 1) live with the reasonably good heuristics we have now, 2) add an option for user customizations that are local and don't affect other users, or 3) add a new-and-improved variant while leaving the old one in-place.

[ericvsmith]

My desire to over-engineer things leads me to Raymond's option 2: a pluggable sniffer mechanism. The existing algorithm could be one such pluggable sniffer. We could either ship others in addition, or the user could supply them, or they could be 3rd party libraries on PyPI. I'd argue against shipping more than the current one, but maybe one could be provided as a recipe in the docs.

But it's a lot of work, and I don't think anyone is up for the task.