BUG: read_csv: on_bad_lines= callable does not call handler function

[Black-Maille]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

dataRaw = None

def badLineHandler(bad_line: list[str]) -> list[str]:
    print("Test")
    cleanedLine = bad_line[:len(dataRaw.columns)]
    print(bad_line[len(dataRaw.columns):])
    return cleanedLine

with open('handlerTestCase.csv') as inFile:
    dataRaw = pd.read_csv(inFile, sep=';', decimal=",", index_col= False, engine= 'python', on_bad_lines= badLineHandler)



Contents of file handlerTestCase.csv:


column1;column2;column3;column4
11;30;0
12;59;0;1;3
13;10;0;1;4

Issue Description

Contrary to documentation, read_csv does not call a function when 'on_bad_lines' is set to a callable function and an overlong line is encountered.
(I provided a sample file instead of a complete DataFrame since the purpose is to generate a DataFrame.)

Function badLineHandler has the appropriate signature (I checked, but omitted the checking code for brevity).
The print command at the beginning signals whether the function is called, and the slicing of the bad_line parameter should ensure the returned list has the appropriate length.

Instead, I get the following warning:

Warning (from warnings module):
File "H:<Path>\handlerTestReport.py", line 12
dataRaw = pd.read_csv(inFile, sep=';', decimal=",", index_col= False, engine= 'python', on_bad_lines= badLineHandler)
ParserWarning: Length of header or names does not match length of data. This leads to a loss of data with index_col=False.

Expected Behavior

I should at least get a "Test" output on the command line for each bad line, also the excess items.
read_csv should continue without warning since the modified rows have the appropriate number of entries.

Installed Versions

INSTALLED VERSIONS

commit : f538741
python : 3.11.3.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19045
machine : AMD64
processor : Intel64 Family 6 Model 140 Stepping 1, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : de_DE.cp1252

pandas : 2.2.0
numpy : 1.26.2
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 65.5.0
pip : 23.3.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.8.2
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : None
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.11.4
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report. The documentation states

Callable, function with signature as described in pyarrow documentation when engine='pyarrow'

Confirmed that the function is called when engine='pyarrow' is specified.

[Black-Maille]

Thank for the reply.
I'll have to check, but if this holds (and I assume it will), then I misunderstood the documentation. There it states:

New in version 1.4.0:

Callable, function with signature (bad_line: list[str]) -> list[str] | None that will process a single bad line. bad_line is a list of strings split by the sep. If the function returns None, the bad line will be ignored. If the function returns a new list of strings with more elements than expected, a ParserWarning will be emitted while dropping extra elements. Only supported when engine='python'

I was assuming the restriction for version 1.4 was also being carried forward to 2.x, I didn't expect it to be for just 1.4.

Thank you for the clarification.

[Black-Maille]

I'll close this issue.
I got the function to be called, and it kinda works, but the row number is always None, and after that I get some weird error message that I have no clue where to even start looking for the source.

I've already wasted more than a day on a nice-to-have feature with this functionality, and it's time to move on.

Thanks for the help.