Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: Setting index_col=False when calling read_csv disables on_bad_lines #49279

Open
2 of 3 tasks
benjsec opened this issue Oct 24, 2022 · 2 comments
Open
2 of 3 tasks

BUG: Setting index_col=False when calling read_csv disables on_bad_lines #49279

benjsec opened this issue Oct 24, 2022 · 2 comments
Assignees
@pbhoopala
Labels
Bug IO CSV read_csv, to_csv Needs Triage Issue that has not been reviewed by a pandas team member

Comments

@benjsec
Copy link

benjsec commented Oct 24, 2022

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> pd.read_csv(io.StringIO("a,b\n1,2\n1,2,3"), engine="python", on_bad_lines=lambda x: print(f"Error: {x}"))
Error: ['1', '2', '3']
   a  b
0  1  2
>>> pd.read_csv(io.StringIO("a,b\n1,2\n1,2,3"), engine="python", on_bad_lines=lambda x: print(f"Error: {x}"), index_col=False)
<stdin>:1: ParserWarning: Length of header or names does not match length of data. This leads to a loss of data with index_col=False.
   a  b
0  1  2
1  1  2

Issue Description

If index_col=False is specified when reading a csv, then the on_bad_lines parameter is ignored.

Expected Behavior

on_bad_lines parameter should be respected even when index_col=False is set

Installed Versions

pd.show_versions()

INSTALLED VERSIONS

commit : 91111fd
python : 3.10.6.final.0
python-bits : 64
OS : Linux
OS-release : 5.19.16-76051916-generic
Version : #202210150742166605324422.04~cf07008 SMP PREEMPT_DYNAMIC Tue O
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 1.5.1
numpy : 1.23.4
pytz : 2022.5
dateutil : 2.8.2
setuptools : 56.0.0
pip : 21.0.1
Cython : None
pytest : 7.1.3
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None
@benjsec benjsec added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Oct 24, 2022
@benjsec benjsec changed the title BUG: Settingindex_col=False when calling read_csv disables on_bad_lines BUG: Setting index_col=False when calling read_csv disables on_bad_lines Oct 24, 2022
@pbhoopala
Copy link

take

@douglatornell douglatornell mentioned this issue Feb 10, 2023
Open
douglatornell added a commit to SalishSeaCast/SalishSeaNowcast that referenced this issue Feb 10, 2023
@douglatornell
Supress superfluous pandas.ParserWarning
ceb060d 
Warning is related to pandas-dev/pandas#49279
though my testing shows that the warning is raised in spite of the lines with
extra fields being processed as specified.
@andrewjprice
Copy link

Any updates on this issue?

douglatornell added a commit to SalishSeaCast/SalishSeaNowcast that referenced this issue May 18, 2023
@douglatornell
Change rivers runoff file generation for SalishSeaCast NEMO v202111 (#…
5d2b230 
…150)

* Copy daily_river_flows.py module in from tools repo

    Source was `tools/I_ForcingFiles/Rivers/DailyRiverFlows.py` where Susan
    developed the module by extracting functions from her development notebook for
    generating runoff forcing files for SalishSeaCast NEMO v202111 runs.
    I plan to develop and refactor the module further here on the way to refactoring
    the `make_runoff_file` worker for production runs of v202111.

* Copy ProductionDailyRiverNCfile notebook from tools repo

    Source was `tools/I_ForcingFiles/Rivers/ProductionDailyRiverNCfile.ipynb`:
    Susan's development notebook for generating runoff forcing files for
    SalishSeaCast NEMO v202111 runs.
    I plan to develop and refactor the notebbok further here on the way to
    refactoring the `make_runoff_file` worker for production runs of v202111.

* Minimum chgs to run ProductionDailyRiverNCfile.ipynb

    In notebook:
    * import `make_runoff_files` from `nowcast.daily_river_flows`
    * change config file path to `../config/nowcast.yaml`
    * change date to calculate runoff file for to 6-Feb-2023 to avoid Missing
       Snohomish River obs on 23-Jan-2023
    * generalize date handling in calculation of runoff forcing file names for
       comparison figure

    In `daily_river_flows` module:
    * Handle extra fields in some limes of .csv files that trigger
       `pandas .ParserError` by adding `_parse_long_csv_line()` and `on_bad_lines`
       args to `pandas.read_csv()` calls
    * Handle path in grid repo clone for coordinates file in dev envs that don't
       have access to `/SalishSeaCast/`

* Update make_readme.py

    Make it consistent with template in
    https://ubc-moad-docs.readthedocs.io/en/latest/github_notebooks_readme.html

* Add title & description to ProductionDailyRiverNCfile.ipynb

* Fix corrupted ERDDAP_datasets.ipynb

    Bad linebreak characters and missing commas, probably from manual edits.

* Fix Test-research_VENUS-Module.ipynb

    * remove spaces from file name so that it is properly processed by
       make_readme.py
    * correct layout of title and description cell at top of notebook
    * correct Markdown style re: space after heading markup

* Add ProductionDailyRiverNCfile.ipynb to README

* Add notebook subtitles & execution notes

* Get river proportion dict module name from config

* Drop unused imports

* Refactor river flow .csv file reading

    Use functools.partial() to reduce repeated code.
    Add unit tests for .csv reading.

* Suppress superfluous pandas.ParserWarning

    Warning is related to pandas-dev/pandas#49279
    though my testing shows that the warning is raised in spite of the lines with
    extra fields being processed as specified.

* Refactor setting date as dataframe index

    Extract repeated code into a function and use inplace=True.

* Add unit tests for daily_river_flows._read_river()

    Also add some type hints in docstrings.

* Refactor daily_river_flows._read_river_Theodosia()

    Added unit tests and type hints in docstring.

* Refactor daily_river_flows._patch_fitting()

    Added unit tests and type hints in docstring.

* Refactor daily_river_flows._patch_missing_obs()

    Renamed function from patch_gaps().
    Added unit tests and type hints in docstring.

* Drop `bad` from daily_river_flows._patch_fitting()

    Unnecessary because invalid flux value is detected by checking
    numpy.isnan(flux).

* Refactor daily_river_flows._do_a_pair()

    Added unit tests and type hints in docstring.

* Refactor daily_river_flows._do_fraser()

    Added unit tests and type hints in docstring.

* Refactor daily_river_flows._calc_watershed_flows()

    Added unit tests and type hints in docstring.

* Refactor daily_river_flows._get_area()

* Refactor daily_river_flows._create_runoff_array()

    Added unit test and type hints in docstring.

* Skip TestCreateRunoffArray test when in GHA workflow

    _get_area() uses file from grid repo that is too annoying to make available
    in Actions workspace.

* Update pkgs & versions to match recent dev env

* Chg to point sarrcenia clients at dd.weather.gc.ca

    This was done on 25-Apr-2023 to avoid disruption due to hpfx server maintenance.
    However, missing file messages and duplicated files issues in HRDPS downloads
    that started with the change to the continental rotated lat-lon grid in Feb-2023
    disappeared with the change to dd.weather.gc.ca.

    (cherry picked from commit ceddac3)

* Create dataset in _calc_runoff_dataset()

* Refactor netCDF file writing

* Update to certifi==2023.5.7 in dev env

* Hard-code runoff file name template

    To avoid disrupting automation for v201905 production.

* Hard-code runoff proportions dict module name

    To avoid disrupting automation for v201905 production.
@simonjayhawkins simonjayhawkins added the IO CSV read_csv, to_csv label Feb 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pbhoopala pbhoopala

Labels
Bug IO CSV read_csv, to_csv Needs Triage Issue that has not been reviewed by a pandas team member
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@benjsec @simonjayhawkins @andrewjprice @pbhoopala