Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix compatibility with Pandas 2 #63

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Conversation

QuLogic
Copy link

@QuLogic QuLogic commented Feb 24, 2024

The line_terminator parameter of read_csv was changed to lineterminator in Pandas 1.5, and the old name was dropped entirely in Pandas 2.

This adds compatibility shims across both, though if you prefer to bump minimum Pandas version to 1.5, then the new name could be used everywhere instead.

The `line_terminator` parameter of `read_csv` was changed to
`lineterminator` in Pandas 1.5, and the old name was dropped entirely in
Pandas 2.
@@ -69,7 +71,8 @@ def write(self, path):

with open(path, 'w') as f:
f.write(f'"sep={self.delimiter}"\n')
write_df.to_csv(f, index=False, sep=self.delimiter, line_terminator=f"{self.delimiter}\n")
write_df.to_csv(f, index=False, sep=self.delimiter,
**{_to_csv_line_terminator: f"{self.delimiter}\n"})

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. This raises a ValueError with Python 3.13.

________________________ ShimmerPlusTestCase.test_write ________________________
self = <test_shimmer.ShimmerPlusTestCase testMethod=test_write>
    def test_write(self):
>       self.reader.write(self.WRITE_PATH)
tests/test_shimmer.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../BUILDROOT/usr/lib/python3.13/site-packages/devicely/shimmer_plus.py:74: in write
    write_df.to_csv(f, index=False, sep=self.delimiter,
/usr/lib64/python3.13/site-packages/pandas/util/_decorators.py:333: in wrapper
    return func(*args, **kwargs)
/usr/lib64/python3.13/site-packages/pandas/core/generic.py:3964: in to_csv
    return DataFrameRenderer(formatter).to_csv(
/usr/lib64/python3.13/site-packages/pandas/io/formats/format.py:1014: in to_csv
    csv_formatter.save()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <pandas.io.formats.csvs.CSVFormatter object at 0x7f2269a70a50>
    def save(self) -> None:
        """
        Create the writer & save.
        """
        # apply compression and byte/text conversion
        with get_handle(
            self.filepath_or_buffer,
            self.mode,
            encoding=self.encoding,
            errors=self.errors,
            compression=self.compression,
            storage_options=self.storage_options,
        ) as handles:
            # Note: self.encoding is irrelevant here
>           self.writer = csvlib.writer(
                handles.handle,
                lineterminator=self.lineterminator,
                delimiter=self.sep,
                quoting=self.quoting,
                doublequote=self.doublequote,
                escapechar=self.escapechar,
                quotechar=self.quotechar,
            )
E           ValueError: bad delimiter or lineterminator value
/usr/lib64/python3.13/site-packages/pandas/io/formats/csvs.py:260: ValueError

So far, I haven't got a clue as to why?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems unrelated to this change. The existing call (slightly) abuses lineterminator to always insert a trailing separator. This must have been made more strict in Python 3.13, and I see a few changelog entries that could be related, most likely "gh-113796: Add more validation checks in the csv.Dialect constructor. ValueError is now raised if the same character is used in different roles."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants