BUG: True cannot be cast to bool in read_excel #58994

rmhowe425 · 2024-06-13T02:17:20Z

closes BUG: True cannot be cast to bool #58159
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

rmhowe425 · 2024-06-13T03:40:19Z

@rhshadrach pinging on green

asishm · 2024-06-13T05:45:23Z

pandas/tests/io/excel/test_readers.py

@@ -164,6 +164,17 @@ def xfail_datetimes_with_pyxlsb(engine, request):


 class TestReaders:
+    def test_read_excel_type_check(self):
+        # GH 58159
+        df = DataFrame({"bool_column": [True]}, dtype=pd.BooleanDtype.name)


since boolean is a nullable EA, can you also add None as a value in this column? the assert can also probably change to tm.assert_frame_equal(df, df2)

rmhowe425 · 2024-06-15T23:42:31Z

@rhshadrach @asishm

PR ready for review. Not sure what's going on with the lone failing test. It's passing locally. Not much info from the test's output to go on.

asishm

Test failure is unrelated - #59018 (comment)

just one comment - may not even be relevant.

asishm · 2024-06-16T01:03:35Z

pandas/io/parsers/base_parser.py

+                        values_str,
                        dtype=cast_type,
                        true_values=self.true_values,
                        false_values=self.false_values,


none_values is not passed anything here. Can the equivalent na_values and keep_default_na be passed in here, or is it irrelevant?

I think I see what you're saying. You're suggesting to propagate na_values and keep_default_na from read_excel() to _cast_types()?

I think that's a good idea. I'll make that change. That should allow me to get rid of _NONE_VALUES in boolean.py, and allow me to dynamically handle any values that equate to None.

Those params are already available in _cast_types since it's an instance method (so self.na_values and self.keep_default_na are available here, similar to self.true_values and self.false_values). I was thinking about progagating them to _from_sequence_of_strings()

Yep we're on the same page. Making those changes now

… unit tests

rmhowe425 · 2024-06-17T02:22:49Z

@asishm @rhshadrach PR is ready for review

mroeschke · 2024-06-18T18:00:59Z

doc/source/whatsnew/v3.0.0.rst

@@ -547,6 +547,7 @@ I/O
 - Bug in :meth:`DataFrame.to_stata` when writing :class:`DataFrame` and ``byteorder=`big```. (:issue:`58969`)
 - Bug in :meth:`DataFrame.to_string` that raised ``StopIteration`` with nested DataFrames. (:issue:`16098`)
 - Bug in :meth:`read_csv` raising ``TypeError`` when ``index_col`` is specified and ``na_values`` is a dict containing the key ``None``. (:issue:`57547`)
+- Bug in :meth:`read_excel` raising ``ValueError`` when passing array of boolean values when ``dtype=pd.BooleanDtype.name``. (:issue:`58159`)


Suggested change

- Bug in :meth:`read_excel` raising ``ValueError`` when passing array of boolean values when ``dtype=pd.BooleanDtype.name``. (:issue:`58159`)

- Bug in :meth:`read_excel` raising ``ValueError`` when passing array of boolean values when ``dtype="boolean"``. (:issue:`58159`)

mroeschke · 2024-06-18T18:01:09Z

pandas/core/arrays/boolean.py

@@ -297,7 +297,13 @@ class BooleanArray(BaseMaskedArray):
    """

    _TRUE_VALUES = {"True", "TRUE", "true", "1", "1.0"}
-    _FALSE_VALUES = {"False", "FALSE", "false", "0", "0.0"}
+    _FALSE_VALUES = {


Could you undo the formatting here?

mroeschke · 2024-06-18T18:04:32Z

pandas/tests/io/excel/test_readers.py

+    def test_read_excel_type_check(self, col):
+        # GH 58159
+        df = DataFrame({"bool_column": col}, dtype=pd.BooleanDtype.name)
+        df.to_excel("test-type.xlsx", index=False)


Can you use the tmp_excel fixture for this path?

mroeschke · 2024-06-18T18:05:16Z

pandas/tests/io/excel/test_readers.py

+
+    def test_pass_none_type(self):
+        # GH 58159
+        with pd.ExcelFile("test7.xlsx") as excel:


Can you use the datapath fixture to access test7.xlsx. Also could you give a test7.xlsx a better name reflective of the contents?

mroeschke · 2024-06-18T18:06:49Z

pandas/tests/io/excel/test_readers.py

+    @pytest.mark.parametrize("col", [[True, None, False], [True], [True, False]])
+    def test_read_excel_type_check(self, col):
+        # GH 58159
+        df = DataFrame({"bool_column": col}, dtype=pd.BooleanDtype.name)


Suggested change

df = DataFrame({"bool_column": col}, dtype=pd.BooleanDtype.name)

df = DataFrame({"bool_column": col}, dtype="boolean")

And could you change other usages here?

rmhowe425 added 2 commits June 12, 2024 22:12

Adding implementation, tests. Updating documentation

1fcbde6

Fixing dependency error by moving test inside of TestReaders

f77c41d

asishm reviewed Jun 13, 2024

View reviewed changes

rmhowe425 added 4 commits June 15, 2024 14:56

Updating implementation based on reviewer feedback

ffb65d4

Creating a more clean implementation

14e565e

Fixing broken unit tests

873bc9b

Fixing docstring error

e918066

rmhowe425 requested a review from asishm June 15, 2024 23:41

asishm reviewed Jun 16, 2024

View reviewed changes

rmhowe425 and others added 2 commits June 16, 2024 21:06

Updating implementation based on reviewer feedback. Adding additional…

7de150e

… unit tests

Merge branch 'pandas-dev:main' into dev/bug/cast-to-boolean

e9fe60c

rmhowe425 requested a review from asishm June 17, 2024 02:22

mroeschke reviewed Jun 18, 2024

View reviewed changes

mroeschke added the IO Excel read_excel, to_excel label Jun 18, 2024

mroeschke reviewed Jun 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: True cannot be cast to bool in read_excel #58994

BUG: True cannot be cast to bool in read_excel #58994

rmhowe425 commented Jun 13, 2024 •

edited

rmhowe425 commented Jun 13, 2024

asishm Jun 13, 2024 •

edited

rmhowe425 commented Jun 15, 2024

asishm left a comment •

edited

asishm Jun 16, 2024

rmhowe425 Jun 16, 2024 •

edited

asishm Jun 16, 2024

rmhowe425 Jun 16, 2024

rmhowe425 commented Jun 17, 2024

mroeschke Jun 18, 2024

mroeschke Jun 18, 2024

mroeschke Jun 18, 2024

mroeschke Jun 18, 2024

mroeschke Jun 18, 2024

	- Bug in :meth:`read_excel` raising ``ValueError`` when passing array of boolean values when ``dtype=pd.BooleanDtype.name``. (:issue:`58159`)
	- Bug in :meth:`read_excel` raising ``ValueError`` when passing array of boolean values when ``dtype="boolean"``. (:issue:`58159`)

	df = DataFrame({"bool_column": col}, dtype=pd.BooleanDtype.name)
	df = DataFrame({"bool_column": col}, dtype="boolean")

BUG: True cannot be cast to bool in read_excel #58994

Are you sure you want to change the base?

BUG: True cannot be cast to bool in read_excel #58994

Conversation

rmhowe425 commented Jun 13, 2024 • edited

rmhowe425 commented Jun 13, 2024

asishm Jun 13, 2024 • edited

Choose a reason for hiding this comment

rmhowe425 commented Jun 15, 2024

asishm left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmhowe425 Jun 16, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmhowe425 commented Jun 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmhowe425 commented Jun 13, 2024 •

edited

asishm Jun 13, 2024 •

edited

asishm left a comment •

edited

rmhowe425 Jun 16, 2024 •

edited