QST: What is the canonical way to convert a column of type `string[pyarrow]` to boolean within a pandas dataframe?

[jamesowers-roo]

Research

• I have searched the [pandas] tag on StackOverflow for similar questions.

• I have asked my usage related question on StackOverflow.

Link to question on StackOverflow

https://stackoverflow.com/questions/76429921/qst-what-is-the-canonical-way-to-convert-a-column-of-type-stringpyarrow-to-bo

Question about pandas

Note, this is a carbon copy of this question asked on stackoverflow - I've only duplicated it here for easier reference and access. Please feel free to delete as you see fit.

I'm wanting to convert string data which is indeed boolean (or null) e.g. values are y / n / NA, or true / false / NA, or even a mix of these.

When using pandas with numpy backend as default, conversion from string data to boolean works smoothly:

import pandas as pd

df = pd.DataFrame({"col1": ["true", None, "false"]})
assert df.dtypes["col1"] == "object", df.dtypes["col1"]
# convert to boolean
df["col1"] = df["col1"].replace({'true': True, 'false': False}).astype(bool)
assert df.dtypes["col1"] == bool, df.dtypes["col1"]

However, when using the pyarrow backend (in my use case, I was actually using pd.read_parquet with dtype_backend - but I set the type explicitly in the example below):

df_pyarrow = pd.DataFrame(
    {"col1": ["true", None, "false"]}, dtype="string[pyarrow]"
)
assert df_pyarrow.dtypes["col1"] == "string", df_pyarrow.dtypes["col1"]
df_pyarrow["col1"] = (
    df_pyarrow["col1"]
    .replace({'true': True, 'false': False})  # fails at this step
    .astype(bool)
)
assert df_pyarrow.dtypes["col1"] == "bool[pyarrow]", df_pyarrow.dtypes["col1"]

but this fails at the .replace() because pyarrow complains, rightly!, that True and False are not valid values for a string[pyarrow]: TypeError: Scalar must be NA or str.

I have found that this method works:

df_pyarrow["col1"] = df_pyarrow["col1"] == "true"

assert df_pyarrow.dtypes["col1"] == "bool[pyarrow]", df_pyarrow.dtypes["col1"]

However:

• df_pyarrow.info() still says col1 is string[pyarrow]
• this method isn't as flexible: what if there were multiple values for True/False

What is the canonical way to convert a column of type string[pyarrow] to boolean within a pandas dataframe?

[rhshadrach]

However:

• df_pyarrow.info() still says col1 is string[pyarrow]

I get bool[pyarrow] on main:

df_pyarrow = pd.DataFrame(
    {"col1": ["true", None, "false"]}, dtype="string[pyarrow]"
)
df_pyarrow["col1"] = df_pyarrow["col1"] == "true"
df_pyarrow.info()
# <class 'pandas.core.frame.DataFrame'>
# RangeIndex: 3 entries, 0 to 2
# Data columns (total 1 columns):
#  #   Column  Non-Null Count  Dtype        
# ---  ------  --------------  -----        
#  0   col1    2 non-null      bool[pyarrow]
# dtypes: bool[pyarrow](1)
# memory usage: 130.0 bytes

What version of pandas are you using?

As answered on SO now, map works well here. With this, is there anything remaining?

[jamesowers-roo]

Interesting! Thank you. I was using the latest version at the time of asking.