BUG: read_csv not respecting object dtype when option is set #56047
Conversation
|
Can we get this one in? |
|
|
||
| df = DataFrame(col_dict, columns=columns, index=index) | ||
| if hasattr(self, "orig_options"): | ||
| dtype_arg = self.orig_options.get("dtype", None) |
There was a problem hiding this comment.
Is the dtype option normally applied in _engine.read? Just curious why it needs to be done here
There was a problem hiding this comment.
Yes, but the DataFrame constructor infers object to string again if the option is set, which would discard the original dtype
There was a problem hiding this comment.
OK makes sense.
Could we defer looping over col_dict if dtype isn't specified to be object-like?
There was a problem hiding this comment.
Updated, only doing this now if we have a dict or object dtype
| dtype_mapping[pa.null()] = pd.Int64Dtype() | ||
| frame = table.to_pandas(types_mapper=dtype_mapping.get) | ||
| elif using_pyarrow_string_dtype(): | ||
|
|
There was a problem hiding this comment.
These mappers don't work, arrow supports type -> type not column -> type
|
cc @mroeschke gentle ping |
|
Thanks @phofl |
| new_rows = len(index) | ||
|
|
||
| df = DataFrame(col_dict, columns=columns, index=index) | ||
| if hasattr(self, "orig_options"): |
There was a problem hiding this comment.
can we do something more explicit than a hasattr check?
There was a problem hiding this comment.
no, you can subclass the reader, so we don't have any control over it
There was a problem hiding this comment.
does anybody actually do this? i judge those people, their ethics, and their hygiene.
There was a problem hiding this comment.
That's something I can't answer, we might want to deprecate maybe, but we are stuck with hasattr here until then
3 participants
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.we are not honouring object dtype here, thoughts on performance @jbrockmendel ?