You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've encountered an issue where to_arrow() throws a ComputeError when I provide a dictionary to the schema parameter in a different order than the initial dataframe. I would expect that I can provide column names and types in any order in the dictionary because the dictionaries are not ordered by default (even though python dictionaries are). Here is an MRE:
import polars as pl
import numpy as np
N = 1_000
def create_data():
data = pl.DataFrame({"id":pl.Series(np.random.randint(N, size=(N,)))})
data = data.with_columns(
("1:" + pl.col('id').cast(pl.Utf8)).alias('idl'),
(N*10 + pl.col('id')*1.0).alias('idl_num')
)
return data
d = create_data()
pl.from_arrow(d.to_arrow(), schema={'id':pl.String, 'idl_num':pl.Int64, 'idl':pl.String})
I found a similar issue here: #11718 and the resolution is that the dictionary overwrites the data schema. However, this behavior is unexpected for most of the users who look into the documentation. Can we add a part that will specify this behavior? For example, change "If you supply a list of column names that does not match the names in the underlying data" to "If you supply a list (or a dictionary) of column names that does not match the names and the order of the columns in the underlying data"?
Description
I've encountered an issue where to_arrow() throws a ComputeError when I provide a dictionary to the schema parameter in a different order than the initial dataframe. I would expect that I can provide column names and types in any order in the dictionary because the dictionaries are not ordered by default (even though python dictionaries are). Here is an MRE:
I found a similar issue here: #11718 and the resolution is that the dictionary overwrites the data schema. However, this behavior is unexpected for most of the users who look into the documentation. Can we add a part that will specify this behavior? For example, change "If you supply a list of column names that does not match the names in the underlying data" to "If you supply a list (or a dictionary) of column names that does not match the names and the order of the columns in the underlying data"?
Link
https://docs.pola.rs/py-polars/html/reference/api/polars.from_arrow.html
The text was updated successfully, but these errors were encountered: