Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support pyarrow RecordBatch and Table in main load_table function #204

Open
randyzwitch opened this issue Apr 9, 2019 · 1 comment
Open
Labels

Comments

@randyzwitch
Copy link
Contributor

randyzwitch commented Apr 9, 2019

Currently, load_table doesn't support passing a pyarrow RecordBatch. RecordBatch can be converted to a pandas dataframe using .to_pandas(), so it seems trivial to support this as well.

Related #185 : In theory, this is supposed to work with load_table_arrow, but it doesn't work with load_table(method='arrow')

@randyzwitch
Copy link
Contributor Author

data = <pyarrow.lib.RecordBatch object at 0x7f6bfa17f098>, preserve_index = False

    def build_row_desc(data, preserve_index=False):
    
        if not isinstance(data, pd.DataFrame):
            # Once https://issues.apache.org/jira/browse/ARROW-1576 is complete
            # we can support pa.Table here too
            raise TypeError("Create table is not supported for type {}. "
                            "Use a pandas DataFrame, or perform the create "
>                           "separately".format(type(data)))
E           TypeError: Create table is not supported for type <class 'pyarrow.lib.RecordBatch'>. Use a pandas DataFrame, or perform the create separately

pymapd/_pandas_loaders.py:197: TypeError

@randyzwitch randyzwitch changed the title Support pyarrow RecordBatch for upload Support pyarrow RecordBatch and Table for upload Apr 10, 2019
@randyzwitch randyzwitch changed the title Support pyarrow RecordBatch and Table for upload Support pyarrow RecordBatch and Table in main load_table function Apr 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant