Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support reading from avro files [FEATURE-REQUEST] #1485

Open
stellaywu opened this issue Aug 3, 2021 · 3 comments
Open

support reading from avro files [FEATURE-REQUEST] #1485

stellaywu opened this issue Aug 3, 2021 · 3 comments

Comments

@stellaywu
Copy link

Description
Support reading from avro files natively from cloud

vaex.open("gs://path_of_many_avro_files", fs_options={'anon': True})

Is your feature request related to a problem? Please describe.
Currently the workaround is to read_in with pandas as pandas dataframe then convert to vaex dataframe which doesn't work when data is too big.

Thanks

@JovanVeljanoski
Copy link
Member

I don't know much about avro files or whether they are memory mappable. Maybe @maartenbreddels knows more.

@bdsaved
Copy link

bdsaved commented Feb 7, 2023

Has this feature been implemented?

@maartenbreddels
Copy link
Member

No, I never used avro files. The only workaround, for now, is to use a different library to read it, then feed it to vaex.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants