Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] Built-in support for loading avro files #125

Closed
kvnkho opened this issue Dec 8, 2020 · 1 comment · Fixed by #179 or #194
Closed

[FEATURE] Built-in support for loading avro files #125

kvnkho opened this issue Dec 8, 2020 · 1 comment · Fixed by #179 or #194
Assignees
@gityow
Labels
core feature enhancement New feature or request
Milestone
0.5.4

Comments

@kvnkho
Copy link
Collaborator

kvnkho commented Dec 8, 2020 •
edited
Loading

Spark has support for reading in avro files.

Pandas will be a bit trickier. We'll need to add fastavro as a dependency. Sample script for how to read avro into pandas:
https://gist.github.com/LouisAmon/300b4a906a6d25a7fb5d2c4d174d242e

Place to add functionality.
https://github.com/fugue-project/fugue/blob/master/fugue/_utils/io.py
@kvnkho kvnkho self-assigned this Dec 8, 2020
@goodwanghan goodwanghan added core feature enhancement New feature or request labels Dec 17, 2020
@kvnkho kvnkho added this to the 0.5.0 milestone Jan 11, 2021
@goodwanghan goodwanghan modified the milestones: 0.5.0, 0.5.1 Jan 15, 2021
@kvnkho kvnkho assigned gityow and unassigned kvnkho Jan 20, 2021
@gityow
Copy link
Collaborator

gityow commented Jan 20, 2021

I'll take this! :)

@goodwanghan goodwanghan modified the milestones: 0.5.1, 0.5.2 Jan 26, 2021
@goodwanghan goodwanghan removed this from the 0.5.2 milestone Mar 23, 2021
@goodwanghan goodwanghan added this to the 0.5.4 milestone Apr 28, 2021
@goodwanghan goodwanghan linked a pull request Apr 28, 2021 that will close this issue
Add avro io #179
Merged
@goodwanghan goodwanghan reopened this Apr 29, 2021
@goodwanghan goodwanghan linked a pull request Apr 29, 2021 that will close this issue
revert dask head, enable avro tests #194
Merged
rdmolony added a commit to rdmolony/fugue-tutorials that referenced this issue Oct 25, 2021
@rdmolony
Add avro file support
f471db1 
See fugue-project/fugue#125
rdmolony added a commit to fugue-project/tutorials that referenced this issue Oct 26, 2021
@rdmolony
Proof fugue sql syntax (#96)
5ac99c4 
* Replace other with somewhere

* Reword input & output

* Fix typo in supported file formats

* Add avro file support

See fugue-project/fugue#125

* Fix execution engine typo

* Revert output no module called fugue_notebook
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gityow gityow

Labels
core feature enhancement New feature or request
Projects
None yet
Milestone
0.5.4
Development

Successfully merging a pull request may close this issue.

Add avro io gityow/fugue
revert dask head, enable avro tests fugue-project/fugue
3 participants
@gityow @goodwanghan @kvnkho