Skip to content

Commit

Permalink
420 polars avro reader writer (#447)
Browse files Browse the repository at this point in the history
Implements polars avro reader and writer.

Enables someone to easily read from and save to avro with polars.

Squashed commits:

* Implement the reader and writer for avro

* Test the reader and writer for avro

* Add example

* Update the notebook in examples
  • Loading branch information
kokobhara committed Oct 9, 2023
1 parent 20c683f commit 08c216c
Show file tree
Hide file tree
Showing 4 changed files with 584 additions and 45 deletions.
8 changes: 8 additions & 0 deletions examples/polars/materialization/my_script.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,12 @@
file="./df.json",
combine=df_builder,
),
to.avro(
dependencies=output_columns,
id="df_to_avro",
file="./df.avro",
combine=df_builder,
),
]
# Visualize what is happening
dr.visualize_materialization(
Expand All @@ -77,10 +83,12 @@
"df_to_parquet_build_result",
"df_to_feather_build_result",
"df_to_json_build_result",
"df_to_avro_build_result",
], # because combine is used, we can get that result here.
inputs=initial_columns,
)
print(materialization_results)
print(additional_outputs["df_to_parquet_build_result"])
print(additional_outputs["df_to_feather_build_result"])
print(additional_outputs["df_to_json_build_result"])
print(additional_outputs["df_to_avro_build_result"])
Loading

0 comments on commit 08c216c

Please sign in to comment.