A web application for viewing Apache Parquet files . This is a Python + Flask application

Convert data to the parquet format with Python dask and pyarrow.

Concise interface to cache numpy arrays and pandas dataframes

ibm_db extension to load a pyarrow table to db2

Converts a whole subdirectory with a big (or small) volume of PDF documents to a dataset (pandas DataFrame) with error tracking and choice of features

A small cast tookit class drived from _ParquetDatasetV2 to support cast in filters argument

Dockerfile and Python 3.9 wheel for PyArrow 3.0.0 built on Alpine 3.14 (does not include Plasma or Parquet)

Complete Guide to Data Munging

Code examples / snippets for website news post

CSV converter

Saving large files on GitHub

(PoC) A very memory-efficient way to read data from PostgreSQL

Define a big data architecture and perform distributed machine learning calculations on an EMR cluster using AWS

En este repositorio se va a compartir todo el material relacionado con la charla "Como compartir grandes Datasets entre procesos sin perder la salud mental" de la Pycones 2021

An example showing how to send compressed RecordBatches over HTTP with PyArrow.

Exploring Chicago crimes dataset with Jupyter notebooks, DuckDB, Malloy and new Panel/PyScript data and dashboard tools.

A simple toolkit to transform datasource generate by img2dataset from parquet file to Huggingface dataset.

provides a convenient and efficient solution for capturing and analyzing system activity logs using Procmon and converting them to the pandas compatible Parquet file format (2% of the original pml file size)

Poor mans simple python api for creating a local or remote datalake based on several (pyarrow) datasets using duckdb

Colección de scripts en Python con PyArrow y Pandas para facilitar el manejo eficiente de archivos Parquet. Incluye herramientas para visualizar esquemas, convertir a CSV, verificar duplicados y fusionar archivos Parquet.