the portable Python dataframe library

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Lightweight and extensible compatibility layer between dataframe libraries!

Type annotations for pyarrow

Work with bioinformatic files using Arrow, Polars, and/or DuckDB

Dremio Arrow Flight Client

A web application for viewing Apache Parquet files . This is a Python + Flask application

Exploring Chicago crimes dataset with Jupyter notebooks, DuckDB, Malloy and new Panel/PyScript data and dashboard tools.

Command-line interface to quickly generate fake CSV and JSON data

Converts a whole subdirectory with a big (or small) volume of PDF documents to a dataset (pandas DataFrame) with error tracking and choice of features

Reading both XLSX and XLSB files, fast and memory-safe, with Python, into PyArrow

highspeed timeseries pandas dataframe database

Code examples / snippets for website news post

Poor mans simple python api for creating a local or remote datalake based on several (pyarrow) datasets using duckdb

db2ixf is a python package with a CLI that simplifies the parsing and processing of IBM Integration eXchange Format (IXF) files.

Saving large files on GitHub

manylinux2014 Python pkg builds

En este repositorio se va a compartir todo el material relacionado con la charla "Como compartir grandes Datasets entre procesos sin perder la salud mental" de la Pycones 2021

provides a convenient and efficient solution for capturing and analyzing system activity logs using Procmon and converting them to the pandas compatible Parquet file format (2% of the original pml file size)