the portable Python dataframe library

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Implementing best practices for PySpark ETL jobs and applications.

🚚 Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

Jupyter magics and kernels for working with remote Spark clusters

pyspark🍒🥭 is delicious，just eat it!😋😋

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

pyspark methods to enhance developer productivity 📣 👯 🎉

PySpark test helper methods with beautiful error messages

Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

Pandas, Polars, and Spark DataFrame comparison for humans and more!

A boilerplate for writing PySpark Jobs

Process Common Crawl data with Python and Spark

t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark

PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster

Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines

Delta Lake helper methods in PySpark

A tool for building feature stores.

MorphL Community Edition uses big data and machine learning to predict user behaviors in digital products and services with the end goal of increasing KPIs (click-through rates, conversion rates, etc.) through personalization

LearningApacheSpark