Skip to content

Latest commit

 

History

History
19 lines (11 loc) · 1.13 KB

README.md

File metadata and controls

19 lines (11 loc) · 1.13 KB

spark_framework

This repository contains the spark_framework package. The package contains many useful functions that simplify the life for PySpark developer.

Package Goal

The purposes are:

  1. to make the PySpark code looking in more pythonic style
  2. to implement many things in more optimal way. E.g. calculation of medians, quantiles, etc., which works too slowly in default Spark implementation.

So, by using spark_framework you will make your code more compact, easy to read and more robust.

Development history

The package was implemented from scratch by the author for three times for different customers, until the last customer gave full permission to expose it to the open source world. The author is a data scientist in present and big data engineer in the past. While doing daily data science work, function after function was added to the package to be re-used in many PySpark notebooks.

Status of documentation

This is just initial commit to github, the author plans to extend the code of the demo notebook in the near future and to add documentation.