Skip to content
A very tiny example of unit testing with PySpark
Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
adder
.gitignore
README.md

README.md

Unit Testing with PySpark

A minimal unit testing setup with PySpark fixtures.

For CDSW users, you must ensure that the PYSPARK_PYTHON environment variable in the CDSW points to the same version of python as the driver (env var PYSPARK_DRIVER_PYTHON.)

For instance, you may want to set a project-level environment variable to something like.

PYSPARK_PYTHON=/usr/local/bin/python3.6
You can’t perform that action at this time.