Skip to content
Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
src
README.md
requirements.txt first commit Jun 14, 2018
setup.py first commit Jun 14, 2018

README.md

pyspark-unit-test

This project showcases how we can use pytest to create unit test cases for pyspark application

The spark application analyses flight data to create an ordered list of the flights with maximum times late arrivals.

Pytest fixtures are used to create spark sessions and supplied them automatically to the pipeline unit cases. Some of the important files are:

  1. conftest.py : Defines the spark pytest fixture with module scope
  2. flight_analyzer_tests.py : creates dataframes used for the unit test cases

unit test can be run as: pytest flight_analyzer_tests.py

You can’t perform that action at this time.