This is the example PySpark test set up for my Medium article written here.
- Install Python 3.8.
- Create a virtual environment via
python3 -m venv venv
and activate itsource venv/bin/activate
- Install poetry
pip3 install poetry
- Install all dependencies
poetry install
- Run tests
pytest tests --dist loadfile -n 2