This repository contains example code to be used with an article I wrote. Here you can find the article: https://medium.com/@dominikschauer/professional-aws-glue-pyspark-development-local-development-and-unit-tests-190be8ed75fb
This repository helps you to setup a local development environment for AWS Glue PySpark jobs. It works for both Windows and Ubuntu/Mac OS.
It contains:
- a template for an AWS Glue PySpark job
- a template for a corresponding pytest unit test
For more details, please read the article. It explains step by step what to do to start developing on your local machine (i.e. without AWS Glue charges!).
- start tests from project root
python3 -m pytest ./tests/
- The code is mainly copied from the official AWS Glue documentation: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html#develop-local-python