airflow1.10.7_spark3.1.2

• Changed Spark version to 3.1.2
• Removed unnecessary Docker folders (the images are being downloaded within docker-compose file)

airflow1.10.7_spark3.0.1

• Added Pyspark app example with Python modules.
• Added a DAG to run the Pyspark app example with Python modules.

v0.0.4

Changes:

• Added an example of how to work with Spark and Postgres data
• Added a DAG to run the example os Spark + Postgres
• Changed the Spark container to run as root container. Reference: https://docs.bitnami.com/tutorials/work-with-non-root-containers/

v0.0.3

Improvements:

• Changed all docker containers to work with the same Spark version (version 3.0.1)
• Airflow and Jupyter docker build now accept arguments to build Spark, Hadoop or Airflow in a different version if needed. More details in readme:
  • https://github.com/cordon-thiago/airflow-spark#build-airflow-docker
  • https://github.com/cordon-thiago/airflow-spark#build-jupyter-docker
• Readme updated

v0.0.2

Fixes:

• Forced pyspark version to be compatible with Spark 2.4.5.
• Added Spark Contex to hello world notebook sample.
• Added flag --use-feature=2020-resolver to the pip command to avoid compatibility errors. Reference: https://discuss.python.org/t/announcement-pip-20-2-release/4863.
• Readme minor fixes.