AWS Glue Streaming Libraries are additions and enhancements to Spark Structured Streaming for ETL operations.
This repository contains:
awsglue- the Python libary you can use to author AWS Glue Streaming ETL job. This library extends Apache Spark with additional data types and operations for Streaming ETL workflows. It's an interface for Glue Streaming ETL library in Python.
bin- this directory hosts several executables that allow you to run the Python library locally or open up a PySpark shell to run Glue Spark code interactively.
If you haven't already, please refer to the official AWS Glue Python local development documentation for the official setup documentation. The following is a summary of the AWS documentation:
awsglue library provides only the Python interface to the Glue Spark runtime, you need the Glue Streaming ETL jar to run it locally. The jar is now available via the maven build system in a s3 backed maven repository. Here are the steps to set up your dev environment locally.
- install Apache Maven from the following location: https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-common/apache-maven-3.6.0-bin.tar.gz
copy-dependenciestarget in Apache Maven to download the jar from S3 to your local dev environment.
- download and extract the Apache Spark distribution based on the Glue version you're using:
- Glue version 0.9:
- Glue version 1.0:
- Glue version 2.0:
- Glue version 3.0:
- Glue version 0.9:
- export the
SPARK_HOMEenvironmental variable to the extracted location of the above Spark distribution. For example:
Glue version 0.9: export SPARK_HOME=/home/$USER/spark-2.2.1-bin-hadoop2.7 Glue version 1.0: export SPARK_HOME=/home/$USER/spark-2.4.3-bin-hadoop2.8 Glue version 2.0: export SPARK_HOME=/home/$USER/spark-2.4.3-bin-hadoop2.8 Glue version 3.0: export SPARK_HOME=/home/$USER/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3
- now you can run the executables in the
bindirectory to start a Glue Shell or submit a Glue Spark application.
Glue shell: ./bin/gluepyspark Glue submit: ./bin/gluesparksubmit pytest: ./bin/gluepytest
gluepytest script assumes that the pytest module is installed and available in the
PATH env variable)
Copyright 2022 Amazon
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.