v0.2.0

AWS Glue Toolkit v0.2.0

AWS Glue Toolkit v0.2.0 expands gtk from packaging-only into a local Glue development loop: run jobs and tests against the official Glue Docker images, with richer dependency forms and container-faithful pip.

What’s new

Local job execution — gtk run

spark-submit on the entry script in the Glue image for glue_version. Passes --JOB_NAME from project.name unless overridden. Extra --key value tokens after [JOB-DIR] go to getResolvedOptions. Shuts down the Spark driver so the container returns.

Local testing — gtk test

python3 -m pytest in the Glue image. PYTHONPATH includes source (and the install target when deps are present). Default target is the optional tests field (default: tests):

[tool.aws-glue-toolkit]
tests = "tests"

Path and git dependencies

project.dependencies now accepts file: (in-job or monorepo) and git/VCS requirements alongside PyPI. Path deps must be installable packages (pyproject.toml or setup.py); editable installs (-e) remain rejected.

dependencies = [
  "pandas>=2",
  "my-lib @ file:./libs/my-lib",
  "shared @ file:../packages/shared",
  "tool @ git+https://github.com/org/tool.git@v1",
]

Host pip configuration in containers

For container pip (check, build, and dep install on run / test), gtk snapshots the host’s effective pip settings (files + PIP_*) into a temporary pip.conf, mounts it, and sets PIP_CONFIG_FILE. Host-local keys (cache-dir, cert, target, and similar) are omitted.

Ephemeral dependency install on run / test

When project.dependencies is non-empty, packages install into a temporary PYTHONPATH target inside the container (Glue pins as constraints) before the job or tests run.

What’s improved

Docker-backed gtk check and gtk build

Dependency resolve and gluewheels pip wheel now run inside the Glue image—same Python and platform as the worker—not on the host. Artifact names and Glue parameters (.dependencies.zip / .gluewheels.zip) are unchanged from v0.1.0.

Runtime metadata for Glue 5.0 and 5.1 now includes the official image references (public.ecr.aws/glue/aws-glue-libs:5 and :5.1).

Exit codes

Code	Meaning
69	Docker not available or could not be started

For run / test, a successful Docker launch returns the container command’s exit code (or pip’s if install fails). Existing codes (65, 66, 70, 78) are unchanged; 78 also covers invalid dependencies.

Breaking changes from v0.1.0

• Docker is required for all gtk commands, including check and build (previously host-side pip).
• Dependency resolve and gluewheels pip execute in the Glue container, not on the host.

v0.1.0

AWS Glue Toolkit v0.1.0

We are pleased to announce the first public release of AWS Glue Toolkit, a command-line utility that streamlines the AWS Glue development lifecycle. The toolkit lets teams validate and package Glue jobs locally—before deployment—using dependency pins that mirror the target Glue runtime.

This release introduces the gtk command and establishes the project's core workflow: describe a job in pyproject.toml, verify its dependencies with gtk check, and produce deployment-ready artifacts with gtk build.

Overview

Each Glue job is defined as a directory containing a pyproject.toml, with toolkit settings under the [tool.aws-glue-toolkit] table. For example,

[tool.aws-glue-toolkit]
glue_version = "5.1"
source = "src"
script = "__main__.py"

Features

Dependency validation — gtk check

Resolves the job's declared dependencies against the bundled runtime pins for the configured Glue version, using the Glue worker's Python version and platform tag. Surfaces unsatisfiable dependencies early, without writing any files.

Artifact packaging — gtk build

Produces two deployment archives in the job directory:

Artifact	Glue parameter	Contents
{name}-{version}.dependencies.zip	--extra-py-files	Project .py sources, excluding the entry script
{name}-{version}.gluewheels.zip	--additional-python-modules	Wheels and requirements.txt, omitting packages already present on the Glue image

Bundled runtime metadata

Ships with curated dependency pins for Glue 5.0 and Glue 5.1, so dependency resolution reflects the actual managed runtime.

Predictable exit codes

Failures map to standard BSD sysexits.h codes for reliable scripting and CI integration:

Code	Meaning
65	Dependencies unsatisfiable against the Glue runtime pins
66	pyproject.toml missing or unreadable
70	Build pipeline failure
78	Invalid configuration, job layout, or unsupported Glue version

Installation

Requires Python 3.11 or later.

pip install aws-glue-toolkit

Compatibility and support

• Supported Glue runtimes: 5.0 and 5.1.
• As an early 0.x release, the public interface may evolve between minor versions; changes will be documented in the release notes.

License

Released under the MIT License.