v0.2.0

AWS Glue Toolkit v0.2.0

AWS Glue Toolkit v0.2.0 expands gtk from packaging-only into a local Glue development loop: run jobs and tests against the official Glue Docker images, with richer dependency forms and container-faithful pip.

What’s new

Local job execution — gtk run

spark-submit on the entry script in the Glue image for glue_version. Passes --JOB_NAME from project.name unless overridden. Extra --key value tokens after [JOB-DIR] go to getResolvedOptions. Shuts down the Spark driver so the container returns.

Local testing — gtk test

python3 -m pytest in the Glue image. PYTHONPATH includes source (and the install target when deps are present). Default target is the optional tests field (default: tests):

[tool.aws-glue-toolkit]
tests = "tests"

Path and git dependencies

project.dependencies now accepts file: (in-job or monorepo) and git/VCS requirements alongside PyPI. Path deps must be installable packages (pyproject.toml or setup.py); editable installs (-e) remain rejected.

dependencies = [
  "pandas>=2",
  "my-lib @ file:./libs/my-lib",
  "shared @ file:../packages/shared",
  "tool @ git+https://github.com/org/tool.git@v1",
]

Host pip configuration in containers

For container pip (check, build, and dep install on run / test), gtk snapshots the host’s effective pip settings (files + PIP_*) into a temporary pip.conf, mounts it, and sets PIP_CONFIG_FILE. Host-local keys (cache-dir, cert, target, and similar) are omitted.

Ephemeral dependency install on run / test

When project.dependencies is non-empty, packages install into a temporary PYTHONPATH target inside the container (Glue pins as constraints) before the job or tests run.

What’s improved

Docker-backed gtk check and gtk build

Dependency resolve and gluewheels pip wheel now run inside the Glue image—same Python and platform as the worker—not on the host. Artifact names and Glue parameters (.dependencies.zip / .gluewheels.zip) are unchanged from v0.1.0.

Runtime metadata for Glue 5.0 and 5.1 now includes the official image references (public.ecr.aws/glue/aws-glue-libs:5 and :5.1).

Exit codes

Code	Meaning
69	Docker not available or could not be started

For run / test, a successful Docker launch returns the container command’s exit code (or pip’s if install fails). Existing codes (65, 66, 70, 78) are unchanged; 78 also covers invalid dependencies.

Breaking changes from v0.1.0

• Docker is required for all gtk commands, including check and build (previously host-side pip).
• Dependency resolve and gluewheels pip execute in the Glue container, not on the host.