Repository navigation
Releases: NatLabRockies/sparkctl
Releases · NatLabRockies/sparkctl
Release list
v0.5.0
0.4.1
Add HPC environment module and SPARKCTL_SETTINGS_FILE support
Add a ready-to-deploy environment module (TCL and Lua flavors) under
hpc/environment_module/ so HPC users can run:
module load sparkctl
sparkctl configure --start
The module activates a shared pre-installed virtualenv, sets
SPARKCTL_SETTINGS_FILE, and puts spark-submit/pyspark on PATH. It reads
the Spark/Java/Hadoop binary paths from the shared sparkctl.toml so that
settings file is the single source of truth (TCL parses it natively; Lua
parses it via awk since Lmod's sandbox lacks reliable file I/O).
Wire up the SPARKCTL_SETTINGS_FILE environment variable in config.py
(it was already documented in the deploy guide but never implemented).
Settings files now load in increasing precedence: ~/.sparkctl.toml, then
$SPARKCTL_SETTINGS_FILE, then ./.sparkctl.toml.
Document the module and the env var in the deploy guide, and bump the
version to 0.4.1 (first release supporting SPARKCTL_SETTINGS_FILE).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>