Skip to content

0.4.1

Choose a tag to compare

View all tags
@daniel-thom daniel-thom released this 30 May 21:45
· 28 commits to main since this release
0.4.1
6d3a96d 
Add HPC environment module and SPARKCTL_SETTINGS_FILE support

Add a ready-to-deploy environment module (TCL and Lua flavors) under
hpc/environment_module/ so HPC users can run:

    module load sparkctl
    sparkctl configure --start

The module activates a shared pre-installed virtualenv, sets
SPARKCTL_SETTINGS_FILE, and puts spark-submit/pyspark on PATH. It reads
the Spark/Java/Hadoop binary paths from the shared sparkctl.toml so that
settings file is the single source of truth (TCL parses it natively; Lua
parses it via awk since Lmod's sandbox lacks reliable file I/O).

Wire up the SPARKCTL_SETTINGS_FILE environment variable in config.py
(it was already documented in the deploy guide but never implemented).
Settings files now load in increasing precedence: ~/.sparkctl.toml, then
$SPARKCTL_SETTINGS_FILE, then ./.sparkctl.toml.

Document the module and the env var in the deploy guide, and bump the
version to 0.4.1 (first release supporting SPARKCTL_SETTINGS_FILE).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Assets 2
Loading