Skip to content

v1.1.0 — GPU lifecycle policies by workload

Latest
Latest

Choose a tag to compare

View all tags
@serithemage serithemage released this 24 Oct 06:54
v1.1.0
b35b018

v1.1.0 — GPU lifecycle policies by workload

This release adds usage modes that control when GPU instances run and stop (#43). CDK context (-c usageMode=...) and the tkf CLI use the same policy rules.

Usage modes

Mode Command GPU lifecycle
interactive (default) tkf up <model> Scales the ASG to zero after 30 minutes without ALB requests. Use --idle-minutes to change the interval. This preserves the existing default behavior.
batch tkf run <model> --region <r> -- <command> Sends a heartbeat every minute to prevent idle shutdown while the command runs, then requests GPU shutdown when the command finishes. The idle alarm remains as a fallback if the CLI process or host is lost.
online tkf up <model> --usage-mode online Disables idle shutdown and maintains a minimum capacity of one after READY. Choose this explicitly when continuous availability justifies continuous GPU charges.

Nonzero idleMinutes values must be multiples of five between 5 and 1440. Interactive mode also accepts 0 to disable idle shutdown; online requires 0, while batch rejects 0. Conflicting settings, such as online with a 30-minute idle timeout or batch with minCapacity=1, fail during synthesis.

Batch refuses to take over a stack with requested capacity or remaining instances. A lock in ~/.token-forge/locks/ also prevents another run, up, or seed on the same host from using that stack. There is no lock across hosts.

The batch wrapper uses POSIX process groups and targets macOS and Linux. Windows is not supported or validated. If tkf run receives SIGKILL or its host is lost, the CLI cannot perform cleanup. The idle alarm may stop the GPU, but this is not guaranteed, especially if the process dies within roughly 5–10 minutes of requesting the GPU. After an abnormal exit, run tkf down --region <region> and verify shutdown with tkf status.

Costs and migration

The English, Korean, and Japanese READMEs now explain costs by usage mode. Batch workloads still incur charges during the few minutes between the shutdown request and instance termination.

Stacks created before this feature have no usage-mode or deployment-settings records. Plain tkf up remains usable and preserves the existing configuration. Before selecting online mode, changing the idle policy, or using tkf run, update the stack once with the context values used for its original deployment:

npx cdk deploy -c model=<m> -c profile=<p> -c region=<r> -c minCapacity=0 \
  -c usageMode=interactive -c idleMinutes=30 [-c allowedCidrs=... -c azs=... -c alertEmail=...]

Preserving existing settings

Stacks now expose UsageMode, IdleMinutes, and DeploymentSettings. The last output is versioned JSON containing allowedCidrs, azs, and alertEmail.

When a requested policy differs from the deployed policy, the CLI passes the recorded settings back to cdk deploy. This preserves the IP allowlist, AZ selection, and email subscriptions. Missing or invalid settings records prevent automatic updates; the CLI instead provides instructions for a manual deployment.

Validation

  • All 246 tests across 25 suites passed with npm test -- --runInBand. npm run build also passed.
  • The offline synthesis script, scripts/verify-usage-mode-synth.ts, passed 13 cases: eight valid configurations and five expected rejections.
  • The extracted package tarball provided version 1.1.0 and working tkf run --help and tkf up --help commands.

Live deployment scope and limitations

An existing qwen3-coder-30b/fp8 stack in Seoul (ap-northeast-2) was updated with interactive mode, a 30-minute idle timeout, and minimum and desired capacity both set to zero. CloudFormation reached UPDATE_COMPLETE without rollback. The three new outputs matched the expected values, and the endpoint remained unchanged.

No GPU was started. Booting the new AMI, vLLM inference, an actual batch run, and switching to online mode were not validated in a live environment.

Other changes since v1.0.2

  • Added English, Korean, and Japanese support to the spot dashboard and fixed stability-card overflow (#36).
  • Moved the dashboard to GitHub Pages at spot.roboco.io, with a 301 redirect from the previous CloudFront URL (#40).
  • Added social sharing previews and the Roboco logo (#42).
  • Added Google Analytics 4 measurement to the dashboard.

Full Changelog: v1.0.2...v1.1.0

Publication record

PR and main CI checks passed, as did the release workflow's tests and build. After the CI npm publish request failed because of an authentication error, the same validated tarball was published manually following a fresh login and security-key approval. The manually published artifact does not include CI provenance.

Assets 2
Loading