feat(pool): resilient restart-on-error workers (#465) + task-bucket IAM flags (#70) - #470
Merged
Conversation
…3-write for task buckets (#70) #465: on-instance worker now runs under a bounded restart-on-error loop — a transient exit re-execs, a clean idle-drain (exit 0) stops and lets the instance terminate. No more billing-but-idle stranded workers. #70 follow-up: --s3-read/--s3-write grant workers access to a task's own input/output buckets beyond the results bucket, closing the documented per-task-bucket IAM limitation. Least-privilege default (results bucket only).
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The two follow-ups from the now-green pool validation (#70).
#465 — resilient workers (restart-on-error)
The on-instance worker ran
spored pool-workeronce at boot, so any early exit (a transient AWS error, an SQS blip, a spot pre-warm race) left a running-but-idle instance billing until its TTL.buildPoolWorkerCommandnow wraps it in a bounded restart-on-error loop with the exit-code contract:Tests: asserts the loop shape + valid bash (
bash -n), and a functional test with a stubbedsporedproving exit-0 runs exactly once (no restart on clean drain).#70 follow-up — per-task-bucket IAM
Workers got S3 on the results/work bucket only, so a task staging from/to a third bucket failed (documented limitation when the scoped profile landed). New
spawn pool create --s3-read <bucket>/--s3-write <bucket>(repeatable) widen the worker policy: read buckets → GetObject + List; write buckets → Get/Put/Delete + List. Least-privilege default unchanged (results bucket only).s3://bucket/prefixforms are accepted and trimmed to the bucket for the ARN.Test asserts the extra grants appear, prefixes are stripped, and a read-only bucket never gets DeleteObject.
All new tests + the cmd SDK-import/flag-convention gates pass. (
TestCatalogValidfails on main independently — private ECR images in the shipped catalog, #392 — not touched here.)Refs #70, closes #465.