fix(pool): scoped worker IAM profile with SQS + S3 (#70) - #467
Merged
Conversation
Pool workers used the bare spored role (no SQS), so GetQueueUrl returned NonExistentQueue (access-denied masked as not-found) and workers never pulled. Found by a real-AWS smoke. Workers now get a scoped instance profile granting SQS on the run queue + S3 on the spec/results + spawn-binaries buckets, like spawn task run.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The root cause behind "pool stalls with the queue full" — found by the third real-AWS smoke.
Symptom
Workers booted, installed spored (v0.96.1/.2 fixes worked), but the queue stayed at "4 waiting" and no task ran. SSM showed the worker exited with:
The queue existed (submitter created it,
get-queue-urlfrom operator creds resolved it).Root cause
spawn pool creategave workers the bare sharedspored-instance-role(SSM + binary-download + spored-policy) — no SQS permissions at all. SQS masks access-denied asNonExistentQueue, so the worker'sGetQueueUrl"couldn't find" a queue it simply wasn't allowed to see. (This also means #464's OpenQueue retry was looping on a permission error — correct for a real consistency race, but not the cure here.)Fix
Workers now get a scoped instance profile via
CreateOrGetInstanceProfile(the same mechanismspawn task runuses), granting exactly:spawn-pool-<runID>): GetQueueUrl / ReceiveMessage / DeleteMessage / GetQueueAttributes.spawn-binaries-*(spored bootstrap download) + read/write the spec/results bucket (fetch staged specs; write completion.json/.exitcode).CreateOrGetInstanceProfilealso guarantees SSM. New test asserts the policy is valid JSON, SQS is scoped to the run queue (never*), and the buckets are granted.Known limitation (documented, follow-up)
A task's OWN input/output buckets beyond the results bucket aren't known at pool-create and aren't granted. The common nf-spawn case keeps work in the results/work bucket; broader per-task bucket grants are a follow-up.
On the smoke-loop
This is the 4th bug a paid smoke surfaced (MissingParameter → checksum → boot-race → IAM). I audited the worker's full happy-path requirements this pass (SQS verbs + S3 buckets for bootstrap/spec/results) rather than fixing one call, so the next smoke should get materially further.
go build, cmd gates, and the new policy test pass.Refs #70.