Severity
May cause deployment errors
Version
N/A - This is a Octopus Cloud hosted Dynamic Worker issue - it does not affect Octopus Server
Latest Version
Not applicable
What happened?
Recent commits to our Dynamic Worker toolset, specifically to a more recent version of the aws iam authenticator has resulted in some AWS interactions to fail. We are currently investigating this as a priority.
This can be displayed as warnings in your logs similar to the below:
level=info msg="failed to get region from IMDS for token generation, defaulting to us-east-1. imds error: operation error ec2imds: GetRegion, failed to get API token, operation error ec2imds: getToken, http response error StatusCode: 400, request to EC2 IMDS failed"
It can also cause some deployments to fail with errors in the logs similar to the below:
E0401 18:01:22.791272 63172 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
error: You must be logged in to the server (the server has asked for the client to provide credentials)
"/opt/homebrew/bin/kubectl" create namespace default --request-timeout=1m
error: You must be logged in to the server (Unauthorized)
Or
aws: [ERROR]: Invalid endpoint: https://sts/..amazonaws.com
Reproduction
Have a deployment which involves AWS Resources and uses our Octopus Cloud Dynamic workers.
Error and Stacktrace
"/opt/homebrew/bin/kubectl" get namespace default --request-timeout=1m
E0401 18:01:22.495277 63172 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0401 18:01:22.570413 63172 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0401 18:01:22.636382 63172 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0401 18:01:22.709735 63172 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0401 18:01:22.791272 63172 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
error: You must be logged in to the server (the server has asked for the client to provide credentials)
"/opt/homebrew/bin/kubectl" create namespace default --request-timeout=1m
error: You must be logged in to the server (Unauthorized)
aws: [ERROR]: Invalid endpoint: https://sts..amazonaws.com
More Information
Initial ticket (internal) - https://octopuscd.zendesk.com/agent/tickets/196906
RnD thread (Internal) - https://octopusdeploy.slack.com/archives/CNHBHV2BX/p1774888154681829
Workaround
If you are affected, adding your AWS region as an environment variable in your script steps may resolve the issue:
export AWS_REGION=[REGION] for Bash
$env:AWS_REGION = "[REGION]" for Powershell
E.g
export AWS_REGION=ap-southeast-2
$env:AWS_REGION = "ap-southeast-2"
Severity
May cause deployment errors
Version
N/A - This is a Octopus Cloud hosted Dynamic Worker issue - it does not affect Octopus Server
Latest Version
Not applicable
What happened?
Recent commits to our Dynamic Worker toolset, specifically to a more recent version of the
aws iam authenticatorhas resulted in some AWS interactions to fail. We are currently investigating this as a priority.This can be displayed as warnings in your logs similar to the below:
level=info msg="failed to get region from IMDS for token generation, defaulting to us-east-1. imds error: operation error ec2imds: GetRegion, failed to get API token, operation error ec2imds: getToken, http response error StatusCode: 400, request to EC2 IMDS failed"It can also cause some deployments to fail with errors in the logs similar to the below:
Or
aws: [ERROR]: Invalid endpoint: https://sts/..amazonaws.comReproduction
Have a deployment which involves AWS Resources and uses our Octopus Cloud Dynamic workers.
Error and Stacktrace
More Information
Initial ticket (internal) - https://octopuscd.zendesk.com/agent/tickets/196906
RnD thread (Internal) - https://octopusdeploy.slack.com/archives/CNHBHV2BX/p1774888154681829
Workaround
If you are affected, adding your AWS region as an environment variable in your script steps may resolve the issue:
export AWS_REGION=[REGION]for Bash$env:AWS_REGION = "[REGION]"for PowershellE.g
export AWS_REGION=ap-southeast-2$env:AWS_REGION = "ap-southeast-2"