Jenkins Startup fails on AWS ECS due to secrets-manager-credentials-provider-plugin #117
Comments
|
First thing to check, what IAM permissions does the Jenkins principal (or if on a cluster, the nearest relevant principal) have? If Jenkins doesn't have the right permissions it will not be able to access secrets. |
|
I SSH'd onto my ECS server, git cloned my Dockerfile, built and ran the image and was able to replicate the error. @chriskilding I also suspected IAM perms were the root cause, but at the time was unsure. You may or may not know this, but where do I attach the IAM Policy?
Any contribution is greatly appreciated. |
|
I've not used Jenkins on ECS myself, but I imagine you could start by putting it on the ECS task definition. Either directly, or you might need to make a Jenkins IAM role with the policy, and set the role ARN on the task definition. |
|
Upon further investigation, the secrets-manager plugin want's access to instance-identity-documents This metadata is only available on EC2 instances. So if someone tries to use JCasC + ECS + secrets-manager-credentials-provider-plugin they are going to run into this issue, cause the container application can't natively access the hosts metadata file. I'll see if I can get Jenkins on ECS to access the hosts metadata file. This will also be an issue on EKS. |
|
I can confirm that IAM is not the root cause, If the plugin fails to pull secrets due to insufficent permissions in the IAM policy, the error message will look something like this... Instead the Error is this... In an attempt to mimic AWS ECS I did the following...
The container started cause it could access, When starting Jenkins in AWS ECS, that endpoint is not available. Not really sure how to procede now. |
|
I believe we have teams in our workplace that do use the plugin on ECS or EKS, so I'll ask around and see what they suggest |
|
As a short term suggestion, you could try setting the |
|
It's also possible that there is simply a bug to do with this in the AWS Java SDK, that may since have been fixed. Could you post which version of the Jenkins AWS Java SDK plugin you have? |
I'm running Jenkins in docker using the jenkins@12bf88526df6:/$ java -version
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)AWS SDK Plugin Amazon Web Services SDK - 1.11.995
|
|
I'm looking again at the original NullPointerException stack trace you posted. Most likely the The client is created in an Could you scan back in your logs and find a message around the time of that stack trace that starts with: "Could not set up AWS Secrets Manager client. Reason:" And post it? |
|
(That message should be logged at WARNING level btw) |
|
Hey @chriskilding , i'll have a look at this soon. Also Parameter Store is free but Scrts Mgr is .40c a request/per month. |
|
It is a good question, AWS themselves seem to have created 2 services that sort-of overlap but not quite. I've never seen an explanation for why they did this. I compared them a while back in #72. From that table, you'd use Secrets Manager if you need:
Beyond that, we'd have to know about the design intent of each service. |
|
I was also running into this issue with other plugins that rely on the AWS SDK. It's annoying cause now I'm locked into 1 region, I could see how this might cause issues with larger organisations. I wonder how other people are solving this problem? @chriskilding |
|
To authenticate with AWS, the plugin is merely creating a standard version of the Secrets Manager client, which uses the DefaultAWSCredentialsProviderChain under the hood. This is what any other user of the AWS Java SDK (V1) would do - it's not specific to Jenkins plugins. Per AWS docs the chain looks for credentials in this order:
The last 2 approaches on the list (any form of Amazon container lookup, be that EC2 or ECS) are handled in EC2ContainerCredentialsProviderWrapper. The docs for that say:
The fact that in your case it's falling through to the EC2 metadata service suggests the ECS code branches didn't supply a credential. Could you check if one of those AWS_CONTAINER environment variables are set (or if not set it anyway for Jenkins as an override) and see if that changes things? |
|
Not sure if this was mentioned: Several online guides for Jenkins + ECS include a guideline to block access from the container to the instance metadata. Here are the instructions to do so https://aws.amazon.com/premiumsupport/knowledge-center/ecs-container-ec2-metadata/ . I would recommend that you check your ECS stack and if there are similar instructions in the USER_DATA of the Launch Configuration or Launch Template used for launching ECS Container Instances. |
|
Hi I had the same problem @thedevopsguyblog note because that my exact stack: casc + ECS + secrets-manager-credentials-provider-plugin.
|
|
Great! If setting AWS_REGION is a solution that consistently works for ECS, we can add that to the README. (It's a little strange that a common AWS_ environment variable wouldn't be set by default in an AWS environment though.) |
|
Hi @chriskilding, I will close this issue. As you mentioned it's not the plugin thats at fault but its the AWS SDK behaving weirdly. It also seems that you will only run into this issue if your are using an EC2 cluster with ECS. @jairov4 has confirmed the workaround. |
|
Great, I've also added the workaround (the note about setting |
Version report
https://stackoverflow.com/questions/68287374/jenkins-startup-fails-on-aws-ecs-due-to-secrets-manager-credentials-provider-plu <- also posted this.
Jenkins and plugins versions report:
Reproduction steps
Results
Expected result:
Jenkins starts up, and can access the secrets.
Actual result:
The text was updated successfully, but these errors were encountered: