aws-ecs: Cannot deploy fargate services with ECR images #3646
Comments
|
The service creation appears to fail because it cannot pull the docker image from ECR. The error message on the created task in the cluster: |
|
Hey, me again! What does your TaskExecutionIamRole look like? Do you have |
|
Hi @fitzee Good morning! I'm not sure what exact role this CDK app creates, but I tried with an existing role called "ecsTaskExecutionRole" (for At first I thought it was some kind of networking issue, and I tried with CDK is very nice. (actually, very very very very nice :)), but it feels like it's a bit of a blackbox, and when you run into issues like this it seems hard to troubleshoot. Thanks! |
|
I ran into the same issue using ECS constructs directly as reported in #3644 . Hence, the issue does not appear to be specific to |
realharry
changed the title
|
I ran into a similar issue when trying to update my stack, which stuck at creating the service: It is a very regular service that pulls images from ECR for task definition and use |
I'm encountering the same issue - using the same "ecsTaskExecutionRole" if that matters. |
|
Got my solution, I hadn't specified an image tag when I called ecs.ContainerImage.fromEcrRepositoryso it defaulted to "latest" - our repository didn't have a latest tag so this failed. Also, @realharry , if you dig into the Service and check under details, the rest of the |
NGL321
added
bug
|
@realharry I'm sorry to hear that the CDK feels like a blackbox. We try to be as open and public with the source code as possible, there is a minimal amount of AWS information we cant give, but if you have any suggestions as to what would make the project feel less closed off, I am all ears! In the interim, did @ConradMearns solution work for you? (we're working hard to get through our issue queue, and it would be wonderful if this was already resolved!) 😸 |
|
@NGL321 Thanks for following up. I no longer work on the deployment task at this point (a colleague in the team took over the task), but my understanding was it was resolved following the direction similar to what was suggested by @ConradMearns. Thanks! I'll close this ticket. I ended up spending a little over a week (or, more like 2 weeks) with CDK, and I think I achieved quite a bit with a help of CDK (although I failed to build the whole stack (not a big architecture by any measure) using CDK within that 1~2 week time frame). My primary problem with CDK, not necessarily specific to this particular issue, was essentially my lack of knowledge across AWS service stacks. When the deployment failed, the error happened somewhere in AWS services, and CDK (being a high level tool) wasn't entirely helpful in pinpointing what exactly when wrong. But, over time, with more experience, it got easier and easier. As a general comment, I don't know how easy it would be, but if CDK could up-propagate the error messages to the user from the low-level stack where the error occurred, I think it would be very useful. Thanks! |
|
I am launching a stack for the very first time, which includes a ECR and a Fargate service that refers to a repo in the ECR that does not (yet) have an image. The intent is to push images later using Ci/CD. But my stack hangs, presumably trying to deploy the docker image that does not yet exist! // Container image
const zcloudRepository = new ecr.Repository( this, 'ZcloudRepository', {
repositoryName: `zcloud-image-${props.env}`,
});
// Lifecycle
zcloudRepository.addLifecycleRule({ maxImageAge: cdk.Duration.days(30) }); // delete older than 30 days
// Grant push access to the deploy user
zcloudRepository.grantPullPush(deployer);
// The task definition
const zcloudDefinition = new ecs.FargateTaskDefinition(this, 'ZcloudDefinition', {
// CPU and MEMORY limits here need to be equal or greater than all container instances added up
cpu: zCPU * 4,
memoryLimitMiB: zMemory * 4,
});
const container = zcloudDefinition.addContainer('zcloud', {
image: ecs.ContainerImage.fromEcrRepository(zcloudRepository),
memoryMiB: zMemory,
cpu: zCPU,
environment: {
NODE_ENV: props.env, // set up NODE_ENV from the props passed in
},
logging: new ecs.AwsLogDriver({
logGroup: props.global.paperwatchLogGroup,
streamPrefix: `zcloud-${props.env}`
})
});
container.addPortMappings({
containerPort: 3000
}); |
|
I'm having a similar problem. Is there any way to define an ECR repository and Fargate service with |
|
@peebles / @desbo any of you guys figure this out? also trying to deploy a new repository and a service in the same stack but hangs on the initial deploy due to no image being in the repo yet. edit: think I'll make a separate stack for the repo edit: ended up ditching the repo setup and using |
|
@djkirby I ended up working around this by manually creating the ECR repo outside of CloudFormation. I think |
|
Yea and I'm also realizing it'd be nice to keep the repository in the stack so it could be torn down as part of |
Running into the same problem. Came here from google, but since this issue is closed and started off as a different problem, I think the discussion should be moved to a new, separate, issue. I'm going to try to work around it by pushing the Docker image manually. |
|
There is a solution, it's not pretty, but what you can do is to find the low level construct CfnService and override the default behavior and set the desiredCount to 0. Then in your pipeline of your app you must remember to update the desiredCount after an image have been pushed to the repository. This can be done with the aws cli. A little example: |
|
Hey AWS team, I'm also having this issue. How do you solve the failed service deploy when there's no image during the first ...VPC code above...
const repository = new ecr.Repository(this, 'app-ecr', {
repositoryName: 'app-v2',
});
const cluster = new ecs.Cluster(this, 'app-cluster', {
vpc,
clusterName: 'app-v2',
});
const loadBalancedFargateService = new ecsPatterns.ApplicationLoadBalancedFargateService(this, 'app-service', {
cluster,
serviceName: 'app-v2',
desiredCount: 1,
cpu: 512,
memoryLimitMiB: 1024,
minHealthyPercent: 100,
maxHealthyPercent: 200,
publicLoadBalancer: true,
taskImageOptions:{
enableLogging: true,
image: ecs.ContainerImage.fromEcrRepository(repository),
containerPort: 3000,
}
});The above code hangs forever during Thanks for the help @drakir, your workaround helped me get this spun up. Is there any official solution to this problem @skinny85? I looked at the thread you mentioned above but it didn't seem like the same thing. Let me know if this could use a new issue, thanks! |
|
@Mk-Etlinger As a workaround the initial deployment you could try |
|
I'm also wondering which is the best practice to avoid the "no image" issue |
|
One workaround to this issue:
Not perfect && a bit ugly, but works. |
Thanks, at the end I came up with this solution 🙂 |
|
(Adding for other folks that might land here one day) FWIW, Another reason that loading from ECR might appear to hang when starting a Fargate service is because the health check is failing. An e.g. |
I'm submitting a ...
What is the current behavior?
If the current behavior is a 🪲bug🪲: Please provide the steps to reproduce
LoadBalancedFargateService.cdk deploycreates all constructs successfully, including ELB, SG, etc., but it fails to create the service, which is practically the final step:It hangs at this point, and the deploy eventually times out.
The stack should be created successfully.
I'd like to use CDK to create a fargate service.
Please tell us about your environment:
Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. associated pull-request, stackoverflow, gitter, etc)
The text was updated successfully, but these errors were encountered: