aws cloudformation deploy fails although deploy continues to run on AWS

[jyrkiput]

We have embedded stack which takes a while to create. Comand "aws cloudformation deploy" fails with error code 255 and "Max attempts exceeded", although the stack continues to be created on AWS and eventually succeeds.

Version:

aws --version
aws-cli/1.11.148 Python/2.7.12+ Linux/4.10.0-35-generic botocore/1.7.6

Last lines of debug output are:

2017-09-14 11:03:04,807 - MainThread - botocore.hooks - DEBUG - Event needs-retry.cloudformation.DescribeStacks: calling handler <botocore.retryhandler.RetryHandler object at 0x7f4de3a36c10>
2017-09-14 11:03:04,808 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2017-09-14 11:03:04,809 - MainThread - awscli.customizations.cloudformation.deployer - DEBUG - Execute changeset waiter exception
Traceback (most recent call last):
  File "/home/jyrki/.virtualenvs/op-aws/local/lib/python2.7/site-packages/awscli/customizations/cloudformation/deployer.py", line 184, in wait_for_execute
    waiter.wait(StackName=stack_name)
  File "/home/jyrki/.virtualenvs/op-aws/local/lib/python2.7/site-packages/botocore/waiter.py", line 53, in wait
    Waiter.wait(self, **kwargs)
  File "/home/jyrki/.virtualenvs/op-aws/local/lib/python2.7/site-packages/botocore/waiter.py", line 329, in wait
    last_response=response
WaiterError: Waiter StackCreateComplete failed: Max attempts exceeded
2017-09-14 11:03:04,811 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/home/jyrki/.virtualenvs/op-aws/local/lib/python2.7/site-packages/awscli/clidriver.py", line 200, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/home/jyrki/.virtualenvs/op-aws/local/lib/python2.7/site-packages/awscli/clidriver.py", line 338, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "/home/jyrki/.virtualenvs/op-aws/local/lib/python2.7/site-packages/awscli/customizations/commands.py", line 187, in __call__
    return self._run_main(parsed_args, parsed_globals)
  File "/home/jyrki/.virtualenvs/op-aws/local/lib/python2.7/site-packages/awscli/customizations/cloudformation/deploy.py", line 184, in _run_main
    parsed_args.notification_arns)
  File "/home/jyrki/.virtualenvs/op-aws/local/lib/python2.7/site-packages/awscli/customizations/cloudformation/deploy.py", line 199, in deploy
    deployer.wait_for_execute(stack_name, result.changeset_type)
  File "/home/jyrki/.virtualenvs/op-aws/local/lib/python2.7/site-packages/awscli/customizations/cloudformation/deployer.py", line 188, in wait_for_execute
    raise exceptions.DeployFailedError(stack_name=stack_name)
DeployFailedError: Failed to create/update the stack. Run the following command
to fetch the list of events leading up to the failure
aws cloudformation describe-stack-events --stack-name op-smooth-operator-development
2017-09-14 11:03:04,814 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

I tried to look if it would be simple to increase maxAttempts in botocore, but that seemed to be pretty complicated.

Workaround for this is to check with aws cloudformation describe-stack if the stack really failed.

[cwgem]

The value is actually here from what I can tell:

https://github.com/aws/aws-cli/blob/develop/awscli/customizations/cloudformation/deployer.py#L182

So you could potentially manually change that (default looks to be 3600 seconds or 1 hour). That said seems like this might be something that should be configurable ala AWS config file.

[jyrkiput]

Thanks, that was added 6 days ago so I didn't have it yet. My builds take less than hour, around 5 minutes actually. So as far as I'm concerned, this has already been fixed 👍

[joguSD]

Closing this out as the fix should already be out in v1.11.152!

[nello]

This problem appears to be back in spades:

[2018-05-03T21:08:09Z] ++ aws elbv2 describe-target-groups --names ... --query 'TargetGroups[].TargetGroupArn' --output text
[2018-05-03T21:08:09Z] + aws elbv2 wait target-in-service --target-group-arn arn:aws:elasticloadbalancing:ap-southeast-2:..
[2018-05-03T21:17:57Z]
[2018-05-03T21:17:57Z] Waiter TargetInService failed: Max attempts exceeded
[2018-05-03T21:17:57Z] 🚨 Error: The command exited with status 255

[rfrad]

I faced the same issue when trying to deploy a conflicted serverless configuration.

The conflict occurred in the path defined in my template.yaml:

...
Resources:
  Profile:
    Type: AWS::Serverless::Function
    Properties:
      ...
      Events:
        CreateResource:
          Type: Api
          Properties:
            RestApiId: !Ref RestAPI
            Path: /resource/{token} # <= Here
            Method: post
        ConfirmResource:
          Type: Api
          Properties:
            RestApiId: !Ref RestAPI
            Path: /resource/{resourceId}/confirm # <= with there
            Method: post
...

The conflict is between the path parameter {token} and {resourceId}. Although both URLs are different, Cloud Formation doesn't seem to like the different names for a same path parameter. As soon as I changed the /resource/{resourceId}/confirm url to /resource/confirm/{resourceId}, it all got fixed.

[iRemjeyX]

Is there anyway to edit the waiter config without touching python code? (Like using an environment var or something?)

[konstantin-constructor]

Is there anyway to edit the waiter config without touching python code? (Like using an environment var or something?)

I am interested in the same question. We really need this as an option as we have a valid use case where the cloudformation-deploy is going over 1h as it is waiting for the ASG to get all its members ready and only then starts waiting for its Warmpool members to get ready, too.

[stealthycoin]

I agree there needs to be a better way of giving the cloudformation deploy command a ballpark estimate of how long it should be expected to take. This issue should be used for design ideas and discussions and tracking interest in this feature.

#6828 details our new contribution process.

[vvucetic]

Here you go #6905