Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add a command to cloudformation wait any final state, and display events while waiting #2887

Open
hnafar opened this issue Oct 11, 2017 · 18 comments
Labels
cloudformation feature-request A feature should be added or improved. p3 This is a minor priority issue waiter

Comments

@hnafar
Copy link

hnafar commented Oct 11, 2017

This is specially useful during automated deployments.

  • wait until stack is in any final state:
    • CREATE_FAILED
    • DELETE_COMPLETE
    • DELETE_FAILED
    • ROLLBACK_COMPLETE
    • ROLLBACK_FAILED
    • UPDATE_COMPLETE
    • UPDATE_ROLLBACK_COMPLETE
    • UPDATE_ROLLBACK_FAILED
  • log stack events emitted while waiting.
    Currently I achieve this using the following script, but having this as part of the cli would be much more helpful:
cloudformation_tail() {
  local stack="$1"
  local region="$2"
  local lastEvent
  local lastEventId
  local stackStatus=$(aws cloudformation describe-stacks --region $region --stack-name $stack | jq -c -r .Stacks[0].StackStatus)

  until \
	[ "$stackStatus" = "CREATE_COMPLETE" ] \
	|| [ "$stackStatus" = "CREATE_FAILED" ] \
	|| [ "$stackStatus" = "DELETE_COMPLETE" ] \
	|| [ "$stackStatus" = "DELETE_FAILED" ] \
	|| [ "$stackStatus" = "ROLLBACK_COMPLETE" ] \
	|| [ "$stackStatus" = "ROLLBACK_FAILED" ] \
	|| [ "$stackStatus" = "UPDATE_COMPLETE" ] \
	|| [ "$stackStatus" = "UPDATE_ROLLBACK_COMPLETE" ] \
	|| [ "$stackStatus" = "UPDATE_ROLLBACK_FAILED" ]; do
	
	#[[ $stackStatus == *""* ]] || [[ $stackStatus == *"CREATE_FAILED"* ]] || [[ $stackStatus == *"COMPLETE"* ]]; do
	lastEvent=$(aws cloudformation describe-stack-events --region $region --stack $stack --query 'StackEvents[].{ EventId: EventId, LogicalResourceId:LogicalResourceId, ResourceType:ResourceType, ResourceStatus:ResourceStatus, Timestamp: Timestamp }' --max-items 1 | jq .[0])
	eventId=$(echo "$lastEvent" | jq -r .EventId)
	if [ "$eventId" != "$lastEventId" ]
	then
		lastEventId=$eventId
		echo $(echo $lastEvent | jq -r '.Timestamp + "\t-\t" + .ResourceType + "\t-\t" + .LogicalResourceId + "\t-\t" + .ResourceStatus')
	fi
	sleep 3
	stackStatus=$(aws cloudformation describe-stacks --region $region --stack-name $stack | jq -c -r .Stacks[0].StackStatus)
  done

  echo "Stack Status: $stackStatus"
}

cloudformation_tail $SERVICE_STACK_NAME $REGION

which produces the following result on an update:

Awaiting completion...
2017-10-11T19:28:51.066Z - AWS::CloudFormation::Stack - my-stack-name - UPDATE_IN_PROGRESS
2017-10-11T19:28:57.591Z - AWS::ECS::TaskDefinition - TaskDefinition - UPDATE_IN_PROGRESS
2017-10-11T19:29:00.282Z - AWS::ECS::Service - Service - UPDATE_IN_PROGRESS
2017-10-11T19:36:05.919Z - AWS::CloudFormation::Stack - my-stack-name - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
Stack Status: UPDATE_COMPLETE
@jamesls
Copy link
Member

jamesls commented Oct 12, 2017

Thanks for the suggestion, marking as a feature request.

@jamesls jamesls added the feature-request A feature should be added or improved. label Oct 12, 2017
@pcgeek86
Copy link

FYI the AWS Tools for PowerShell currently offer this feature, starting with version 3.3.119.0. The SDK includes a Wait-CFNStack command, which pauses execution until the desired CloudFormation stack state has been reached.

There's an example of its use in the blog post and documentation, linked to below.

AWS Developer Blog Post (2017-07-11)
AWS Tools for PowerShell Documentation Page

Cheers,
Trevor Sullivan

@ASayre
Copy link
Contributor

ASayre commented Feb 6, 2018

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

@ASayre ASayre closed this as completed Feb 6, 2018
@jamesls jamesls reopened this Apr 6, 2018
@jamesls
Copy link
Member

jamesls commented Apr 6, 2018

Based on community feedback, we have decided to return feature requests to GitHub issues.

@ghost
Copy link

ghost commented Mar 26, 2019

Would be great to have this implemented.
My current solution look like this

aws cloudformation deploy --stack-name "${stack_name}" \
  --template-file "${stack_template_file}" \
  --no-fail-on-empty-changeset \
  | while read -r line; do echo "$line"
      if [[ "$line" = "Waiting for stack create/update to complete" ]]; then
        "aws-cloudformation-deploy-watch.sh" --stack-name "${stack_name}"
      fi
    done

aws-cloudformation-deploy-watch.sh

#!/usr/bin/env sh -euo pipefail

############################## Usage ################################
#
# aws-cloudformation-deploy-watch.sh --stack 'STACK_NAME'
#
#####################################################################

while [ $# -gt 0 ]; do
  case $1 in
    --stack-name) stack_name_opt="$2"; shift 2;;    
    --profile)    profile_opt="$2"; shift 2 ;;
    --region)     region_opt="$2"; shift 2;;
    *) echo "$0: unexpected argument: $1" >&2; exit 1;;
  esac
done

_TAB=$'\t'
most_recent_event_id='???'
while : ; do
  # get recent stack events
  most_recent_event_id_sed_escaped="$(echo "$most_recent_event_id" | sed -e 's/[]\/$*.^[]/\\&/g')"
  recent_events="$(aws cloudformation describe-stack-events \
      $([ -n "${stack_name_opt:-}" ] && echo "--stack-name $stack_name_opt") \
      --query 'StackEvents[*].[EventId,Timestamp,LogicalResourceId,ResourceStatus,ResourceStatusReason]' \
      $([ -n "${profile_opt:-}" ] && echo "--profile $profile_opt") \
      $([ -n "${region_opt:-}" ] && echo "--region $region_opt") \
      --output text \
    | sed  "/${_TAB}User Initiated\$/q" | sed "/^${most_recent_event_id_sed_escaped}${_TAB}/,\$d" \
    | tail -r)"
  if [ -n "$recent_events" ]; then
    # save last event id
    most_recent_event_id=$(echo "$recent_events" | tail -1 | awk -F $'\t' '{printf $1}')
    while read -r event; do
      # print new event
      echo "$event" | awk -F $'\t' '{printf "%-28s %-48s %-24s %s\n",$2,$3,$4,$5}' | sed "s/None$//"
      # looking for stack resource events
      event_resouce=$(echo "$event" | awk -F $'\t' '{printf $3}')
      if [ "$event_resouce" == "$stack_name_opt" ]; then
        stack_status="$(echo "$event" | awk '{printf $4}')"
        # exit on final stack status
        if [[ "$stack_status" = *'_FAILED' ]] || [[ "$stack_status" = *'_ROLLBACK_COMPLETE' ]]; then exit 1; fi
        if [[ "$stack_status" = *'_COMPLETE' ]]; then exit 0; fi
      fi
    done <<< "$recent_events"
  fi
  sleep 1
done

@thearthur
Copy link

Can we re-open this here since the uservoice migration appears no longer to be a thing? (or at least the links here are broken)

@lantrix
Copy link

lantrix commented Oct 30, 2019

It seems this has been implemented here:
https://docs.aws.amazon.com/cli/latest/reference/cloudformation/wait/index.html

However you need to call different operations with the CLI to differentiate between waiting for a Stack Create vs. a Stack Update to be complete:

aws cloudformation wait stack-create-complete --stack-name mystack
aws cloudformation wait stack-update-complete --stack-name mystack

The Powershell implementation of this (Wait-CFNStack) uses one command to hit DescribeStacks API and wait for the status of your choice:

Wait-CFNStack -StackName mystack -Timeout 300 -Status CREATE_COMPLETE,UPDATE_COMPLETE

It's a bit of a strange situation though. CLI and Powershell SDK usage are flipped in the way they work.

With the CLI I can use aws cloudformation deploy (creating and then executing a change set) which will update a stack or create a new stack with one command. Great for my CD that doesn't need to know if it is a first time deploy or an update. However I then have to determine if the stack was created OR updated to pick one of the two AWS CLI commands:

However with PowerShell SDK I can also use the DescribeStacks API to determine if the stack was created OR updated, however I get a single cmdlet I can use Wait-CFNStack. Great for CD - HOWEVER I'm unable to have a single command to create or update my stack. I have to pick one or the other.

  • New-CFNStack - the CreateStack API operation
    • New-CFNStack -StackName "myStack" -TemplateBody "{TemplateContent}"
  • Update-CFNStack -the UpdateStack API operation
    • Update-CFNStack -StackName "myStack" -TemplateBody "{TemplateContent}"

I've written PowerShell to detect if the stack exists, and do change sets - but this is onerous.

@revmischa
Copy link

This feature would be super useful for scripting test deployments of my cloudformation stack in CI

@weickmanna
Copy link

+1

1 similar comment
@weickmanna
Copy link

+1

@mattBrzezinski
Copy link

Would love to see, any updates on this feature request?

@mdforbes500
Copy link

Any progress on this? It is really annoying to have to specify in jenkins whether you need an update or creation.

@jmclean-cnexus
Copy link

+1

1 similar comment
@3lricko
Copy link

3lricko commented Feb 11, 2021

+1

@covertbert
Copy link

Are there any plans to implement this? It's crazy that we all have to write scripts ourselves to do this (and they always end up being a little flakey in my experience). This command would be more useful than any of the individual "wait for a specific state" commands that already exist.

@tim-finnigan
Copy link
Contributor

These are the wait commands currently available for cloudformation:

  • change-set-create-complete
  • stack-create-complete
  • stack-delete-complete
  • stack-exists
  • stack-import-complete
  • stack-rollback-complete
  • stack-update-complete
  • type-registration-complete

(https://awscli.amazonaws.com/v2/documentation/api/latest/reference/cloudformation/wait/index.html)

A few different topics have been mentioned here in the discussion so I want to confirm what the feature request is. @hnafar I believe you’re asking for the ability to see every stack status logged while waiting to reach one of those states above. Is that correct?

@dfehrenbach
Copy link

dfehrenbach commented Nov 16, 2021

I don't mean to speak for @hnafar, but it's the idea that "wait commands" should be allowed to be bundled. Either do that by saying:

  1. "Any one of the final states have been reached."
  2. "Any one of the final states that you have defined have been reached."

As an example, imagine a ci/cd pipeline where you want to auto-update your cloud formation stack when you've pushed new code. It's a simple idea, but to avoid errors from the AWS CLI, you must define a series of waits based on the current status of the stack.

  1. Is it not created yet? Well, you wait create-stack-complete before using either create-stack or deploy.
  2. Is it created but currently updating? Then, you wait update-stack-complete.
  3. Was the stack actually rolling back and you're pushing a quick change to fix whatever caused the rollback? Then, you wait stack-rollback-complete.
    etc.

This is not to mention that it's not clear what happens mid update when your stack has to roll back, but you're in the middle of wait update-stack-complete since your code inspected the status and at the time the cf stack was mid-update.

Simply, it's cumbersome. I'd not say it's all bad, however. The wait clauses being there are far better than the alternative (but that's neither here nor there).

Here's an example implementation (still a bit limited) that collects final states: https://www.npmjs.com/package/@cumulusds/aws-cloudformation-wait-ready
It uses just a simple string match on the end to inspect for final states: (i.e. https://github.com/CumulusDS/aws-cloudformation-wait-ready/blob/master/src/index.js#L22 )

Another implementation to look at would be the aforementioned one in PowerShell (as called out by @pcgeek86 & @lantrix). Doing this but in AWS CLI is what I would think the goal is. I imagine a day where we're done in 2 lines (please don't take the following as a concrete task, it's an expression of thought and is simply for ideation):

aws cloudformation wait cfn-stack --status CREATE_COMPLETE,UPDATE_COMPLETE ... (new command)
aws cloudformation deploy ... (existing command)

Or, what if one line?
aws cloudformation deploy --wait-for-status CREATE_COMPLETE,UPDATE_COMPLETE ...

@covertbert
Copy link

covertbert commented Nov 17, 2021

While I also don't claim to speak for @hnafar, what @dfehrenbach says describes my issue exactly.

We also use the npm package aws-cloudformation-wait-ready but it's not perfect at all and sometimes we still have issues where that script will be "done waiting" but when the next step in the CI pipeline (a cloudformation update) runs it will still error due to the stack still creating/updating/rollbacking/whatever.

The usefulness of this feature for CI pipelines cannot be overstated IMO.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cloudformation feature-request A feature should be added or improved. p3 This is a minor priority issue waiter
Projects
None yet
Development

No branches or pull requests