EventScheduleRate parameter does not exist

[sudsha50]

Describe the bug
When using the latest yaml to deploy the buildkite stack, it errors out in with the error Parameters: [EventScheduleRate] do not exist in the template. Seems like there is a bug with the parameter since removing the last line where the EventScheduleRate is entered for the serverless allows the cloudformation to build.

Steps To Reproduce
Run the deploy for the https://s3.amazonaws.com/buildkite-aws-stack/latest/aws-stack.yml template to update the existing cfn stack.

Expected behavior
Deploy successfully by using the default value for ScalerEventScheduleRate in EventScheduleRate

Actual behaviour
Parameters: [EventScheduleRate] do not exist in the template

Stack parameters (please complete the following information):

• AWS Region: us-east-1
• Version [Buildkite stack v6.0.0]

Additional context
The buildkite test seems to have fail tests for the commit with the change as shown in the commit status.

[DrJosh9000]

Hi @sudinshakyatl!
EventScheduleRate was added to the scaler in buildkite-agent-scaler#93, included in scaler v1.5.0, and the v6 stack defaults the scaler version to v1.5.0 in #1169.

That param is here in the template:

elastic-ci-stack-for-aws/templates/aws-stack.yml

Lines 182 to 185 in 09eed04

	BuildkiteAgentScalerVersion:
	Description: Version of the buildkite-agent-scaler to use
	Type: String
	Default: "1.5.0"

and it looks like it defaults to v1.5.0 in latest/aws-stack.yml.

So - I'm curious - have you set an older version for BuildkiteAgentScalerVersion?

[jasoncodes]

I just ran into this issue myself just now while performing a test upgrade of a new 5.2.0 stack to 6.0.0 prior to upgrading our production stacks. I am using the AWS Console to drive CloudFormation and when updating the stack the BuildkiteAgentScalerVersion field contains 1.3.2 from the 5.2.0 template. I did not touch this field at all when I provisioned the stack with the 5.2.0.

To successfully update the stack to 6.0.0 via the UI I had to both re-enter values for the renamed parameters and update the scaler version to 1.5.0. I first tried blanking out the existing scaler version, wondering if it would reset back to the template’s default, but that failed quickly with “Resource with id [Autoscaling] is invalid. Property 'SemanticVersion' cannot be blank.”

[triarius]

Hmm, so it looks like if you create a stack with the default value for the BuildkiteAgentScalerVersion parameter and then upgrade to a stack that has a new default value for that parameter, the CloudFormation stack retains the original default parameter.

I think we can make the default value "" and then use a conditional to detect if the value has been explicitly set. Then the parameter should probably be called OverrideBuildkiteAgentScalerVersion, but changing the parameter name may cause more breakage. It should be fine to leave it as is.

[jasoncodes]

I think changing the parameter name would be a good move but presumably that’d have to wait for 7.0. In the meantime, changing the default to an empty string and a note in the change log to clear the existing value if you aren‘t intentionally overriding it is probably sufficient.

[triarius]

So I've had a bit of trouble trying to reproduce this issue. And we've definitely upgraded stacks internally where BuildkiteAgentScalerVersion was not set.

Turns out, it's a known issue when you use the CLI command aws cloudformation deploy instead of aws cloudformation update-stack: https://repost.aws/questions/QUSvocJcK-SA6rNu7QOydbIA/aws-cloudformation-deploy-doesn-t-recognize-change-in-default-value

And my command line history confirms I'd been using update-stack.

So we recommend anyone experiencing this to try that.

But the fix I suggested above still seems worth implementing, as some customers will be using the AWS Console, which also seems to suffer from this issue.

[triarius]

I think we can make the default value "" and then use a conditional to detect if the value has been explicitly set

Hmm, looks like my attempt to do this was thwarted by the Serverless Application Model (SAM):

  | Transform AWS::Serverless-2016-10-31 failed with: Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [Autoscaling] is invalid. Property 'SemanticVersion' cannot be resolved. Only FindInMap and Ref intrinsic functions are supported.
-- | --

In particular, !If in the syntax:

SemanticVersion: !If [ UseDefaultBuildkiteAgentScalerVersion, 1.5.0, !Ref BuildkiteAgentScalerVersion ]

is not supported by the AWS::Serverless-2016-10-31 transform that is used to transform the serverless application into a CloudFormation substack.

I don't see any way around this if we keep using the SAM.

There was a time when we didn't use this and hand crafted the lambda for buildkite-agent-scaler in the main stack. This is the PR where we started to use the SAM: #685.

We can explore reversing this decision, but that's going to take some time. For now, using aws cloudformation update-stack seems to be the preferred work around.