Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ecs): adding a circuit breaker causes Service replacement (under feature flag) #22467

Conversation

fergusdixon
Copy link
Contributor

@fergusdixon fergusdixon commented Oct 12, 2022

Fixes #16126

Copied from #16919 with review comments as it went stale and closed.
Initial implementation #22328 reverted because it was not backward compatible.

Introduces feature flag:

  • @aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker

Enable this feature flag to avoid setting the "ECS" deployment controller when adding a circuit breaker to an
ECS Service, as this will trigger a full replacement which fails to deploy when using set service names.
This does not change any behaviour as the default deployment controller when it is not defined is ECS.

cdk.json

{
  "context": {
    "@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true
  }
}

All Submissions:

Adding new Unconventional Dependencies:

  • This PR adds new unconventional dependencies following the process described here

New Features

  • Have you added the new feature to an integration test?
    • Did you use yarn integ to deploy the infrastructure and generate the snapshot (i.e. yarn integ without --dry-run)?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@gitpod-io
Copy link

gitpod-io bot commented Oct 12, 2022

@aws-cdk-automation aws-cdk-automation requested a review from a team October 12, 2022 10:52
@github-actions github-actions bot added beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK bug This issue is a bug. effort/small Small work item – less than a day of effort p2 labels Oct 12, 2022
@fergusdixon fergusdixon changed the title Fix/remove explicit ecs deployment type for circuit breaker fix(ecs): option to remove explicit addition of ecs deployment type when circuit breaker is enabled Oct 12, 2022
@mergify mergify bot dismissed TheRealAmazonKendra’s stale review October 19, 2022 07:49

Pull request has been modified.

* To avoid a service replacement when adding a CircuitBreaker to an existing ECS service, set this to false.
* @default true
*/
readonly useExplicitEcsDeploymentController?: boolean;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fergusdixon I've been thinking about this and as much as a don't want to introduce a new feature flag I think that's the best thing to do here. We want the new default behavior to be false and a feature flag is the only way to accomplish that.

https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md#feature-flags

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure will give it a bash next week, thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@corymhall I have a working implementation with the flag, however I'm struggling to see this to false in the integration tests specifically. Should the integration tests cover new or old behaviour (I would imagine both), however I cant seem to set the feature flag value the same way as the unit tests do?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems the integ-helper sets all feature flags to FUTURE_FLAGS (new behaviour), and this takes precedence over the context prop passed to App in the test. So from this it seems integration tests should test new behaviour and unit tests for both?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want to test the old behavior you can use the postCliContext parameter on App.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have done, worked a charm, thanks @corymhall!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really want to ship this, but this property is giving me pause. I really don't see the need for it.

  • If you want the ECS controller explicitly, you can just supply it ("The customer is always right", below)
  • If you do NOT want the ECS controller, you can set the feature flag
    • If you want the ECS controller for some services and not other, you can either set the feature flag globally and add the explicit parameter to some services, or set the feature flag locally (you can flip a flag at any construct scope, it's all just context which is freely manipulable).

I think we can do without this, no?

Other than that, this is great!

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to cross-check "the need for the 'correct' default after this PR is merged": Now with current CDKv2 versions today, CFN output always has an ECS DeploymentController defined, if you have an CircuitBreaker enabled (even if you not aware as CDK user).
Once a stack has an ECS DeploymentController definied (= all stack build by CDK with CircuitBreaker enabled), you are not able to simply "remove this CFN-lines from JSON output" - If a new/upcoming CDK version would start to not output ECS DeploymentController (by default) to JSON, all existing Services of existing Stacks would run into a replacement - Just because you update your CDK version.

As I understood the current code from Fergus, this default = true would remain the same behaviour as older CDKv2 version, right? - From my point of view that, it's an important brick, isn't it?

@corymhall corymhall self-assigned this Oct 21, 2022
@mergify mergify bot dismissed corymhall’s stale review October 25, 2022 07:36

Pull request has been modified.

@fergusdixon fergusdixon changed the title fix(ecs): option to remove explicit addition of ecs deployment type when circuit breaker is enabled fix(ecs): option to remove explicit addition of ecs deployment type when circuit breaker is enabled (under feature flag) Oct 25, 2022
@fergusdixon fergusdixon force-pushed the fix/remove-explicit-ecs-deployment-type-for-circuit-breaker branch from 9e3f064 to 5aa23fb Compare October 25, 2022 15:38
@mergify mergify bot dismissed rix0rrr’s stale review November 3, 2022 15:17

Pull request has been modified.

rix0rrr
rix0rrr previously approved these changes Nov 4, 2022
@mergify
Copy link
Contributor

mergify bot commented Nov 4, 2022

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@rix0rrr rix0rrr changed the title fix(ecs): option to remove explicit addition of ecs deployment type when circuit breaker is enabled (under feature flag) fix(ecs): adding a circuit breaker replaces (and downscales) a Service (under feature flag) Nov 4, 2022
@rix0rrr rix0rrr changed the title fix(ecs): adding a circuit breaker replaces (and downscales) a Service (under feature flag) fix(ecs): adding a circuit breaker causes Service replacement (under feature flag) Nov 4, 2022
@mergify
Copy link
Contributor

mergify bot commented Nov 4, 2022

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@fergusdixon
Copy link
Contributor Author

@rix0rrr any idea why the deploy queue failed?

@aws-cdk-automation
Copy link
Collaborator

This PR cannot be merged because it has conflicts. Please resolve them. The PR will be considered stale and closed if it remains in an unmergeable state.

@mergify mergify bot dismissed rix0rrr’s stale review November 7, 2022 10:54

Pull request has been modified.

@mergify
Copy link
Contributor

mergify bot commented Nov 9, 2022

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: f65b647
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit 9437d4f into aws:main Nov 9, 2022
@mergify
Copy link
Contributor

mergify bot commented Nov 9, 2022

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@fergusdixon fergusdixon deleted the fix/remove-explicit-ecs-deployment-type-for-circuit-breaker branch November 14, 2022 10:45
rix0rrr added a commit that referenced this pull request Nov 18, 2022
The behavior introduced in #22467 was too eager, changing the no-feature
flag behavior for all services, not just services with a circuit breaker.
mergify bot pushed a commit that referenced this pull request Nov 18, 2022
The behavior introduced in #22467 was too eager, changing the no-feature flag behavior for all services, not just services with a circuit breaker.


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
TheRealAmazonKendra pushed a commit that referenced this pull request Nov 18, 2022
The behavior introduced in #22467 was too eager, changing the no-feature flag behavior for all services, not just services with a circuit breaker.


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@shellscape
Copy link

@rix0rrr is there a means to programmatically set feature flags? we've got around 20 fargate services going up, each with their own cdk.json and managing feature flags (specifically setting this one, since all of our services use set names) via cdk.json for each is a bit of a PITA. happy to open a new issue but wanted to ping first.

@corymhall
Copy link
Contributor

@shellscape yes you can set it programmatically using setContext

node.setContext('@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker', true);

This will apply the feature flag to all constructs within that scope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK bug This issue is a bug. effort/small Small work item – less than a day of effort p2
Projects
None yet
8 participants