Orchestra Multiple Canary's

[johnatowers]

Describe the feature

My development team at work uses flux/flagger for all of our deployments. The biggest problem we have is the fact that our application is a monolithic monorepo. We deploy our backend and frontend (sometimes multiple front ends to one backend) at the same time, with multiple canary's and AC tests and whatnot.

However, we want to the frontends to continue in canary analysis if the backend has successfully deployed. If the backend fails, it should fail the dependent canary's.

Proposed solution

• webhooks that only execute open canary success or canary failure
• some sort of field within canary object that allows for them to communicate with each other?

Any alternatives you've considered?

• manual gating. We do not want manual processes
• separate charts into multiple charts, that then use api requests to GitHub to bump the frontend chart versions in order to trigger the deployment process (current implementation).

Are there any other solutions? I spent a considerable amount of time looking at the gates, and they are close to being a fit but not quite due to lack of support for webhooks that only execute on success or only on failure. Also they can be convoluted. We don't want to have our actests sending requests to the flagger deployment if we can avoid it.

Looking for other suggestions, thanks!

[tmcg-gusto]

@johnatowers I'm actually implementing a similar thing right now. My goal is to make sure a set of Canaries stay roughly in sync, as the deployments they manage can come up at pretty different rates, and make sure they all roll back if one does.

My approach is a system outside of Flagger that takes webhooks from the various Canaries in a "group", waits for each to be ready for traffic ramp (considering them ready when the first confirm-traffic-increase hook comes in), and responds 200 when it has gotten a hook from each canary. It'll also use the event webhook to track failures and adjust the response to the rollback webhook accordingly. It also needs to use the confirm-promotion hook to make sure nothing promotes while others are still running.

Managing the grouping of Canaries is done by setting specific metadata in the webhook configs. It would be nice to have something like this built into flagger, but building it outside for now felt the most expedient.

[aryan9600]

while there are certainly a lot of use cases for this, we can't implement this in the current project. the codebase will need to be refactored significantly to natively support this. imo, the best way to enable something like this is to build a system that orchestrates Canary rollouts using webhooks to be alerted about events and perform actions when required.