best practice for checking canary success in CI/CD

[iacou]

Is there any guidance on how to check for a successful canary rollout as part of a CI/CD stage?
I've seen the 'CI Example', on "https://docs.flagger.app/usage/how-it-works" essentially have to roll your own checks currently, which is a bit messy when you have lots of canaries to watch the status of.

how do you know that the canary resource you are waiting to go into 'progressing' is the helm chart you have just deployed?

need to cater for the fact that first time you deploy it never goes into 'progressing', just remains in 'initialized'

Seems like a lot of hand rolled code required...will there ever be a Flagger CLI command to help make this simpler?

[kingdonb]

I think the most straightforward answer is (not particularly straightforward, I guess) to load the Kubernetes client library and query the Canary status, read the conditions, and follow the transition. If you want to be sure your change is the one progressing, I'm not sure how you do that. When I implemented this into my app, I ran into all of the issues you suggested (I wanted the app itself to be aware of whether a Canary rollout was ongoing and report in an admin panel, what state it's in)

The app had built a Canary option into the helm chart, so it first checks for a Canary in the expected location (since it might not be enabled) and then, it reads status.conditions[0].reason, (I found this strategy was somewhat lacking as a proxy for full state-machine tracking canary transitions through their lifecycle, as you surmised there might be multiple canaries in-flight or one canary that interrupts another, and this might confuse status reporting mechanisms for CI/CD if they can't be reliably returned as "Superseded" or "Failed" status, which I'm not sure how to go about capturing.)

You can look for "Succeeded" which is the positive terminating state, or "Failed" as the other terminating state, for the garden path. I had intended to use this as a place to build manual gates for releases that would not get tangled up with or hold up CI. The CI doesn't need to be involved in deployment, that's Flux's job as the CD tool (was my disposition anyway, at the time).

Let the application trigger the approval gate when an admin user visits the deployment status page, and clicks "Approve." Do it all inside the app! There's no need to report on deploy status from CI, (maybe for your requirements it's not the case.)

Is there a simpler way, than talking directly to the Kubernetes API with a CLI or from your application using client libraries, I don't think so, but should there be? What would it look like?

I have an idea of what I think you're looking for (a CLI tool that can wait for rollouts?), I don't have an idea of how to implement it or a good suggestion for exactly how it should work, given constraints you've identified. It was very easy to implement this into my app in Ruby (I used k8s-client from Kontena), but if you don't have Ruby or Go in your pipeline, it might be harder.

[iacou]

interesting...i'm driving the checks in a less sophisticated way in bash with kubectl.
I've got the status check working good, but my issue now is there is a delay between a deployment being updated and flagger starting put the canary in intializing/progressing state. so my check returns good before the canary has even started. presumably just checking all canaries in a given namespace and waiting for all them to be in progressing state (before checking status again) is not an option, for canaries where an associated deployment resource has not been updated.

Is there any way to reduce the delay before flagger starts, or poke flagger to start progressing/initializing each (or all) canaries? This would be the CLI command I am looking for, I think. then a status check could be done afterwards.

[stefanprodan]

but my issue now is there is a delay between a deployment being updated and flagger starting put the canary in intializing/progressing state

Flagger has an internal routine that scans for new deployments every 15s, this may change in the future but even if we would use Kubernetes watchers there will still be a delay as Kubernetes events are not instantaneous.

[iacou]

is there any attribute that could be checked on across all canaries to confirm whether the associated deployment has been scanned?

Looking for something better than just putting a sleep 30 in my script...