[Bug]: `wait_for_idle` does not wait for podspec charms when deployed along sidecar charms

[DnPlas]

Description

It seems like when deploying sidecar and podspec charms, wait_for_idle does not wait for podspec charms to be active and idle.

In this case for instance, the training-operator is a sidecar charm and the other two are podspec.

test_pod_spec_and_sidecar.py::test_deploy_pod_spec_chamrs PASSED
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Model                           Controller  Cloud/Region        Version  SLA          Timestamp
test-pod-spec-and-sidecar-61ew  uk8s        microk8s/localhost  2.9.42   unsupported  13:46:41Z

App                Version                Status   Scale  Charm              Channel         Rev  Address         Exposed  Message
argo-server        res:oci-image@576d038  waiting      1  argo-server        3.3/stable      185                  no       
minio              res:oci-image@1755999  waiting      1  minio              ckf-1.7/stable  186  10.152.183.89   no       
training-operator                         active       1  training-operator  1.6/stable      215  10.152.183.191  no       

Unit                  Workload  Agent  Address     Ports              Message
argo-server/0*        waiting   idle   10.1.15.22  2746/TCP           waiting for container
minio/0*              waiting   idle   10.1.15.23  9000/TCP,9001/TCP  waiting for container
training-operator/0*  active    idle   10.1.15.12      

This behaviour is not observed in a model with only podspec or only sidecars. Also, it appears to not happen when all units (podspec and sidecar) are active and idle before the timeout runs out.

An example in CI

Another much more complex example is:
I have this PR, where I'm interested in running integration tests. The first step is to build and deploy 10+ charms from this bundle definition. The second step is to wait for idle.
Eventually the execution continues past the wait_for_idle call, but if I check the juju status, some of the units are still in a waiting status.

Workaround

Use tenacity to iterate over all podspec Deployments and assert the replicas are ready and available. See here for more info.

Urgency

Annoying bug in our test suite

Python-libjuju version

juju==2.9.42.4

Juju version

2.9.42-ubuntu-amd64

Reproduce / Test

Use the files in: https://gist.github.com/DnPlas/7090b044f2d52d9666bdac284a73a776

1. tox -ve combined
2. juju status --watch 5s
3. Observe the training-operator being active and idle, observe other charms in waiting status
4. Observe once the training-operator is active and idle (and the timeout runs out) the test case passes

[cderici]

Initial triage complete:

While:

App                Version                Status  Scale  Charm              Channel         Rev  Address         Exposed  Message
argo-server        res:oci-image@576d038  active      1  argo-server        3.3/stable      185                  no
minio              res:oci-image@1755999  active      1  minio              ckf-1.7/stable  186  10.152.183.52   no

Unit                  Workload  Agent  Address      Ports              Message
argo-server/0*        active    idle   10.1.49.100  2746/TCP
minio/0*              active    idle   10.1.49.99   9000/TCP,9001/TCP

Running:

>>> from juju import model;m = model.Model(); await m.connect();
>>> await m.deploy("training-operator", channel="1.6/stable", trust=True)
<Application entity_id="training-operator">
>>> await m.wait_for_idle(status="active", raise_on_blocked=False, raise_on_error=True, timeout=300)
>>>

A wait_for_idle call correctly waits until the training-operator is up and ready to go:

Now trying the exact setup from the reproducer, the training-operator is there, and we add minio and argo-server:

App                Version  Status  Scale  Charm              Channel     Rev  Address         Exposed  Message
training-operator           active      1  training-operator  1.6/stable  215  10.152.183.118  no

Unit                  Workload  Agent  Address      Ports  Message
training-operator/0*  active    idle   10.1.49.101

>>> from juju import model;m = model.Model(); await m.connect();
>>> await m.deploy("minio", channel="ckf-1.7/stable");await m.deploy("argo-server", channel="3.3/stable")
<Application entity_id="minio">
<Application entity_id="argo-server">
>>> await m.wait_for_idle(status="active", raise_on_blocked=False, raise_on_error=True, timeout=300)
>>>

The wait_for_idle call prematurely returns as reported, while the minio workload status is "waiting", despite that we have status="active" parameter set:

App                Version                Status   Scale  Charm              Channel         Rev  Address         Exposed  Message
argo-server        res:oci-image@576d038  active       1  argo-server        3.3/stable      185                  no
minio              res:oci-image@1755999  waiting    1/0  minio              ckf-1.7/stable  186  10.152.183.52   no       waiting for container
training-operator                         active       1  training-operator  1.6/stable      215  10.152.183.118  no

Unit                  Workload  Agent  Address      Ports     Message
argo-server/1*        active    idle   10.1.49.85   2746/TCP
minio/1*              waiting   idle                          waiting for container
training-operator/0*  active    idle   10.1.49.101

Reason seems to be that the unit is not updated in time by the AllWatcher events, therefore we see the incorrect unit workload status:

<Unit entity_id="minio/1">
(Pdb) unit.agent_status
'idle'
(Pdb) unit.workload_status
'active'

reported incorrectly as active, which is a convoluted topic when it comes to caas models, discussed in detail in #849.

I'd have to check the FullStatus, but a patch for this issue here might be similar to the one in #849, where we get the most up-to-date status for the unit from the API before making a decision. Alternatively, we could compare that status with the up-to-date application status we already have, which is correctly set to waiting:

(Pdb) app_status
'waiting'

The most significant solution to all of this is to fine tune the AllWatcher delta updates to make them more reliable (or maybe to trigger updates at every iteration, not sure, need to investigate), but this issue in particular should be easy to patch quickly.

[jameinel]

... A wait_for_idle call correctly waits until the training-operator is up and

ready to go: Now trying the exact setup from the reproducer, the training-operator is there, and we add minio and argo-server: App Version Status Scale Charm Channel Rev Address Exposed Message training-operator active 1 training-operator 1.6/stable 215 10.152.183.118 no Unit Workload Agent Address Ports Message training-operator/0* active idle 10.1.49.101 >>> from juju import model;m = model.Model(); await m.connect();>>> await m.deploy("minio", channel="ckf-1.7/stable");await m.deploy("argo-server", channel="3.3/stable")<Application entity_id="minio"><Application entity_id="argo-server">>>> await m.wait_for_idle(status="active", raise_on_blocked=False, raise_on_error=True, timeout=300)>>> The wait_for_idle call prematurely returns as reported, while the minio workload status is "waiting", despite that we have status="active" parameter set: App Version Status Scale Charm Channel Rev Address Exposed Message argo-server ***@***.*** active 1 argo-server 3.3/stable 185 no minio ***@***.*** waiting 1/0 minio ckf-1.7/stable 186 10.152.183.52 no waiting for container training-operator active 1 training-operator 1.6/stable 215 10.152.183.118 no Unit Workload Agent Address Ports Message argo-server/1* active idle 10.1.49.85 2746/TCP minio/1* waiting idle waiting for container training-operator/0* active idle 10.1.49.101 *Reason* seems to be that the unit is not updated in time by the AllWatcher events, therefore we see the incorrect unit workload status:

So is the issue that there *is* a brief period of time where everything is active and idle, and it's just that shortly thereafter another hook is fired which removes it from active and idle? eg if the events are: 1. argo-server/1, waiting 2. minio/1, waiting 3. argo-server/1, active idle 4. minio/1, active idle 5. minio/1, waiting idle While at (5) it is true that minio is waiting, at (4) both argo-server and minio reported that they were active. Or is the issue that while the units are still initializing (before they get deployed at all) they report as active,idle? Or that argo-server actually comes up active and idle before minio gets a unit deployed far enough to even be seen? (I would have expected that deploy wouldn't return until at least the unit is represented in the model, though perhaps its events haven't propagated to the AllWatcher yet?) Inside juju itself, we actually use a slightly different model, where you say "watch for X, then modify X" rather than "modify X, watch for X", because the latter model is always racy (do you see your mutation or not is indeterminate if you watch after you make the request.) I've also considered threading through things like transaction requests into tokens so that you could "make this change to the model, get an identifier back for it, and then wait on this other activities thread until we see the change that we made" so that you know you can be in sync with the things that you just did. It would require some invasive changes on the Juju side. If the issue is just that deploy/deploy/wait, may or may not see the second application show up, would there be some other way to indicate *what* you are waiting for rather than "just waiting for an arbitrary state". eg "wait_for(app_a==idle, app_b==idle)" [not with that exact syntax] would be much clearer that if app_b isn't there, you want to keep waiting until you see it, and then wait until you see it become idle. John =:->

…

<Unit entity_id="minio/1"> (Pdb) unit.agent_status'idle' (Pdb) unit.workload_status'active' reported incorrectly as active, which is a convoluted topic when it comes to caas models, discussed in detail in #849 <#849>. I'd have to check the FullStatus, but a patch for this issue here might be similar to the one in #849 <#849>, where we get the most up-to-date status for the unit from the API before making a decision. Alternatively, we could compare that status with the up-to-date application status we already have, which is correctly set to waiting: (Pdb) app_status'waiting' The most significant solution to all of this is to fine tune the AllWatcher delta updates to make them more reliable (or maybe to trigger updates at every iteration, not sure, need to investigate), but this issue in particular should be easy to patch quickly.

[cderici]

@jameinel I don't believe this is a race issue. I have to take a closer look to confirm, but I bet this is about the operator_status again, juju/juju#15588, the status of an application is determined/reported somewhat differently for caas and iaas models. I'll update when I have more details from the Juju's side.

If the issue is just that deploy/deploy/wait, may or may not see the second
application show up, would there be some other way to indicate what you
are waiting for rather than "just waiting for an arbitrary state". eg
"wait_for(app_a==idle, app_b==idle)" [not with that exact syntax] would be
much clearer that if app_b isn't there, you want to keep waiting until you
see it, and then wait until you see it become idle.

This is sort of implicit in wait_for_idle. You can't specify status per app, however, when you run model.wait_for_idle(status='active') it means you're waiting on all the units of all the applications in the model to have their workload_status to be active. You can also specify a subset of apps, like model.wait_for_idle([app_a, app_b], status='active')