Duplicates conditions added to a status list

[teabot]

JOSDK v4.9.7

I have a resource whose status includes a list of conditions. This is serialised as an array. In my reconcile method I update the conditions list like so:

shouldPatchStatus = false
for each condition type {
  var previous = get previous condition from list
  var desired = compute new condition state
  if (previous != desired) {
    if (previous == null) {
      add desired to list
    } else if (desired == null) {
      remove previous from list
    } else {
      apply state of desired to previous (update in place)
    }
    shouldPatchStatus = true
  } 
}
return shouldPatchStatus ? UpdateControl.patchStatus : UpdateControl.noUpdate

In practice I find that I get duplicate entries appearing in the conditions list, usually differing only by lastTransitionTime with a delta of a few ms.

When I look at logs, I see the two instances of the condition apparently added by two reconciliation cycles, ms apart. One follows the other (no overlap), but they can take place on different threads. They both seem to operate on the same resource version. The logs look something like:

1  Reconcile resource 42
2. Add condition
3. PatchStatus
4. Reconcile resource 42
5. Add condition
6. PatchStatus

As I undertand it:

1. JOSDK does not perform multiple concurrent reconciles on the same resource, so if this is true, I do not see how there could be concurrent updates to the same list. The logs appear sequential also - suggesting that the reconciliations are not concurrent.
2. With SSA, array updates are atomic, so in the event of a race, I might either see a conflict, or "last writer wins", but inexplicably I see two elements for a given type persisted in the status.

Can you suggest why this may be happening and how I might guard against it?

[csviri]

Hi @teabot , yeah, it's probably not a concurrency issue, I don't recall any tbh. I would suggest to log the requests and responses, and see if something buils up from that, there are somethimes funky issues with SSA.

[teabot]

@csviri I've spent a few hours trying to get the requests/responses, but I do not seem to be able to get a working log config that emits anything bar my own debug and a tiny amount of JOSDK debug. Can you suggest what I might need? Here is what I have tried:

    config.put("quarkus.log.category.\"io.javaoperatorsdk\".level", "TRACE");
    config.put("quarkus.log.category.\"io.fabric8.kubernetes.client\".level", "TRACE");
    config.put("quarkus.log.category.\"io.fabric8.kubernetes.client.dsl\".level", "TRACE");

    config.put("quarkus.log.category.\"io.fabric8.kubernetes.client.http.vertx\".level", "TRACE");
    config.put("quarkus.log.category.\"io.vertx.core.http\".level", "TRACE");
    config.put("quarkus.log.category.\"io.vertx.core.http.impl\".level", "TRACE");
    config.put("quarkus.log.category.\"io.vertx.core.net.impl\".level", "TRACE");
    config.put("quarkus.log.category.\"io.netty.handler.logging\".level", "TRACE");
    config.put("quarkus.log.category.\"io.vertx.core.net.impl.ConnectionBase\".level", "TRACE");

    config.put("quarkus.log.min-level", "TRACE");
    config.put("quarkus.log.level", "TRACE");
    config.put("quarkus.log.console.level", "TRACE");