Changes for OWLS-82011 to reflect introspector status in domain status

[ankedia]

Changes for OWLS-82011 to reflect introspector status in domain status in below conditions -

1. Missing image
2. Missing PV/C mount
3. Introspector job timeout (deadline exceeded)

[russgold]

What do these statuses indicate? I'd have thought that once you selected a status, that indicated what had to be updated in the domain status - and yet this seems to do a fair bit of additional processing. Can you explain your approach?

[ankedia]

Unfortunately Kubernetes seem to have different way of dealing with various failures and reason/message for failure is captured in different json stanza/object.

1. The UNSCHEDULABLE status indicates that pod can't be scheduled and is in pending state (for e.g. due to missing pv/c). The reason/message for this condition is captured in pod conditions which is part of pod status.

2. The WAITING_NON_NULL_MESSAGE indicates that pod container is stuck in waiting state (for e.g. to due to image pull error). The reason/message for this is captured in waiting state object in container statuses which is part of pod status.

3. The PHASE_FAILED status indicates that pod has failed due to timeout (Deadline exceeded). In this case both conditions and container statuses in pod status are null (pls see below). The pod status phase is "Failed" and reason/message is captured directly as part of pod status.
   V1PodStatus:
   {
   conditions: null
   containerStatuses: [class V1ContainerStatus {
   containerID: docker://b823ffd3fe05d101b22529485ce9d946b652631487e8ce82e94f2e8ec1c2acb0
   image: model-in-image-1:v1
   imageID: docker://sha256:1cb49428989aec3007cd887d5b31cbf8da6fa0da30998e03d2ebec29ed4bf5ec
   lastState: class V1ContainerState {
   running: null
   terminated: null
   waiting: null
   }
   name: domain1-introspect-domain-job
   ready: false
   restartCount: 0
   started: null
   state: class V1ContainerState {
   running: class V1ContainerStateRunning {
   startedAt: 2020-07-23T23:07:08.000Z
   }
   terminated: null
   waiting: null
   }
   }]
   ephemeralContainerStatuses: null
   hostIP: null
   initContainerStatuses: null
   message: Pod was active on the node longer than the specified deadline
   nominatedNodeName: null
   phase: Failed
   podIP: 192.168.4.110
   podIPs: null
   qosClass: Burstable
   reason: DeadlineExceeded
   startTime: 2020-07-23T23:07:07.000Z
   }

4. There are other cases where pod/container is terminated due to error and reason/message is captured in terminated state in container statuses which is part of pod status. This is indicated by TERMINATED_ERROR_REASON.

Earlier logic was updating domain status unconditionally with reason/message from waiting state of container statuses which is part of pod status after receiving pod added/modified watch event.

Additionally, there's job watcher and pod watcher and both of them update domain status. We might have to decide which one gets higher priority or if we can update domain status only in pod watcher based on additional testing and results of integration tests.

[russgold]

I was hoping for a simple way to make this more readable, but I don't see one that is obvious. I may want to revisit in the future.

[russgold]

Is there some reason not to create this progressing state in the bringManagedServersUp method?

[ankedia]

There is no particular reason. I will make the change to create progressing state in bringManagedServersUp like this. Thanks.

  private static Step bringManagedServersUp(Step next) {
    return DomainStatusUpdater.createProgressingStep(MANAGED_SERVERS_STARTING_PROGRESS_REASON, true,
            new ManagedServersUpStep(next));
  }

[russgold]

This branch creates the FailedStep. Where is it being run?

[ankedia]

Thanks for catching this, I meant to run it with "delegate.runSteps" and missed it. I will change it like this =>

          delegate.runSteps(
                  DomainStatusUpdater.createFailedStep(
                          info, pod.getStatus().getReason(), pod.getStatus().getMessage(), null));

[russgold]

That suggests that you are missing a unit test. If you had the test, it would have caught it.

[ankedia]

Added unit test IntrospectionStatusTest.whenIntrospectorJobPodPhaseFailed_patchDomain() to cover this. Thanks.

[doxiao]

Looks good to me.
Can we add a unit test for the change where we terminate the fiber when the backofflimit is exceeded?

[ankedia]

Looks good to me.
Can we add a unit test for the change where we terminate the fiber when the backofflimit is exceeded?
Added unit test JobWatcherTest.whenWaitForReadyAppliedToTimedOutJobWithBackoffLimitExceeded_terminateWithException for backofflimit exceeded. Thanks