Kubernetes Job not displayed in Rancher UI when job is created from CronJob with kubectl 1.19+ version

[gaktive]

Internal reference: SURE-3592
Reported in 2.5.9 and 2.6.2 on both Cluster Explorer and Manager

When manually triggering a job using the CLI and the following command:
kubectl create job <JOB_NAME> --from=cronjob/<CRONJOB> -n <NAMESPACE>
the job does successfully get created, but doesn't show up under the deployment pods list in the Rancher UI.

Looks like this is a reported issue per rancher/rancher#29606 for K8s v1.20.8,

#################### 

Github Details:
What kind of request is this (question/bug/enhancement/feature request):
This is a bug, most probably at the interface layer.
The API call to https://rancher_url/v3/project/c-CID:p-PID/pods?limit=-1&sort=name returns the job names but they are not displayed in the interface.

Rancher UI is hiding stuff that exists on the cluster may not be the best idea for observability. (Maybe displaying orphaned objects in the interface would be a better idea instead of hiding them under the rug?)

Steps to reproduce (least amount of steps as possible):

1. get any kubectl client that is at least 1.19 -- https://storage.googleapis.com/kubernetes-release/release/v1.19.0/bin/linux/amd64/kubectl
2. create a cronjob cronjob-test-ui -- https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#creating-a-cron-job
3. create a job from cronjob
   ./kubectl create job --from=cronjob/hello hello-manual-100
4. check Rancher hello CronJob UI

Result:
It won't display the hello-manual-100 Job under the hello CronJob as it does with an earlier kubectl client version

Update: from 2.6.2...

Cluster explorer shows the job where in the New 2.6.x not showing the Jobs even though it shows the job count as 3 but not showing the details. (shows empty pane)

Steps:

Created a cronjob in the pots namespace
⎈ rco-discovery-01  kubectl apply -f Test_CronJob.yaml -n pots

cronjob.batch/hello created

**

⎈ rco-discovery-01  kubectl get cj -n pots

NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE

hello */1 * * * * False 0 19s 67s

Created a job from the CronJob
⎈ rco-discovery-01  kubectl create job --from=cronjob/hello deehello -n pots

job.batch/deehello created

deehello Job shows up via kubectl
⎈ rco-discovery-01  kubectl get jobs -n pots

NAME COMPLETIONS DURATION AGE

deehello 1/1 3s 5m36s

hello-1634578440 1/1 4s 2m12s

hello-1634578500 1/1 4s 71s

hello-1634578560 1/1 3s 11s

deehello Job doesn’t show up in Rancher UI (see screenshot)

[aalves08]

Couldn't reproduce with the latest versions.

Also setup v2.6.2 rancher backend on Digital Ocean + checked out v2.6.2 rancher UI and wasn't able to reproduce this... Maybe I am doing something wrong... When creating the Job from the Cronjob I get a strange message, but the Job is indeed created but with a different name, while using the commands above mentioned

@nwmac any thoughts?

[aalves08]

We believe that this is caused by a version mismatch between the kubectl CLI client and server versions.

v2.6.2

From our testing, v2.6.4 appears to have this issue fixed (although there's still a mismatch) and there's an action item from yesterday's retro to handle this the backend team.

v2.6.4

@gaktive what do think? Shall we move it forward for testing and recommend that the client update's rancher to v2.6.4 once it's released? 🙏

[gaktive]

@aalves08 I'm not sure what the difference was in the most recent screenshots but I do see that cron jobs does have listings in your earlier comment so this looks good to pass for a reviewer. Obviously, there's additional backend work required to sort out the version mismatch but you have addressed the main concern.

[aalves08]

@gaktive the images are just highlighting the versions of kubectl CLI on versions 2.6.2 and 2.6.4 of rancher.
Don't know if the action item from the retro generated a ticket/issue which could be linked here.

Nevertheless, I'll be moving this forward with the recommendation that the complainer updates to 2.6.4 once we release it.

Let me know if any changes. Thanks!

[sgapanovich]

Couldn't reproduce the issue in 2.6.2. Tried with 3 clusters:

• client version: 1.19.7
• local: v1.21.3+k3s1
• rke1 downstream: v1.20.15
• rke1 downstream: v1.19.16

I was able to create jobs from a cronjob and they are visible on UI having 1.19.7 client and v1.19.16 & v1.20.15 servers

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.16", GitCommit:"e37e4ab4cc8dcda84f1344dda47a97bb1927d074", GitTreeState:"clean", BuildDate:"2021-10-27T16:20:18Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.15", GitCommit:"8f1e5bf0b9729a899b8df86249b56e2c74aebc55", GitTreeState:"clean", BuildDate:"2022-01-19T17:23:01Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

A job couldn't be created on the local cluster v1.21.3+k3s1 with v1.19.7 client since cronjobs have different apiVersion in 1.19.x (batch/v1beta1) vs 1.21.x (batch/v1)

---

Test Environment:

Rancher version: v2.6-head
Rancher cluster type: HA
Docker version: 20.10

local k8s: v1.22.6
client: v1.21.8

---

Testing:

Was able to create a job from a cronjob via kubectl command on the local cluster with the following setup

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.8", GitCommit:"4a3b558c52eb6995b3c5c1db5e54111bd0645a64", GitTreeState:"clean", BuildDate:"2021-12-15T14:52:11Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.6", GitCommit:"f59f5c2fda36e4036b49ec027e556a15456108f0", GitTreeState:"clean", BuildDate:"2022-01-19T17:26:47Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

@gaktive couldn't reproduce the issue in 2.6.2, verified the issue is not present in 2.6-head
please let me know if there is anything else needed before closing it (can't close an internal issue if it was not reproduced unless devs agree to close it)

[gaktive]

Pinged the original submitter with the update. Waiting to hear back if there are additional repro steps.

[ronhorton]

ronhorton said: PASS verified in v2.6.4-rc12

1. created cron job with CLI
2. watched cronjob to verify complete run
3. checked UI - hello cronjob listed
4. deleted cronjob via CLI
5. confirmed cronjob gone in UI