slack: periodic jobs missing URL

[howardjohn]

Since the format is the same for all job types I am guessing .Status.URL} is not set somehow?

[howardjohn]

Actually even prow.istio.io has no link, so I guess its not really slack reporting broken but something lower level

[howardjohn]

Timing aligns with 8c51a11 @cjwagner

[cjwagner]

I think we accidentally nuked the test-pods namespace (and in the process the deployment job itself) due to calling kubectl apply twice with the same pruning resource list. We may need to use server-side apply for everything or stop using this pruning pattern.
For now I've reverted the change and manually redeployed. Jobs seem to be starting now, e.g. https://prow.istio.io/view/gs/istio-prow/logs/ci-test-infra-branchprotector/1481082039586263040

[cjwagner]

I'm not sure why this change didn't cause the ProwJob CRD to be pruned: #3744
As far as I can tell it should have, but it didn't before or when I ran make deploy locally just now.

I'm kind of thinking this pruning logic may be more hazardous than it is worth.

[howardjohn]

Does k8s do anything like this or just plain kubectl apply? If so, does it handle removal of resources at all? I can imagine that needing to remove is pretty uncommon

…

On Tue, Jan 11, 2022, 6:09 PM Cole Wagner ***@***.***> wrote: I'm not sure why this change didn't cause the ProwJob CRD to be pruned: #3744 <#3744> As far as I can tell it should have, but it didn't before or when I ran make deploy locally just now. I'm kind of thinking this pruning logic may be more hazardous than it is worth. — Reply to this email directly, view it on GitHub <#3759 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXMODOGXJZGBJYZYOEDUVTPGPANCNFSM5LXTSHMQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[cjwagner]

Does k8s do anything like this or just plain kubectl apply? If so, does it handle removal of resources at all? I can imagine that needing to remove is pretty uncommon

K8s uses bazel to do a kubectl apply there is no resource removal. Needing to remove is pretty uncommon and is extremely easy.
The original motivation for the pruning is to help avoid tech debt by ensuring everything in the cluster is checked in. Its a good motivation, but I don't think we really have issues with folks manually applying to the cluster and not checking in the changes any more. I think this pruning mechanism is more often hurting us than helping us in its current form.

[howardjohn]

ok, removal sgtm if it's not giving value. If we do want something similar there are plenty of tools that provide similar behavior in a better way (or at least different - I don't have a ton of experience with them) such as kpt, helm, flux, etc. But no need to over engineer it

…

On Tue, Jan 11, 2022, 6:46 PM Cole Wagner ***@***.***> wrote: Does k8s do anything like this or just plain kubectl apply? If so, does it handle removal of resources at all? I can imagine that needing to remove is pretty uncommon K8s uses bazel to do a kubectl apply there is no resource removal. Needing to remove is pretty uncommon and is extremely easy. The original motivation for the pruning is to help avoid tech debt by ensuring everything in the cluster is checked in. Its a good motivation, but I don't think we really have issues with folks manually applying to the cluster and not checking in the changes any more. I think this pruning mechanism is more often hurting us than helping us in its current form. — Reply to this email directly, view it on GitHub <#3759 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXIHIR5SCFX5456OM6LUVTTQVANCNFSM5LXTSHMQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[ericvn]

Removal seems to have put the URL back, but now the job is failing with:

Test started yesterday at 7:54 PM failed after 15m0s. (more info)
The pod could not start because it could not mount the volume "oauth": secret "oauth-token" not found

[ericvn]

The ci-prow-autobump is also failing with the missing token.

[cjwagner]

Everything should be fixed now.
Here is a cover for the pitfall: #3764

/assign
/close

[istio-testing]

@cjwagner: Closing this issue.

In response to this:

Everything should be fixed now.
Here is a cover for the pitfall: #3764

/assign
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.