Detect when a config map doesn't exist on a running pod for > X time, fail

[smarterclayton]

A common error mode is forgetting to create a dependent config map. We should do a slightly better job of conveying that condition back (either as a job failure or a log message)

[stevekuznetsov]

We should? Or k8s should?

[stevekuznetsov]

Configmaps we have issues with are:

• CI Operator config
• any of the assorted configmaps for the templates

The first is fixed now with an updated approach to uploading those
The second should be mitigated from copy-pasta errors for extant templates by generating the Prow config. For net-new templates, agreed this would be an issue.

[smarterclayton]

I think in this case the ci-operator "wait for pod success" job could easily report the error that k8s puts on the container when the config doesn't exist after a period of time.

[stevekuznetsov]

Yes, and with secrets too