we have no alerts for label_sync failing to run

[spiffxp]

/kind cleanup
/area label_sync

ref: #9054

While trying to deploy a new copy of the label_sync cronjob, I noticed that it had hung a few days ago. An invalid label description caused it to hang, and no more copies were run subsequently. How could we alert on this kind of situation?

I'm thinking of a pattern where logs are uploaded to GCS and consumed by testgrid just like all of our other maintenance jobs, and an alert is setup to fire if we see failure (or no success within a certain time)

[stevekuznetsov]

Can we migrate it to a periodic job to give it an entry in a infrastructure TestGrid view along with the commenters, etc? I'd image peribolos and other tools would join it there.

[fejta]

periodic job

We need to create a cluster key that targets a cluster with trusted credentials, and write automation to ensure that only sig-testing jobs can use it.

[spiffxp]

/milestone v1.13
We should be able to do this now, follow the model used by peribolos

[stevekuznetsov]

Unfortunately the test grid approach isn't useful for us and we have the same problem! Maybe we could have a Prometheus alert if we publish per-job failure metrics?

[spiffxp]

/milestone clear

I'm opting to punt this if the testgrid approach isn't good enough. Feel free to add back in to v1.14 if you're inclined to take the approach you suggested @stevekuznetsov

[stevekuznetsov]

You guys have the background to run it as a periodic now, so I think that should work ...

[spiffxp]

/milestone v1.14

OK, let's see if we can get to this

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[stevekuznetsov]

It's running as a periodic now after @cjwagner changed it so alerting on it in testgrid should not be hard?

[cjwagner]

It's running as a periodic now after @cjwagner changed it so alerting on it in testgrid should not be hard?

Already done:

test-infra/testgrid/config.yaml

Lines 2191 to 2194 in ae2330f

	- name: ci-test-infra-label-sync
	gcs_prefix: kubernetes-jenkins/logs/ci-test-infra-label-sync
	alert_stale_results_hours: 12
	num_failures_to_alert: 6 # Runs every 1h. Alert when it's been failing for 6 hours

We could probably tighten the alert window now.
/shrug