Dynamic Per Resource Timeouts

[tommyp1ckles]

To prevent k8s resource types that take a long time to sync from prematurely crashing agent Pods upon init. Adding per resource timeouts that do not timeout sync unless the watcher has exceeded the timeout period after the last informer event of that type to be received.
That is, each watcher records the time of the last event, per API resource type. If the K8s-Synced-Timeout is reached, only timeout if event of that type was not received during the timeout period.

As well, to facilitate diagnosing such issues, adding more labels to EventsTS metrics and refactoring all uses of this metric into a single GaugeVec.

Fixes: #18776

[tommyp1ckles]

/test

[tommyp1ckles]

nm about the GKE tests - missed the memo about those not working. Everything else seems to be ok test wise.

[tommyp1ckles]

/test

[tommyp1ckles]

I recently discovered the below handy git tricks, which might help for addressing review comments

# commit changes linked to original commit, auto complete can be used for easier selection
git commit --fixup <the original commit> 

# rebase with autosquash linked commits for clean commit history.
git rebase master -i --autosquash

Thanks yeah, I've been doing a lot of interactive rebasing. I just ended up mixing changes into commits which made reintegrating them back into the original very trick as they would depend on changes made later in somewhat unrelated commits.
Definitely need to try to keep my commits smaller and more self contained in the future...

[tommyp1ckles]

@christarazi I'll squash down the last commit once you're done reviewing.

[nathanjsweet]

K8s API changes look good.

[tommyp1ckles]

@tklauser can I get another review when you have a moment.

[tklauser]

/test

Job 'Cilium-PR-K8s-1.16-kernel-4.9' failed:

Click to show.

Test Name

K8sPolicyTest Basic Test checks ICMP policies

Failure Output

FAIL: Expected

If it is a flake and a GitHub issue doesn't already exist to track it, comment /mlh new-flake Cilium-PR-K8s-1.16-kernel-4.9 so I can create one.

Job 'Cilium-PR-K8s-1.23-kernel-net-next' failed:

Click to show.

Test Name

K8sPolicyTest Basic Test checks ICMP policies

Failure Output

FAIL: Expected

If it is a flake and a GitHub issue doesn't already exist to track it, comment /mlh new-flake Cilium-PR-K8s-1.23-kernel-net-next so I can create one.

[tommyp1ckles]

/test

Job 'Cilium-PR-K8s-1.16-kernel-4.9' hit: #20217 (91.88% similarity)

[tommyp1ckles]

/test

[tommyp1ckles]

ICMP test keeps failing, rebasing to see if recent datapath changes fix that.

[tommyp1ckles]

1.23-kernel-net-next failing seems unrelated to branch

[tommyp1ckles]

/test

[tklauser]

@tommyp1ckles FYI, you can re-trigger individual tests using the trigger phrases mentioned in brackets next to their name. For example, to re-trigger just k8s-1.23-kernel-net-next you can use /test-1.23-net-next. This will avoid running all other tests and potentially hitting other flakes.

[tommyp1ckles]

/test-1.23-net-next