move pod label update to goroutine with retries

[oseoin]

Proposed changes

Behaviour when can't get pod after max attempts:

E1222 10:21:35.852087       1 main.go:809] Error getting pod on attempt 1 of 10: test error
E1222 10:21:40.865891       1 main.go:809] Error getting pod on attempt 2 of 10: test error
E1222 10:21:45.873099       1 main.go:809] Error getting pod on attempt 3 of 10: test error
E1222 10:21:50.879271       1 main.go:809] Error getting pod on attempt 4 of 10: test error
E1222 10:21:55.894960       1 main.go:809] Error getting pod on attempt 5 of 10: test error
E1222 10:22:00.908234       1 main.go:809] Error getting pod on attempt 6 of 10: test error
I1222 10:22:03.069004       1 leaderelection.go:260] successfully acquired lease default/nginx-ingress-leader
E1222 10:22:05.919809       1 main.go:809] Error getting pod on attempt 7 of 10: test error
E1222 10:22:10.930658       1 main.go:809] Error getting pod on attempt 8 of 10: test error
E1222 10:22:15.940513       1 main.go:809] Error getting pod on attempt 9 of 10: test error
E1222 10:22:20.955110       1 main.go:809] Error getting pod on attempt 10 of 10: test error
E1222 10:22:20.955185       1 main.go:838] Failed to update pod labels after 10 attempts

Behaviour when can't get pod for first N attempts and then succeeds:

E1222 10:22:58.249082       1 main.go:809] Error getting pod on attempt 1 of 10: test error
E1222 10:23:03.258668       1 main.go:809] Error getting pod on attempt 2 of 10: test error
E1222 10:23:08.266618       1 main.go:809] Error getting pod on attempt 3 of 10: test error
E1222 10:23:13.287607       1 main.go:809] Error getting pod on attempt 4 of 10: test error
E1222 10:23:18.290680       1 main.go:809] Error getting pod on attempt 5 of 10: test error
I1222 10:23:23.313543       1 main.go:833] Pod label updated: my-release-nginx-ingress-controller-4db48

Behaviour when can't update pod for first N attempts and then succeeds:

E1222 10:24:08.009645       1 main.go:829] Error updating pod with labels on attempt 1 of 10: test error
E1222 10:24:13.035016       1 main.go:829] Error updating pod with labels on attempt 2 of 10: test error
I1222 10:24:18.058095       1 main.go:833] Pod label updated: my-release-nginx-ingress-controller-2brmb

Confirmed that labels are updated correctly after failures.

This also takes the update out of the hot path so should slightly reduce startup time.

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

[codecov]

Codecov Report

Attention: 32 lines in your changes are missing coverage. Please review.

Comparison is base (6aca0be) 52.13% compared to head (a1551e2) 52.08%.

Files	Patch %	Lines
cmd/nginx-ingress/main.go	0.00%	32 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4835      +/-   ##
==========================================
- Coverage   52.13%   52.08%   -0.05%     
==========================================
  Files          60       60              
  Lines       17368    17376       +8     
==========================================
- Hits         9054     9050       -4     
- Misses       8000     8010      +10     
- Partials      314      316       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.