Retry NGINX provisioning on all errors

[sjberman]

Problem: Formerly, we would not retry provisioning if we got an error that a resource already exists. However, in certain cases, the timing was just right where back to back calls to CreateOrUpdate would result in the second call initially not finding the resource (as it was being created but didn't exist yet), and then when it decided to call Create, the resource now existed, and it would fail to update the resource to the new change.

Solution: Retry the CreateOrUpdate call no matter what error is returned. This ensures that if we happen to hit this quick succession scenario, the second update call would eventually succeed and not fail immediately, after the CreateOrUpdate function determined that the resource does exist and simply needs an update, not a Create.

Also added a return statement to safeguard against potential panics if an object is nil, due to a similar potential timing issue.

Testing: Multiple attempts at reproducing the issue, didn't see it occur anymore.

Closes #4326

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

Fix an issue where NginxProxy config might not be honored if applied at the same time as the Gateway.

[codecov]

Codecov Report

❌ Patch coverage is 14.28571% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.11%. Comparing base (a1afc9d) to head (42e660a).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/controller/provisioner/provisioner.go	7.69%	11 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4399      +/-   ##
==========================================
- Coverage   86.16%   86.11%   -0.06%     
==========================================
  Files         132      132              
  Lines       14376    14381       +5     
  Branches       35       35              
==========================================
- Hits        12387    12384       -3     
- Misses       1780     1786       +6     
- Partials      209      211       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bjee19]

nice fix! Might also be good to add a release note about it

[sjberman]

Might also be good to add a release note about it

Good call, meant to do that and forgot. Thanks.