fix(manager): set InitCacheSyncDuration to 5s by default and allow it…

[backjo]

… to be configured via cli

What this PR does / why we need it:
This sets the default value for InitCacheSyncDuration to 5 seconds - as it was in #2249 - to avoid 'partial' writes to the proxy configuration before the Kubernetes objects have a had a chance to sync. This default value appears to have been inadvertently changed as part of #4101 . Additionally, this PR exposes the value for configuration by end-users.

Which issue this PR fixes:

Closes #5175

Special notes for your reviewer:

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

• the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

[CLAassistant]

All committers have signed the CLA.

[rainest]

Does this actually need to be configurable, or is fixing the regression sufficient? We'd left it inaccessible originally because we weren't sure if cases where the resource count was large enough where population did exceed 5s were likely in practice.

It's unfortunately not an intuitive setting to understand if you aren't familiar with the implementation internals, so it felt like it'd likely be an obscure knob where nobody knows when/if they should tweak it.

That's not a huge reason to avoid adding it, but if you have encountered a sufficiently large config set to need it and have logs or other indicators that'd show when users need to increase it, that'd be useful for docs.

[backjo]

Does this actually need to be configurable, or is fixing the regression sufficient? We'd left it inaccessible originally because we weren't sure if cases where the resource count was large enough where population did exceed 5s were likely in practice.

It's unfortunately not an intuitive setting to understand if you aren't familiar with the implementation internals, so it felt like it'd likely be an obscure knob where nobody knows when/if they should tweak it.

That's not a huge reason to avoid adding it, but if you have encountered a sufficiently large config set to need it and have logs or other indicators that'd show when users need to increase it, that'd be useful for docs.

Yeah, I decided to add it because from the logs we put in, we felt a little too close for comfort (~4ish seconds) on the time it took for the initial configuration to be loaded - especially given that a partial configuration results in 4XX errors all over the place for us. I totally agree it's a non-intuitive setting if one is unfamiliar with the implementation details - and ideally a fix per #2315 would get rid of the need for it altogether - but given that it's the current lever used to solve the cache sync problem and the latency may vary depending on the size / performance of the cluster it's being run on, I think it makes sense to expose for operators to configure.

[czeslavo]

Added backport release/2.12.x to have it ported to the LTS.

[mlavacca]

Looks good to me, apart from a minor nit.

[rainest]

Blah, forgot that you can't quite suggest doc changes there from the PR due to the need for make generate to copy those elsewhere.

[team-k8s-bot]

The backport to release/2.12.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-release/2.12.x release/2.12.x
# Navigate to the new working tree
cd .worktrees/backport-release/2.12.x
# Create a new branch
git switch --create backport-5238-to-release/2.12.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 17372534e2caa29742d1a9a723f9a3c6d1906069
# Push it to GitHub
git push --set-upstream origin backport-5238-to-release/2.12.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-release/2.12.x

Then, create a pull request where the base branch is release/2.12.x and the compare/head branch is backport-5238-to-release/2.12.x.

[piotrwielgolaski-tomtom]

when can we hope to get this in 3.x line? getting partial configuration is very dangerous

[team-k8s-bot]

The backport to release/3.0.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-release/3.0.x release/3.0.x
# Navigate to the new working tree
cd .worktrees/backport-release/3.0.x
# Create a new branch
git switch --create backport-5238-to-release/3.0.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 17372534e2caa29742d1a9a723f9a3c6d1906069
# Push it to GitHub
git push --set-upstream origin backport-5238-to-release/3.0.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-release/3.0.x

Then, create a pull request where the base branch is release/3.0.x and the compare/head branch is backport-5238-to-release/3.0.x.