feat(konnect): allow configuring license polling periods

[czeslavo]

What this PR does / why we need it:

Adds --konnect-initial-license-polling-period and --konnect-license-polling-period CLI flags that allow configuring periods at which we poll license from Konnect.

Changes the license.Agent's behavior so that if it fails to fetch a license at the first attempt, it will retry with the initial period (which by default is 1m) instead of the regular period (which is 12h). It will make it possible to perform smoother upgrades from free to higher Konnect tiers with no KIC restart required.

Which issue this PR fixes:

Closes #4147.

Special notes for your reviewer:

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

• the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

[team-k8s-bot]

E2E (targeted) tests were started at https://github.com/Kong/kubernetes-ingress-controller/actions/runs/5268606896

[codecov]

Codecov Report

Patch coverage: 91.9% and project coverage change: +0.4 🎉

Comparison is base (c780e7f) 60.5% compared to head (ea5bbd6) 61.0%.

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #4178     +/-   ##
=======================================
+ Coverage   60.5%   61.0%   +0.4%     
=======================================
  Files        150     150             
  Lines      16822   16897     +75     
=======================================
+ Hits       10194   10320    +126     
+ Misses      5984    5931     -53     
- Partials     644     646      +2     

Impacted Files	Coverage Δ
internal/adminapi/konnect.go	0.0% <ø> (ø)
internal/manager/run.go	50.0% <0.0%> (-1.0%)	⬇️
internal/license/agent.go	89.7% <94.3%> (+26.8%)	⬆️
internal/dataplane/parser/parser.go	76.4% <100.0%> (+0.2%)	⬆️
internal/konnect/license/client.go	82.9% <100.0%> (+82.9%)	⬆️
internal/manager/config.go	94.9% <100.0%> (+<0.1%)	⬆️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[rainest]

The upstream endpoint apparently does weird things that it kinda shouldn't re 404s, but it looks like it was already kinda doing that. It's not clear whether the existing 404 case should have ever occurred in practice, but it was in the code, so 🤷

Main change request is to treat that as not an error, since it's expected, so we shouldn't make that the odd error out that's actually control flow. Rest is mostly comment/log clarification.

[czeslavo]

The upstream endpoint apparently does weird things that it kinda shouldn't re 404s, but it looks like it was already kinda doing that. It's not clear whether the existing 404 case should have ever occurred in practice, but it was in the code, so 🤷

I was coding this following the GET /licenses behavior that the upstream PR is expected to introduce. I agree that 404 might be weird, but as I can see it's already there even without the changes in the linked PR (i.e. it returns 404 when it couldn't get the license singleton in Koko).

To address the concerns about the weirdness of 404 being handled in our code differently based on the error type down the stack, I refactored the license.Client to return mo.Option[KonnectLicense], err) which should better reflect the reality. It tells the caller that the license might be missing hence mo.Option (which is the case when we get 404). I also added tests there so that it's clear what's the expected behavior. PTAL 🙏

[github-actions]

The backport to release/2.10.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-release/2.10.x release/2.10.x
# Navigate to the new working tree
cd .worktrees/backport-release/2.10.x
# Create a new branch
git switch --create backport-4178-to-release/2.10.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 991908b2e489add251064425e3879b5145a66dfa
# Push it to GitHub
git push --set-upstream origin backport-4178-to-release/2.10.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-release/2.10.x

Then, create a pull request where the base branch is release/2.10.x and the compare/head branch is backport-4178-to-release/2.10.x.