feat(prom): Add prometheus metrics reset support #5287

[victorboissiere]

Add support for Prometheus metrics reset schedule to fix #5287

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• Optional. My organization is added to USERS.md.
• I have signed off all my commits as required by DCO
• My build is green (troubleshooting builds).

[codecov-io]

Codecov Report

Merging #5304 (1b889fe) into master (e02b864) will decrease coverage by 0.11%.
The diff coverage is 82.60%.

@@            Coverage Diff             @@
##           master    #5304      +/-   ##
==========================================
- Coverage   41.08%   40.97%   -0.12%     
==========================================
  Files         136      137       +1     
  Lines       18365    18585     +220     
==========================================
+ Hits         7546     7615      +69     
- Misses       9741     9883     +142     
- Partials     1078     1087       +9     

Impacted Files	Coverage Δ
controller/appcontroller.go	53.55% <50.00%> (-0.13%)	⬇️
controller/metrics/metrics.go	76.42% <89.47%> (+2.38%)	⬆️
util/cache/inmemory.go	35.13% <0.00%> (-14.87%)	⬇️
server/application/broadcaster.go	71.42% <0.00%> (-7.15%)	⬇️
controller/state.go	69.64% <0.00%> (-5.65%)	⬇️
util/cache/appstate/cache.go	62.50% <0.00%> (-4.17%)	⬇️
util/cache/cache.go	70.21% <0.00%> (-3.13%)	⬇️
pkg/apis/application/v1alpha1/types.go	55.64% <0.00%> (-0.80%)	⬇️
server/application/application.go	28.37% <0.00%> (-0.70%)	⬇️
reposerver/repository/repository.go	62.41% <0.00%> (-0.19%)	⬇️
... and 7 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e02b864...1b889fe. Read the comment docs.

[jessesuen]

Is there a use case for using a cron expression over something simpler such as an interval? We're thinking to make this simpler to use, as well as more consistent with our other controller flags (e.g. resync period, cache durations) and control this using an interval instead.

[victorboissiere]

@jessesuen, the only use case would be to make it explicit when the reset schedule runs.
But you're absolutely right. I haven't thought of using an interval. I will make the changes.
To keep it consistent with other flags, I propose to use --metrics-cache-expiration.

[victorboissiere]

@jessesuen I've updated the PR with the changes

[alexmt]

Thank you @victorboissiere . LGTM