Add Prometheus metrics to count ANP and ACNP Status updates

[antoninbas]

Too frequent Status updates could generate too many versions of the CRD,
that would need to be stored in etcd until the next compaction by
kube-apiserver. Too many updates could also cause fragmentation of the
database. It is useful to have access to the number of updates over time
in production clusters.

[jianjuns]

I am ok with the change. Feel people might not understand what such metrics are for, but probably no harm either.

[abhiraut]

does update to the Status sub-resource generate a new version? i thought the generation count remained same.
but maybe it still updates the resourceVersion even if it does not generate a new resource.

[antoninbas]

does update to the Status sub-resource generate a new version? i thought the generation count remained same.
but maybe it still updates the resourceVersion even if it does not generate a new resource.

I believe ResourceVersion is incremented and there is definitely a write to etcd, but we can wait for @tnqn to confirm and comment on the usefulness of this PR

[tnqn]

does update to the Status sub-resource generate a new version? i thought the generation count remained same.
but maybe it still updates the resourceVersion even if it does not generate a new resource.

I believe ResourceVersion is incremented and there is definitely a write to etcd, but we can wait for @tnqn to confirm and comment on the usefulness of this PR

Yes, generation is different from resourceVersion. Any update to any field will lead to a new version and a write to etcd. Although the original issue is not caused by status update, this metrics can help rule it out quickly and generally help us understand the overhead of status manager so I'm good to add them.

[tnqn]

LGTM

[antoninbas]

Thanks for the feedback @tnqn

[antoninbas]

/test-all

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (main@57dcaec). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1801   +/-   ##
=======================================
  Coverage        ?   26.49%           
=======================================
  Files           ?      179           
  Lines           ?    15169           
  Branches        ?        0           
=======================================
  Hits            ?     4019           
  Misses          ?    10625           
  Partials        ?      525           

Flag	Coverage Δ
e2e-tests	26.49% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.