Add support for reconcile policy

[matthchr]

This closes #1633.

Adds support for a new reconcile-policy annotation that allows users to
prevent all modifications to Azure resources (PUT, DELETE), or just all
DELETE's.

If applicable:

• this PR contains documentation
• this PR contains tests

[jonnylangefeld]

Thanks for coming around so quickly with this PR!

[jonnylangefeld]

However, this also makes so that as soon as the serviceoperator.azure.com/reconcile-policy annotation is present we skip, no matter the string value. That also makes the verb skip unnecessary (as if it doesn't matter if I put skip or sk1p or piks in there). I think this has great implications that as soon as the annotation is present, the operator skips reconcile.

That is fine, I would just be more explicit in the docs about that. Essentially there's three states:

1. I don't set anything and the operator reconciles
2. I set the serviceoperator.azure.com/reconcile-policy annotation with any value and the operator doesn't reconcile.
3. I set serviceoperator.azure.com/reconcile-policy: skip-delete and the operator reconciles but skips delete.

Given the name of the annotation however, I probably wouldn't expect that the operator skips reconciling as soon as the annotation is set. I'd probably expect that if the annotation was called serviceoperator.azure.com/skip-reconcile. But since it's called serviceoperator.azure.com/reconcile-policy, I'd probably expect that I have to set the correct policy to achieve the behavior I want (either skip or skip-delete).

[matthchr]

Agree that an empty reconcile-policy doing skip probably doesn't make sense. I'm going to change this behavior and default to manage (aka behave normally). That seems to be the standard behavior if you get an annotation like this "wrong".

If we really want to enforce that people don't set an incorrect value here we could prevent it at the webhook level (which has the advantage of being able to actually tell them they did it wrong as opposed to just doing something they didn't expect), but that seems overkill for now. We could investigate doing that in the future if there's actual desire for it from users.

[codecov-commenter]

Codecov Report

Merging #2060 (26f2ce0) into main (ed772b2) will increase coverage by 0.02%.
The diff coverage is 72.25%.

@@            Coverage Diff             @@
##             main    #2060      +/-   ##
==========================================
+ Coverage   56.67%   56.69%   +0.02%     
==========================================
  Files         481      484       +3     
  Lines      100338   100468     +130     
==========================================
+ Hits        56863    56959      +96     
- Misses      36456    36487      +31     
- Partials     7019     7022       +3     

Impacted Files	Coverage Δ
v2/internal/testcommon/kube_matcher.go	78.57% <0.00%> (-21.43%)	⬇️
v2/internal/testcommon/kube_per_test_context.go	86.05% <0.00%> (-1.69%)	⬇️
...g/genruntime/conditions/ready_condition_builder.go	78.78% <0.00%> (-21.22%)	⬇️
...ternal/reconcilers/azure_generic_arm_reconciler.go	73.91% <75.34%> (+0.53%)	⬆️
...l/util/annotations/select_annotations_predicate.go	77.77% <77.77%> (ø)
v2/internal/reconcilers/reconcile_policy.go	91.30% <91.30%> (ø)
v2/internal/controllers/generic_controller.go	81.81% <100.00%> (+0.56%)	⬆️
v2/internal/genericarmclient/generic_client.go	75.00% <100.00%> (ø)
v2/internal/reconcilers/annotations.go	100.00% <100.00%> (ø)
v2/pkg/genruntime/base_types.go	93.10% <100.00%> (+0.51%)	⬆️
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed772b2...26f2ce0. Read the comment docs.

[theunrepentantgeek]

Looks good. A couple of minor suggestions.

[matthchr]

/ok-to-test sha=779930e

[matthchr]

/ok-to-test sha=779930e

[matthchr]

/ok-to-test sha=26f2ce0

[matthchr]

/ok-to-test sha=ceb136c

[matthchr]

/ok-to-test sha=823f0eb