Update SLO reconcialition loop to avoid creating duplicate SLOs

[philippeVV]

What does this PR do?

We observed that the SLO controller creates duplicate SLO when running with more than 1 replica. (issue 1062)

The PR fixes the issue by adding a new SyncStatus state. The "creating SLO" syncStatus is used to ensure that only one reconcile loop can be creating an SLO at the time.

Motivation

(issue 1062)

Additional Notes

N/A

Minimum Agent Versions

N/A

Describe your test plan

When running the latest version of the operator with more than 1 replica, we observed that a DatadogSLO would create 2 SLOs in Datadog. When building the PR, you will observe that only one SLO is created when applying a DatadogSLO.

Checklist

• PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
• PR has a milestone or the qa/skip-qa label

[github-actions]

This pull request does not contain a valid label. Please add one of the following labels: bug, enhancement, refactoring, documentation, tooling, dependencies

[philippeVV]

Hey @khewonc @celenechang,

Can I have some help merging this PR and adding the required labels?

First-time contributor, please let me know if I need to change anything

[celenechang]

@philippeVV thank you very much for your contribution. I happened to be working on this issue as well; would you be willing to test an alternative solution if/when I push my branch? Or perhaps you can share the SLO manifest you used? Thank you.

[philippeVV]

@celenechang Yes, I'll be interested to see/test your implementation. In my test, I used the SLO defined in the config/samples/datadoghq_v1alpha1_datadogslo.yaml.

[celenechang]

@philippeVV PR opened at #1185. Thank you

[philippeVV]

Hey @celenechang,

You mentioned on #1185 that you had some reservations regarding the changes in this PR. Can you please elaborate?
While we didn't fully understand the issue and it's not reliably reproducible, updating the syncStatus before creating the SLO eliminates the possibility of duplicate SLO.

Thank you

[levan-m]

@philippeVV, first thank you for your contribution and for helping with testing the other PR.

Our reservations are mainly around two points:

• From our investigation root cause of duplication is concurrent reconciliation (and SLO API not being idempotent at least with current implementation). This PR doesn't address concurrent reconciliation and I believe creating duplicated SLOs with this approach is still matter of timing.
• Proposed implementation may also cause reconciler to get stuck and never create an SLO. In order this change to work as intended below setting status and creating SLO should both succeed in the same reconcile. If first block succeeds and second fails, we end up with status set to DatadogSLOSyncCreating and never attempt to create the SLO again exiting here. Given r.checkRequiredTags() and r.create() make multiple API calls to API server and Datadog, this is quite possible.

Sorry, we didn't provide these details earlier. Let me know if you have more questions.

[philippeVV]

Thanks for the feedback @levan-m, it's much appreciated.

I believe I understand the issues and will try to address them.

[levan-m]

@philippeVV did you have a chance to test the most recent Operator release?
Any reason to keep this PR open?

[philippeVV]

Hi @levan-m, I didn't get a chance to revisit the issue since then. I'll close the PR until I get time to test it again.