[Bug] Dashboards constantly refreshing in operator

[redminion0]

Describe the bug
A clear and concise description of what the bug is.

When using the v5.0.0-uid for dashboard UID support if multiple dashboards are imported the operator gets stuck in a loop constantly updating all dashboards.

Version
Full semver version of the operator being used e.g. v4.10.0, v5.0.0-rc0
quay.io/weisdd/grafana-operator:v5.0.0-uid (#1027)
To Reproduce
Steps to reproduce the behavior:

load the quay.io/weisdd/grafana-operator:v5.0.0-uid image.
then load in a large number of gzipJSON dashboards (in our case 36)

or

load dashboard one at a time (using the resource) then force the pods to move node

Expected behavior
A clear and concise description of what you expected to happen.
dashboards are not constantly updated

Suspect component/Location where the bug might be occurring
Please provide this if you know where this bug might occur otherwise leave as unknown

Screenshots
If applicable, add screenshots to help explain your problem.

• Grafana Operator Version [e.g. v5.0.0]
• Environment: K8s

Additional context
Add any other context about the problem here.

[redminion0]

when comparing json the only change grafana notices is the version

[NissesSenap]

@weisdd seems like we have some fun uid overwrite issue.

[NissesSenap]

@redminion0 thanks for reporting this issue. Could you provide an example of a dashboard that we can use to verify this?

[weisdd]

@redminion0 have you applied the CRDs from that PR as well? (There's a new status field to store UID)

[redminion0]

Hi @weisdd yes I have applied the CRD's.
After some more investigation it goes like this:

adding dashboards one at a time

1. from 0 dashboards in the cluster
2. load dashboards 1 at a time (i left around 2mins per dashboard) in my case up to 36.
3. dashboards get the guids from their json (in this case gzip and base64'd) correctly.
4. dashboards are resynced only at resync period all seems fine.
5. pods restart and the operator goes into a loop of constantly reconciling/updating the dashboards.

mass adding dashboards

1. from 0 dashboards in the cluster
2. apply all 36 dashboards and the loop starts immediately.

The error doesnt happen if smaller numbers are mass loaded at once.

if the operator is left in this state we eventually get a error of:
dial tcp <IP>:3000: connect: cannot assign requested address"}

I will attempt to find the number of dashboards that causes the problem

[redminion0]

it seems to happen at around 18 dashboards (although im also thinking it might be size rather than volume) I can easily recreate it by applying all the dashboards from https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/grafana-dashboardDefinitions.yaml (although i have updated some of the panels (graph to time series etc)

[weisdd]

@redminion0 Great to hear it's reproducible. Could you share it in a form of a step-by-step guide + archive with manifests? (a full set of manifests would be useful for us to have the same resync timers and other settings)

[weisdd]

Alright, I think the repro in the archive should be enough to investigate it further:
dashboards-loop.zip

[weisdd]

@redminion0 I've prepared a fix in #1051, the test image is here: quay.io/weisdd/grafana-operator:v5.0.0-correct-uid. Sorry for the bug and thanks for reporting it :)

[evgenii-denisov]

Seems this problem still exists even in v5.0.0 with Simple Dashboard from basic example

      image: ghcr.io/grafana-operator/grafana-operator:v5.0.0

Operator constantly creates new version of dashboard every 30 second. And same problem, only version changed

Please describe what to check? And what I should provide to reproduce on your side?

[weisdd]

@evgenii-denisov The behaviour you're seeing is not a bug and has nothing to do with the original issue that got fixed in #1051. - The basic example contains resyncPeriod: 30s in its spec, that's why the operator re-uploads the dashboard every 30 seconds.
The mechanism itself is needed to cope with dashboard spec drift (e.g. due to users changing panels through UI), and calculating diffs in this case would have been over-engineering (keys can come in different order with different indentation => everything needs to be represented in the same way before "original" dashboard and the one that exists in Grafana can be compared), so the operator simply re-uploads the dashboard. The default interval is 5 minutes, though you're free to choose any other number or even disable resync :)