23.11.0 grafana backup fails

[pilot7777]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please let us know in a comment

Problem

We use "harvest grafana export" to back up our grafana dashboards. After upgrading from 23.08 to 23.11, the backup fails with many .tmp files.

HOST: "grafana.prd3028.tools5c.ash.tktm.io"
GRAFANA_URL: "https://$GRAFANA_USER:$GRAFANA_PASSWORD@$HOST"

for f in $(curl -ks $GRAFANA_URL/api/folders | jq -r '.[]|.title')
do
echo n | /opt/harvest/bin/harvest grafana export --config /opt/harvest/harvest.yml -k -a https://$HOST -d $dir/$f -t $GRAFANA_TOKEN -f $f
done

-rw------- 1 gitlab-runner gitlab-runner 0 Nov 21 11:20 db_0c2f7379_8196_5dda_b8c3_1b169022942a.json.tmp
-rw------- 1 gitlab-runner gitlab-runner 0 Nov 21 11:20 db_203ae283_897f_51ac_874f_d9c2a80e1fa9.json.tmp
-rw------- 1 gitlab-runner gitlab-runner 0 Nov 21 11:20 db_363bc2e7_e6d7_5804_ae7e_b97c1f4138bf.json.tmp
-rw------- 1 gitlab-runner gitlab-runner 0 Nov 21 11:20 db_4e626517_9f67_5c01_987d_115ed642d4d2.json.tmp
-rw------- 1 gitlab-runner gitlab-runner 0 Nov 21 11:20 db_6ae4b769_0deb_5755_a3c4_086e2e93e435.json.tmp
-rw------- 1 gitlab-runner gitlab-runner 0 Nov 21 11:20 db_792bb739_cc42_5d94_b528_270efe57a8df.json.tmp
-rw------- 1 gitlab-runner gitlab-runner 0 Nov 21 11:20 db_7ae96355_4566_5171_b4d3_8f00d49f928e.json.tmp
-rw------- 1 gitlab-runner gitlab-runner 0 Nov 21 11:20 db_9797adb8_029b_51d2_a610_cab0d04f62b4.json.tmp
-rw------- 1 gitlab-runner gitlab-runner 0 Nov 21 11:20 db_aggregate_and_volume_efficiency.json.tmp
-rw------- 1 gitlab-runner gitlab-runner 0 Nov 21 11:20 db_all_coresys_prod_vax_volumes_global_tm.json.tmp
-rw------- 1 gitlab-runner gitlab-runner 0 Nov 21 11:20 db_ams_hyper_v_hvc02ams_tm.json.tmp

Configuration

No response

Poller

n/a

Version

23.11.0

Poller logs

No response

OS and platform

Oracle Linux 7.9

ONTAP or StorageGRID version

9.10.1P14

Additional Context

No response

References

No response

[rahulguptajss]

@Hardikl Can you take a look.

[Hardikl]

@pilot7777 ,

I have tried to recreate the issue in-house with latest Harvest release and grafana version 8.1.8, but harvest grafana export cli would work as expected without any issue.

Can you share which version of Grafana you are using with Harvest ? which can help us to differentiate the issue belongs to grafana or Harvest ?

Also, As you are consuming the grafana cli via script, Could you share your script with us and send it to ng-harvest-files@netapp.com.

[pilot7777]

@Hardikl Sorry I should have provided Grafana version. We are running 10.2.1 in on-prem K8s which does not have persistent storage. So we have to back up datasource/dashboard via Gitlab pipeline. Everything is in the gitlab-ci.yaml file. I have emailed it to ng-harvest-files@netapp.com.

Thank you!

[rahulguptajss]

@pilot7777 We've identified and resolved the issue you reported via PR #2505. Fix is available via nightly builds. Thanks!

[rahulguptajss]

verified in main