Invalid syntax in 24.05 when VolumeAnalytics enabled #3023

db-wally007 · 2024-06-29T17:12:10Z

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please let us know in a comment

Problem

Hello,

I've upgraded Harvest from 24.02 to the latest (as of today) 24.05 and upon container start, I see the following error flooding container log output (journald in my case)

Jun 29 17:00:28 podman1 netapp-harvest-exporter[3241772]: 2024-06-29T17:00:28Z ERR volumeanalytics/volumeanalytics.go:253 > set metric error="strconv.ParseFloat: parsing \"\": invalid syntax" Poller=agora object=VolumeAnalytics plugin=Rest:VolumeAnalytics value=

config file:

---

Exporters:
  troy:
    exporter: Prometheus
    local_http_addr: 0.0.0.0
    port: 12990
    global_prefix: netapp_
  agora:
    exporter: Prometheus
    local_http_addr: 0.0.0.0
    port: 12991
    global_prefix: netapp_

Pollers:
  troy:
    datacenter: EQX
    addr: troy-cluster.xxx.de
    auth_style: basic_auth
    username: $__env{NETAPP_HARVEST_READONLY_USERNAME}
    password: $__env{NETAPP_HARVEST_READONLY_PASSWORD}
    use_insecure_tls: true
    exporters:
      - troy
    collectors:
      - Rest
      - RestPerf
      - Ems
  agora:
    datacenter: EQX
    addr: agora-cluster.xxx.de
    auth_style: basic_auth
    username: $__env{NETAPP_HARVEST_READONLY_USERNAME}
    password: $__env{NETAPP_HARVEST_READONLY_PASSWORD}
    use_insecure_tls: true
    exporters:
      - agora
    collectors:
      - Rest
      - RestPerf
      - Ems

Container file:

    FROM {{ netapp_harvest_exporter_podman_template }}
     EXPOSE 12990 12991
     COPY harvest.yml /opt/harvest/harvest.yml
     COPY harvest_entrypoint.sh /opt/harvest/harvest_entrypoint.sh
     ENV NETAPP_HARVEST_READONLY_USERNAME
     ENV NETAPP_HARVEST_READONLY_PASSWORD
     WORKDIR /opt/harvest
     ENTRYPOINT ["./harvest_entrypoint.sh"]
     CMD ["start", "--config", "harvest.yml", "--loglevel=4"]

harvest_entrypoint.sh:

      #!/bin/bash
      bin/harvest "$@"
      `exec /bin/sleep infinity`

Configuration

No response

Poller

agora poller

Version

root@podman1:~>podman exec -it  netapp-harvest-exporter bin/harvest version
harvest version 24.05.2-1 (commit bae3aad2) (build date 2024-06-13T07:58:11-0400) linux/amd64

Poller logs

No response

OS and platform

RHEL 9.4 , podman

ONTAP or StorageGRID version

NetApp Release 9.13.1P7

Additional Context

root@podman1:~>podman exec -it  netapp-harvest-exporter bin/harvest zapi -p agora show system
connected to AGORA (NetApp Release 9.13.1P7: Wed Jan 24 16:17:54 UTC 2024)
[results]                                          -                                   *
  [build-timestamp]                                -                          1706113074
  [is-clustered]                                   -                                true
  [version]                                        - NetApp Release 9.13.1P7: Wed Jan 24 16:17:54 UTC 2024

  [version-tuple]                                  -                                   *
    [system-version-tuple]                         -                                   *
      [generation]                                 -                                   9
      [major]                                      -                                  13
      [minor]                                      -                                   1

References

No response

The text was updated successfully, but these errors were encountered:

db-wally007 · 2024-06-29T17:23:57Z

I just checked our 2 clusters and issue is a bit strange and maybe it will help you to replicate the issue.

1, TROY appliance - has volume with VolumeAnalytics feature enabled = no errors from the poller and VolumeAnalytics metrics are present in the Prometheus/Grafana

2, AGORA appliance - has no volume with VolumeAnalytics feature enabled but we have SVM-DR relationship setup from TROY -> to -> AGORA (i.e. SVM from TROY that has the VolumeAnalytics enabled is replicated to AGORA) = error in the logs as described in the earlier post.

rahulguptajss · 2024-07-01T11:00:10Z

It appears that the AGORA appliance has volumes with analytics enabled. Could you run the following command for the AGORA appliance? Please note that the USERNAME, PASSWORD, URL should be replaced with the appropriate credentials.

curl -s -k -u USERNAME:PASSWORD 'https://URL/api/storage/volumes?return_records=true&fields=name,svm.name,uuid&analytics.state=on&max_records=20&order_by=space.used%20desc&ignore_unknown_fields=true'

If there are records returned, could you take the UUID of a volume and replace UUID in the following command to check if percentages are available in the response? It seems they are empty resulting in error log.

curl -s -k -u USERNAME:PASSWORD 'https://URL/api/storage/volumes/UUID/files?return_records=true&fields=analytics.by_accessed_time.bytes_used.percentages&type=directory&max_records=100'

Also, could you also share the ONTAP version of the Troy appliance?

You can also share output of these curl commands with us via ng-harvest-files@netapp.com

db-wally007 · 2024-07-01T18:27:27Z

ng-harvest-files@netapp.com

Email sent with output of the curl commands from both SVM-DR source (TROY) and SVM-DR destination (AGORA)

It includes home directories with internal usernames - which might be somewhat sensitive.
Let me know if you received it.

TROY (AFF-250) has the exact same version as AGORA (AFF-220) (we were advised by the Netapp techs to try to keep ONTAP versions in sync for best results with SVM-DR functionality)

cgrinds · 2024-07-01T18:37:15Z

Thanks @db-wally007 we got the curl emails

cgrinds · 2024-07-01T19:05:22Z

hi @db-wally007 your curl request shows that ONTAP believes analytics are enabled for two volumes on agora, both of those volumes are on the same SVM. It sounds like you didn't think volume analytics were enabled on agora. Is that correct?

Interestingly, even though those two volumes say they have volume analytics enabled, when queried for analytics they return no analytics. 😄 I wonder if this is an ONTAP bug related to the SVM-DR relationship you mentioned? Are you on Discord? If so, please ask if this is a bug on the #ontap channel.

Regardless, Harvest needs to do a better job ignoring these faux volume-analytics-enabled volumes. We'll fix that.

db-wally007 · 2024-07-01T20:06:17Z

hi @db-wally007 your curl request shows that ONTAP believes analytics are enabled for two volumes on agora, both of those volumes are on the same SVM. It sounds like you didn't think volume analytics were enabled on agora. Is that correct?

Interestingly, even though those two volumes say they have volume analytics enabled, when queried for analytics they return no analytics. 😄 I wonder if this is an ONTAP bug related to the SVM-DR relationship you mentioned? Are you on Discord? If so, please ask if this is a bug on the #ontap channel.

Regardless, Harvest needs to do a better job ignoring these faux volume-analytics-enabled volumes. We'll fix that.

VolumeAnalytics is enabled on volumes that reside on TROY (plato_svm) and that SVM gets replicated to AGORA via SVM-DR feature into dr_plato_svm SVM.

dr_plato_svm on AGORA is stopped, maybe that is the issue ?
(while source SVM plato_svm is running, destination SVM dr_plato_svm must be stopped, obviously)

AGORA serves as a SVM-DR destination.

Re: Discord, (un)fortunately I do not (no social media accounts)

rahulguptajss · 2024-07-02T10:42:16Z

@db-wally007 It will be fixed via #3026.

In the meantime, you may try to disable volume activity tracking for these volumes which are in an SVM-DR stopped vserver.

List Relevant Volumes

To list the relevant volumes with activity tracking enabled, use the following command:

volume show -activity-tracking-state on -vserver

Disable Activity Tracking

To disable activity tracking, use the following command:

volume activity-tracking off -volume

Ideally, in a DR relationship SVM, activity tracking should be disabled. I tried enabling it in version 9.15 but encountered the following error:

Volume activity tracking wasn't enabled on volume "umeng_aff300_05_svm5_root" in storage VM "xyz" due to the following reason: "This operation is not permitted on a Vserver that is configured as the destination for Vserver DR."

db-wally007 · 2024-07-02T10:50:38Z

@db-wally007 It will be fixed via #3026.

In the meantime, you may try to disable volume activity tracking for these volumes which are in an SVM-DR stopped vserver.

Ideally, in a DR relationship SVM, activity tracking should be disabled. I tried enabling it in version 9.15 but encountered the following error:

Like I wrote a few times, VolumeAnalytics IS off on AGORA.
The problem is with the query or query response that Harvest is doing.

Here is ONTAP output:

AGORA::*> volume activity-tracking show
vserver volume                                     state
------- ------------------------------------------ -----
AGORA   MDV_CRS_e26ccfc6e20911ea9b9cd039ea1acb75_A off
AGORA   MDV_CRS_e26ccfc6e20911ea9b9cd039ea1acb75_B off
AGORA-01
        vol0                                       off
AGORA-02
        vol0                                       off
agora_svm
        agora_svm_root_volume                      off
agora_svm
        vmware_backup_agora                        off
dr_plato_svm
        backup                                     off
dr_plato_svm
        home_dirs                                  off
dr_plato_svm
        pcap                                       off
dr_plato_svm
        plato_svm_root_volume                      off
dr_plato_svm
        rdf                                        off
dr_plato_svm
        shared_data                                off
dr_plato_svm
        xml                                        off
13 entries were displayed.

AGORA::*>

and

AGORA::*> volume show -activity-tracking-state on
There are no entries matching your query.

AGORA::*>

rahulguptajss · 2024-07-02T11:08:32Z

Understood. Activity Tracking is used for Top K metrics in ONTAP. I should have asked you to check the output of the following CLI, as these are what we filter on in the REST call.

You should see some results for this SVM with the following query:

volume show -analytics-state on

However, we get empty records when we call the following REST curl, which makes sense as files were never accessed on this volume on the SVM-DR site yet. This is something we need to handle in Harvest. It will be fixed via #3026. Thanks.

curl -s -k -u xxx:xxx 'https://xxxx/api/storage/volumes/xxxx/files?return_records=true&fields=analytics.by_accessed_time.bytes_used.percentages&type=directory&max_records=100'

db-wally007 · 2024-07-02T11:13:32Z

Understood. Activity Tracking is used for Top K metrics in ONTAP. I should have asked you to check the output of the following CLI, as these are what we filter on in the REST call.

You should see some results for this SVM with the following query:
volume show -analytics-state on

Here is the output from SVM-DR destination:

AGORA::*> volume show -analytics-state on
There are no entries matching your query.

AGORA::*>

and here is the output from SVM-DR source:

TROY::*> volume show -analytics-state on
Vserver   Volume       Aggregate    State      Type       Size  Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
plato_svm home_dirs    -            online     RW      42.10TB    21.78TB    0%
plato_svm shared_data  -            online     RW      42.10TB    21.89TB    0%
2 entries were displayed.

TROY::*>

rahulguptajss · 2024-08-07T07:05:01Z

verified in 24.08

db-wally007 added the status/needs-triage label Jun 29, 2024

rahulguptajss self-assigned this Jul 2, 2024

rahulguptajss added bug Something isn't working 24.08 customer status/open and removed status/needs-triage labels Jul 2, 2024

rahulguptajss linked a pull request Jul 2, 2024 that will close this issue

fix: handle volume analytics error logging #3026

Merged

cgrinds closed this as completed in #3026 Jul 2, 2024

cgrinds added status/testme and removed status/open labels Jul 8, 2024

cgrinds unassigned rahulguptajss Aug 6, 2024

rahulguptajss added status/done and removed status/testme labels Aug 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Invalid syntax in 24.05 when VolumeAnalytics enabled #3023

Invalid syntax in 24.05 when VolumeAnalytics enabled #3023

db-wally007 commented Jun 29, 2024 •

edited

Loading

db-wally007 commented Jun 29, 2024 •

edited

Loading

rahulguptajss commented Jul 1, 2024

db-wally007 commented Jul 1, 2024 •

edited

Loading

cgrinds commented Jul 1, 2024

cgrinds commented Jul 1, 2024

db-wally007 commented Jul 1, 2024 •

edited

Loading

rahulguptajss commented Jul 2, 2024

db-wally007 commented Jul 2, 2024

rahulguptajss commented Jul 2, 2024 •

edited

Loading

db-wally007 commented Jul 2, 2024

rahulguptajss commented Aug 7, 2024

Invalid syntax in 24.05 when VolumeAnalytics enabled #3023

Invalid syntax in 24.05 when VolumeAnalytics enabled #3023

Comments

db-wally007 commented Jun 29, 2024 • edited Loading

A note for the community

Problem

Configuration

Poller

Version

Poller logs

OS and platform

ONTAP or StorageGRID version

Additional Context

References

db-wally007 commented Jun 29, 2024 • edited Loading

rahulguptajss commented Jul 1, 2024

db-wally007 commented Jul 1, 2024 • edited Loading

cgrinds commented Jul 1, 2024

cgrinds commented Jul 1, 2024

db-wally007 commented Jul 1, 2024 • edited Loading

rahulguptajss commented Jul 2, 2024

List Relevant Volumes

Disable Activity Tracking

db-wally007 commented Jul 2, 2024

rahulguptajss commented Jul 2, 2024 • edited Loading

db-wally007 commented Jul 2, 2024

rahulguptajss commented Aug 7, 2024

db-wally007 commented Jun 29, 2024 •

edited

Loading

db-wally007 commented Jun 29, 2024 •

edited

Loading

db-wally007 commented Jul 1, 2024 •

edited

Loading

db-wally007 commented Jul 1, 2024 •

edited

Loading

rahulguptajss commented Jul 2, 2024 •

edited

Loading