ZAPI collection of cloud_target information

[mamoep]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please let us know in a comment

Problem

Follow up to issue #2081. FabricPool dashboard is still not 100% working in 23.08. The "Buckets" panel shows no data because the "Container" variable has no values.

The REST collector has a cloud_target template that I guess generates the necessary metrics / labels. I couldn't find this template for ZAPI.

Poller

ZAPI

Version

harvest version 23.08.0-1 (commit 874a1f8) (build date 2023-08-21T08:36:53-0400) linux/amd64

OS and platform

RHEL 8.8

ONTAP or StorageGRID version

NetApp Release 9.11.1P8: Fri Apr 07 00:02:50 UTC 2023

References

#2081

[cgrinds]

hi @mamoep the cloud_target labels are collected via this ZapiPerf template

harvest/conf/zapiperf/cdot/9.8.0/wafl_comp_aggr_vol_bin.yaml

Line 11 in a3d37f8

- object_store_name => cloud_target

Can you double check that you are running with that template? It is enabled by default for ZapiPerf.

[mamoep]

According to the logs the template is working.
{"level":"info","Poller":"redacted","collector":"ZapiPerf:WAFLCompBin","apiMs":433,"parseMs":17,"calcMs":1,"pluginMs":0,"skips":0,"metrics":3276,"instances":273,"caller":"collector/collector.go:483","time":"2023-09-05T12:26:53+02:00","message":"Collected"}

This only provides a label for metric fabricpool_cloud_bin_operation. That doesn't match the query of the dashboard variable.

[cgrinds]

Thanks @mamoep. You're right. The dashboard uses label_values(cloud_target_labels{cluster=~"$Cluster"}, container) which comes from the Rest cloud_target.yaml template.

Issue #2081 was about including the cloud_stores label in aggregate metrics (also needed by the FabricPool dashboard).

We'll check if there is a Zapi that returns cloud_target instances.

[cgrinds]

I don't see a way to gather cloud targets via ZAPI. Asked the broader community on Discord.

[cgrinds]

@mamoep does this return records for you? Replace $poller with your poller name.

bin/harvest zapi -p $poller show data --api aggr-object-store-config-get-iter

[mamoep]

Yes that call does return records:

<root>
<attributes-list>
<aggr-object-store-config-info><access-key>REDACTED</access-key><auth-type>key</auth-type><ipspace>Default</ipspace><is-certificate-validation-enabled>true</is-certificate-validation-enabled><object-store-name>REDACTED-NAME-A</object-store-name><object-store-uuid>2096b8b0-3090-11eb-adc1-00a0989b207a</object-store-uuid><port>10443</port><provider-type>sgws</provider-type><s3-name>REDACTED-BUCKET-NAME-A</s3-name><server>storagegrid-a.local</server><ssl-enabled>true</ssl-enabled><used-space>221054229526528</used-space></aggr-object-store-config-info>
<aggr-object-store-config-info><access-key>REDACTED</access-key><auth-type>key</auth-type><ipspace>Default</ipspace><is-certificate-validation-enabled>true</is-certificate-validation-enabled><object-store-name>REDACTED-NAME-B</object-store-name><object-store-uuid>0050c7d8-3090-11eb-adc1-00a0989b207a</object-store-uuid><port>10443</port><provider-type>sgws</provider-type><s3-name>REDACTED-BUCKET-NAME-B</s3-name><server>storagegrid-b.local</server><ssl-enabled>true</ssl-enabled><used-space>221054229526528</used-space></aggr-object-store-config-info>
</attributes-list>
<num-records>2</num-records>
</root>

[cgrinds]

Thanks @mamoep. I've included a fix in #2350

The following labels are missing in the Zapi call, so they are not exported.

• owner
• scope

When comparing the cloud target Rest and Zapi templates, these are the only differences.

[mamoep]

I tested it and the collection works. But the the dashboard panel is still empty.
Variable name $Container query: label_values(cloud_target_labels{cluster=~"$Cluster"},container
Panel query: bucket_bytes{cluster=~"$SGCluster",bucket=~"$Container"}

"container" is not the right property here. It only works if the object-store config name and the bucket name are identical. You have to use the property "name" to get the bucket name.
Suggestion:
Variable name $BucketName query: label_values(cloud_target_labels{cluster=~"$Cluster"},name
Panel query: bucket_bytes{cluster=~"$SGCluster",bucket=~"$BucketName"}