Following Instructions but no Config File #13

ApickettLGA · 2022-11-19T06:23:58Z

I'm new to Docker and containers but I have the basic configuration running and I have gained access to the page so I know the container is running. Which example is the best configuration file to have basic metrics pulled and then how do I tell the Docker Build or what configuration file do I need to modify to use the config file i created?

genegr · 2022-11-21T16:08:01Z

The exporter is specifically built with the intent of not having any configuration file and with a very minimal set of configuration parameters, so there is nothing that you need to pass it in order to work.
All what you need to have is a working Prometheus instance with a configuration file containing a job for each FlashArray/FlashBlade you want to collect the metrics from. A very basic Prometheus config file can be seen here https://github.com/PureStorage-OpenConnect/pure-fa-openmetrics-exporter/blob/master/examples/config/k8s/prometheus-configmap.yaml, as the value of the 'prometheus.yml' key. Because of the authentication that is specifically required by the exporter you cannot leverage the Prometheus labeling as a mean to keep the configuration concise and in a single job. You need instead to define a job for each appliance you want to monitor, which somewhat simplifies the configuration but at the same time originates a proliferation of entries in the file.

ApickettLGA · 2022-11-21T17:25:45Z

I have set that configuration file up and placed it in a temp folder, but how do I get the Docker Container to see that file? Or is there a specific place i need to put it so the Container sees it automatically?

andrewm659 · 2022-11-21T17:35:07Z

Not being able to take advantage of the labeling seems like a bad ieda. Is there a reason it was created for both exporters this way? FB and FA.

ApickettLGA · 2022-11-22T05:28:42Z

I ran this command and I still don't see the the container isn't seeing the config file:
docker run -d --restart unless-stopped -p 9490:9490 --name Pure-Metrics -v /etc/docker/configs/pure-configmap.yaml:/etc/prometheus/prometheus.yml quay.io/purestorage/pure-fa-om-exporter:1.0.1

genegr · 2022-11-22T23:56:38Z

@andrewm659 I completely understand you point, but the issue arises from the fact that providing authentication/authorization credentials or tokens as query parameters is not a good practice, so we decided to remove that possibility. Prometheus provides the authorization config key for that specific purpose. This approach unfortunately requires to define a job for each target FlashArray, as it is not possible to create unique API token for a pool of arrays.

ApickettLGA · 2022-11-23T00:52:40Z

@genegr Does it seem that I have the command correct? When it loads it doesn't seem to be seeing the config

genegr · 2022-11-24T02:44:17Z

@ApickettLGA The command you are using runs the exporter properly, but it is the additional volume you are trying to pass to the container that is not understood/used by the exporter. The exporter does not use any config file and it has only the few options that are shown when it is executed with the -h/--help flag

docker run  quay.io/purestorage/pure-fa-om-exporter:v1.0.2 -h

Usage of /usr/local/bin/pure-fa-om-exporter:
  -debug
    	Debug
  -host string
    	Address of the exporter (default "0.0.0.0")
  -port int
    	Port of the exporter (default 9490)

All the configuration happens at the Prometheus side, in which config file you have to add the target array as the query parameter of the scraped endpoint, like this:

...
scrape_configs:
      - job_name: 'purestorage-fa'
        metrics_path: /metrics
        authorization:
          credentials: 2b74f9eb-a35f-40d9-a6a6-33c13775a53c    <-- api token
        params:
          endpoint: ['10.11.112.6']   <--  array management ip address (or hostname)
        static_configs:
        - targets:
          - 10.10.8.4:9490  <-- exporter ip address (or hostname) and port
          labels:
            location: uk
            site: London
            instance: fa-prod-01
...

ApickettLGA · 2022-12-02T03:32:22Z

@genegr I now see the Server itself needs this installed and then the scrape can be configured, but I did that and its still not loading the config:

my global config

global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_timeout is set to the global default (10s).

Alertmanager configuration

alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093

Load rules once and periodically evaluate them according to the global 'evaluation_interval'.

rule_files:

- "first_rules.yml"

- "second_rules.yml"

A scrape configuration containing exactly one endpoint to scrape:

Here it's Prometheus itself.

scrape_configs:

The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

job_name: "prometheus"

metrics_path defaults to '/metrics'

scheme defaults to 'http'.

static_configs:
- targets: ["localhost:9090"]
job_name: 'purestorage-fa'
metrics_path: /metrics
authorization:
credentials: blah
params:
endpoint: ['10.6.100.71']
static_configs:
- targets:
  - 10.6.25.132:9490
    labels:
    location: Frederick
    site: Maryland
    instance: mapsfa01

When I go to the page:
http://mvlmddstor01:9490/metrics/array?endpoint=host

Target authorization token is missing

ApickettLGA · 2022-12-08T20:32:50Z

I was hoping I could get any more assistance as I know this is close but not sure what else would be missing

ApickettLGA · 2022-12-15T14:51:13Z

@genegr or Anyone, I configured Prometheus on the Ubuntu Server and configured the script as above, but the Pure Import Container is still not seeing the correct information, is there something I missing or can check to further troubleshooting?

james-laing · 2022-12-15T21:29:10Z

@ApickettLGA it's a little unclear to understand your prometheus.yaml config as it's not in a code block and markdown has formatted it all. It looks like Prometheus is not picking up your authorization credentials as yaml format is expecting an indentation.
There are some enhancements to the README files on the way. In the meantime, take a look at this example for configuring prometheus.yaml.

We recently posted some additional content on how to deploy Prometheus and Grafana with overview dashboards for FA/FB.
https://github.com/PureStorage-OpenConnect/pure-fa-openmetrics-exporter/tree/master/extra/grafana

There is a troubleshooting section there for Prometheus, try running this to prove your config works.

> promtool check config /etc/prometheus/prometheus.yml
Checking prometheus.yml
 SUCCESS: prometheus.yml is valid prometheus config file syntax

Also I notice you are running the query against /metrics. While this will work, it is an expensive query and it may take longer than the timeout and therefore may fail. It is recommended to configure a query job for specific metric endpoints /metrics/array, /metrics/volumes, /metrics/hosts, etc.

I have gone to the effort to format your yaml configuration and included jobs to /metrics/array, /metrics/volumes and /metrics/hosts and processed through a yaml validator online to ensure it conforms to yaml formatting.

global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
      - targets:
        # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:

  - job_name: "prometheus"
    static_configs:
    - targets: ["localhost:9090"]

# Pure Storage OpenMetrics Exporter - mapsfa01
  - job_name: 'purefa_array_mapsfa01'
    metrics_path: /metrics/array
    # Bearer authorization token
    authorization:
      credentials: a12345bc6-d78e-901f-23a4-56b07b89012
    params:
      endpoint: ['10.6.100.71']

    static_configs:
    # purefa openmetrics exporter
    - targets:
      - 10.6.25.132:9490
      labels:
        location: Frederick
        site: Maryland
        instance: mapsfa01
        env: prod

  - job_name: 'purefa_hosts_mapsfa01'
    metrics_path: /metrics/hosts
    # Bearer authorization token
    authorization:
      credentials: a12345bc6-d78e-901f-23a4-56b07b89012
    params:
      endpoint: ['10.6.100.71']

    static_configs:
    # purefa openmetrics exporter
    - targets:
      - 10.6.25.132:9490
      labels:
        location: Frederick
        site: Maryland
        instance: mapsfa01
        env: prod

  - job_name: 'purefa_volumes_mapsfa01'
    metrics_path: /metrics/volumes
    # Bearer authorization token
    authorization:
      credentials: a12345bc6-d78e-901f-23a4-56b07b89012
    params:
      endpoint: ['10.6.100.71']

    static_configs:
    # purefa openmetrics exporter
    - targets:
      - 10.6.25.132:9490
      labels:
        location: Frederick
        site: Maryland
        instance: mapsfa01
        env: prod

Run the promtool check config prometheus.yaml and restart Prometheus.

There are some troubleshooting steps in the README.md which might help you with each component.

For example, to check the exporter is working we need to pass the bearer token.

curl -H 'Authorization: Bearer a12345bc6-d78e-901f-23a4-56b07b89012' -X GET http://10.6.25.132:9490/metrics/array?endpoint=10.6.100.71

ApickettLGA · 2022-12-28T03:16:56Z

@james-laing Thank you for all your help and its really appreciated. I have taken your information and applied and seems to be passing the check, and I can run the curl but when I go to the page I'm still not getting any results back.

Promtheus Check:

promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

And Curl Check looks fine:

curl -H 'Authorization: Bearer abc-blah' -X GET http://10.6.25.132:9490/metrics/array?endpoint=10.6.100.71

HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.

TYPE go_gc_duration_seconds summary

go_gc_duration_seconds{quantile="0"} 3.9432e-05
go_gc_duration_seconds{quantile="0.25"} 0.000119313
go_gc_duration_seconds{quantile="0.5"} 0.000154718
go_gc_duration_seconds{quantile="0.75"} 0.000213092
go_gc_duration_seconds{quantile="1"} 0.011197259
go_gc_duration_seconds_sum 369.193937972
go_gc_duration_seconds_count 2.215424e+06

HELP go_goroutines Number of goroutines that currently exist.

TYPE go_goroutines gauge

go_goroutines 93

Browser Check:
Page:
http://mvlmddstor01.lgamerica.com:9490/metrics?endpoint=host
Result:
Target authorization token is missing

james-laing · 2022-12-28T18:04:55Z

@ApickettLGA The browser can't pass an authorisation token as you can see in the result of the browser check it is failing.
Result: Target authorization token is missing

The curl output you provided looks truncated. You need to scroll down and check you can see purefa_ metrics.

It looks like you are almost there, please try the troubleshooting section of the setup README.md.
https://github.com/PureStorage-OpenConnect/pure-fa-openmetrics-exporter/blob/master/extra/grafana/README.md

If there is an issue please provide feedback. If you are still requiring consultation to pull data from FlashArray, Pure Storage Professional Services offer a Monitoring and Observability service specifically for this. Just get in touch with your sales representative.

james-laing · 2023-01-11T22:20:09Z

@ApickettLGA Did you see any purefa_xxxx results returned from the curl command?

Did you see targets listed in Prometheus and can you see results in with a PromQL query?

ApickettLGA · 2023-01-25T19:31:32Z

@james-laing Thank you for your response; Sorry We have had tons of outages that i couldn't get back to this.

`When` I run the Curl command I can see it respond back,
 go_memstats_stack_sys_bytes 917504
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 5.49562496e+08
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 9
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 1.46949344e+06
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 51
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.96665344e+08
#HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
#TYPE process_start_time_seconds gauge
process_start_time_seconds 1.67122622665e+09
#HELP process_virtual_memory_bytes Virtual memory size in bytes.
#TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.237245952e+09
#HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
#TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes `1.8446744073709552e+19

So I know it is connecting with the API Key, I'm going to try troubleshooting with Prometheus now, Thank you

ApickettLGA · 2023-01-25T19:51:49Z

@james-laing I was able to Connect to the Prometheus Connection and I'm able to run active queries on my Pure Storage and see live data.

james-laing · 2023-01-27T08:50:03Z

@ApickettLGA great news, excellent!

Are you planning on pointing Grafana to your Prometheus TSDB to view broad and historical metrics?
https://github.com/PureStorage-OpenConnect/pure-fa-openmetrics-exporter/blob/master/extra/grafana/README.md

ApickettLGA · 2023-01-30T15:47:28Z

@james-laing - Well right now I'm able to see the Data which is very good, but I'm still having an issue with the Container Web page; I'm still getting "Target authorization token is missing" when I visit the page

And the plan is to have our System monitoring software DataDog query the page and pull the information then put it to their page. This way all the stats will be collected, but need the Container Page to work. Seems like I have all the pieces working but the container.

sdodsley · 2023-01-30T15:51:15Z

@ApickettLGA just out of curiosity why not use the Pure FlashArray DataDog integration that we already have?
https://docs.datadoghq.com/integrations/purefa/

ApickettLGA · 2023-01-30T19:06:00Z

Funny enough, DataDog sent us down the rabbit hole of using this step to send data to DD.. Trying the Link you sent me to see how that works. Thank you

sdodsley · 2023-01-30T19:12:59Z

@andrewm659 this as well: http://theansibleguy.com/datadog-and-pure-storage/

ApickettLGA · 2023-01-30T21:42:15Z

@sdodsley Reading the steps DataDog steps and it does require for me to have the container setup;
The Pure Storage Prometheus exporter is installed and running in a containerized environment. Refer to the GitHub repo for installation instructions.

So this leads me back to my page not working correctly.. If I can get the container webpage to work then I would be completed this with step.. I have Prometheus working but it doesn't seem like the page is working from the container

james-laing · 2023-01-30T22:12:38Z

@ApickettLGA if your objective is to get data into Datadog, you would simplify your solution by using the Datadog integration for Pure FA. It works with both the depricated pure-exporter and purefa_openmetrics-exporter.

Configure the /etc/datadog-agent/conf.d/purefa.d/mapsfa01.conf.yaml like the following example:

init_config:
   timeout: 60

instances:

# Pure Storage OpenMetrics Exporter - mapsfa01
  - openmetrics_endpoint: http://10.6.25.132:9490/metrics/array?endpoint=10.6.100.71
    tags:
       - env:prod
       - host:mapsfa01
       - fa_array_name:mapsfa01
    headers:
       Authorization: Bearer a12345bc6-d78e-901f-23a4-56b07b89012
    min_collection_interval: 30

Restart Datadog and check the metrics are being pulled with datadog-agent status in the purefa section

    purefa (1.0.99.22)
    ------------------
      Instance ID: purefa:123456789abcdef [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/purefa.d/mapsfa01.conf.yaml
      Total Runs: 10
      Metric Samples: Last Run: 414, Total: 7,731,703
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 19,638
      Average Execution Time : 1.273s
      Last Execution Date : 2023-01-30 22:01:53 UTC (1675116113000)
      Last Successful Execution Date : 2023-01-30 22:01:53 UTC (1675116113000)

If the data is already in Prometheus and you wish to continue with this solution, you will need to point Datadog to the Prometheus instance, not the exporter.

In your previous comments you've stated you are trying to connect to the purefa-openmetrics-exporter from the browser. This won't work as the browser cannot pass the bearer token like cURL, Prometheus and Datadog can.

Hope this helps.

ApickettLGA · 2023-01-31T04:28:45Z

That makes sense and I didn't realize it wouldn't work from my browser. So I have configured my DD Agent and now waiting for my networking team to open the port for connection, then I think I might be good to go!

Thank you all for your assistance!

chrroberts-pure · 2023-02-01T17:45:17Z

To correct @james-laing 's comment - the current published datadog-purefa==1.0.1 does NOT include all metrics required for both pure-fa-openmetrics-exporter and pure-exporter.

I'm working on the datadog-purefa==1.1.0 which will add the other required metrics and dashboards for pure-fa-openmetrics-exporter. I'll be submitting that to DataDog this week.

datadog-purefa==1.0.99.22 that we are running in our test env as above is the beta/RC build
Please let me know if you would like a wheel of datadog-purefa==1.0.99.22 to install on your datadog-agent.

ApickettLGA · 2023-03-03T15:34:13Z

@james-laing ; Thank you for all your help! @chrroberts-pure I finally have everything up and running and we are getting some metrics into DataDog, but we arn't seeing all and now DataDog is asking for us to upgrade to 1.1.0, but i don't see this version published? Can I upgrade to the latest version to get more metrics or do you know when 1.1 will be released?

chrroberts-pure · 2023-03-03T15:38:21Z

Hi, @ApickettLGA - Datadog PureFA integration v1.1 was released on Feb 28, 2023.

DataDog/integrations-extras#1750

https://docs.datadoghq.com/integrations/purefa/ also lists the version as v1.1.0

chrroberts-pure · 2023-03-03T15:44:51Z

Also - feel free to reach out to the Observability channel in our Slack channel, I'll be in there if you'd like to connect.

https://code-purestorage.slack.com/messages/C0357KLR1EU

sdodsley · 2023-03-06T22:28:40Z

Closing this as the latest OME release and DD integration provide all the fixes.

sdodsley closed this as completed Mar 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Following Instructions but no Config File #13

Following Instructions but no Config File #13

ApickettLGA commented Nov 19, 2022

genegr commented Nov 21, 2022

ApickettLGA commented Nov 21, 2022

andrewm659 commented Nov 21, 2022

ApickettLGA commented Nov 22, 2022

genegr commented Nov 22, 2022

ApickettLGA commented Nov 23, 2022

genegr commented Nov 24, 2022 •

edited

Loading

ApickettLGA commented Dec 2, 2022 •

edited

Loading

metrics_path defaults to '/metrics'

scheme defaults to 'http'.

ApickettLGA commented Dec 8, 2022

ApickettLGA commented Dec 15, 2022

james-laing commented Dec 15, 2022

ApickettLGA commented Dec 28, 2022

james-laing commented Dec 28, 2022

james-laing commented Jan 11, 2023

ApickettLGA commented Jan 25, 2023 •

edited

Loading

ApickettLGA commented Jan 25, 2023

james-laing commented Jan 27, 2023

ApickettLGA commented Jan 30, 2023

sdodsley commented Jan 30, 2023

ApickettLGA commented Jan 30, 2023

sdodsley commented Jan 30, 2023

ApickettLGA commented Jan 30, 2023

james-laing commented Jan 30, 2023

ApickettLGA commented Jan 31, 2023

chrroberts-pure commented Feb 1, 2023 •

edited

Loading

ApickettLGA commented Mar 3, 2023 •

edited

Loading

chrroberts-pure commented Mar 3, 2023 •

edited

Loading

chrroberts-pure commented Mar 3, 2023

sdodsley commented Mar 6, 2023

Following Instructions but no Config File #13

Following Instructions but no Config File #13

Comments

ApickettLGA commented Nov 19, 2022

genegr commented Nov 21, 2022

ApickettLGA commented Nov 21, 2022

andrewm659 commented Nov 21, 2022

ApickettLGA commented Nov 22, 2022

genegr commented Nov 22, 2022

ApickettLGA commented Nov 23, 2022

genegr commented Nov 24, 2022 • edited Loading

ApickettLGA commented Dec 2, 2022 • edited Loading

my global config

scrape_timeout is set to the global default (10s).

Alertmanager configuration

Load rules once and periodically evaluate them according to the global 'evaluation_interval'.

- "first_rules.yml"

- "second_rules.yml"

A scrape configuration containing exactly one endpoint to scrape:

Here it's Prometheus itself.

The job name is added as a label job=<job_name> to any timeseries scraped from this config.

metrics_path defaults to '/metrics'

scheme defaults to 'http'.

ApickettLGA commented Dec 8, 2022

ApickettLGA commented Dec 15, 2022

james-laing commented Dec 15, 2022

ApickettLGA commented Dec 28, 2022

HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.

TYPE go_gc_duration_seconds summary

HELP go_goroutines Number of goroutines that currently exist.

TYPE go_goroutines gauge

james-laing commented Dec 28, 2022

james-laing commented Jan 11, 2023

ApickettLGA commented Jan 25, 2023 • edited Loading

ApickettLGA commented Jan 25, 2023

james-laing commented Jan 27, 2023

ApickettLGA commented Jan 30, 2023

sdodsley commented Jan 30, 2023

ApickettLGA commented Jan 30, 2023

sdodsley commented Jan 30, 2023

ApickettLGA commented Jan 30, 2023

james-laing commented Jan 30, 2023

ApickettLGA commented Jan 31, 2023

chrroberts-pure commented Feb 1, 2023 • edited Loading

ApickettLGA commented Mar 3, 2023 • edited Loading

chrroberts-pure commented Mar 3, 2023 • edited Loading

chrroberts-pure commented Mar 3, 2023

sdodsley commented Mar 6, 2023

genegr commented Nov 24, 2022 •

edited

Loading

ApickettLGA commented Dec 2, 2022 •

edited

Loading

The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

ApickettLGA commented Jan 25, 2023 •

edited

Loading

chrroberts-pure commented Feb 1, 2023 •

edited

Loading

ApickettLGA commented Mar 3, 2023 •

edited

Loading

chrroberts-pure commented Mar 3, 2023 •

edited

Loading