💡 [Feature] Log download stats from THREDDS server

[huard]

Description

It would be useful for reporting purposes to monitor data downloads from THREDDS:

• total volume of downloads (Gb/day);
• per-file download volume (Gb/day);
• per-datasets opendap streaming volumes (Gb/day);

References

This information can be parsed from NGINX logs, but those logs need to be exposed to Prometheus to be aggregated and archived within the current architecture.

Possible solutions:

• https://www.martin-helmich.de/en/blog/monitoring-nginx.html: parses logs and exposes to Prometheus
• Loki (Prometheus for logs, integrates with Graphana), seems possible to configure metrics from logs and expose them to Prometheus
• https://github.com/nginxinc/nginx-prometheus-exporter: only exposes content from the API, which is very limited
• https://github.com/google/mtail (general purpose log scraper)

Additional info

• https://www.martin-helmich.de/en/blog/monitoring-nginx.html

See also:

• Need to add some monitoring and notification to the automated deployment system and PAVICS in general #12
• 💡 [Feature] monitoring: Add a second Prometheus to scrap the first Prometheus to keep stats for long term at a lower granularity #277

Concerned Organizations

[fmigneault]

Consider downloads from WPS outputs and STAC data proxy endpoints as well for the same reasons.

[huard]

ESGF uses Beats and Logstash to collect logs and compute their stats. See https://drive.google.com/drive/folders/1LbvoYeQ_6L_bzTsO-EEhwqjIx1jZ-G1k

[fmigneault]

If the "node collector" can be located on the same instance, logstash seems like an interesting candidate. If there is no distinction between beats or logstash as "log producers", I would favor the 2nd architecture to limit the number of configurations/technologies involved.