Skip to content

Metric collection compatibility enhancement#1621

Merged
yorickdowne merged 8 commits into
ethstaker:mainfrom
aliask:monitoring-compatibility
Nov 25, 2023
Merged

Metric collection compatibility enhancement#1621
yorickdowne merged 8 commits into
ethstaker:mainfrom
aliask:monitoring-compatibility

Conversation

@aliask
Copy link
Copy Markdown
Contributor

@aliask aliask commented Nov 11, 2023

Change overview

  • Increased default polling rate to 15s
  • Manually reduced the price-check poll rates to 5m, no point hammering the APIs
  • Allow override of job name and polling interval via Docker labels

Explanation

Eth-docker currently does not set the polling configs, so it uses the default values (1 minute interval).

Several publicly shared Grafana dashboards expect Prometheus metrics to be available at a higher rate than what eth-docker provides, so importing these dashboards into Grafana causes some widgets to display "No Data".

These values were taken from the Rocketpool config, but the official lighthouse-metrics config also uses 15s, and geth even suggests 10s.

I've also added some extra config overrides to set the job name and poll interval via Docker labels. If these labels are omitted it still defaults to the compose service name.

In my case, this was done for compatibility with the Rocketpool dashboard as they name their jobs differently service name, but there may be other situations where this would be helpful.

I know that the polling settings can be overriden via a custom-prom.yml file, however the job name and individual container interval settings cannot (as far as I understand).

The increased polling rate will give users higher resolution metrics as a bonus, and the additional load on resources should not be significant.

- Increased default polling rate to 15s
- Set price-check poll rates to 5m
- Allow override of job name and polling interval via Docker labels
@aliask aliask force-pushed the monitoring-compatibility branch from e59aa23 to 0249704 Compare November 12, 2023 03:45
@aliask
Copy link
Copy Markdown
Contributor Author

aliask commented Nov 12, 2023

I've added a few new features:

  • Uses yq tool to perform a merge of the default and custom configs. Ref
  • Rootless configs now also use a folder, rootless.d for their scrape targets
  • Configuration is back to being regenerated at each startup instead of at build-time
  • A new .env setting has been added to allow bypassing config generation, and using custom-prom.yml as a standalone configuration file

I'm not so sure about that last one - the other features should make the configuration extensible enough for 99% of users.

@aliask aliask force-pushed the monitoring-compatibility branch from 281ec0f to 3f9d0fc Compare November 12, 2023 09:44
Service has been killed off since end of Sept '23
@aliask aliask force-pushed the monitoring-compatibility branch from 3f9d0fc to 835f088 Compare November 12, 2023 09:53
Remove scrape jobs for node_exporter and cadvisor - they aren't running
Add scrape job for json-exporter, it's monitored normally
Don't provision dashboards that rely on missing providers
Comment thread prometheus/global-prom.yml Outdated
Comment thread grafana-rootless.yml
Comment thread prometheus/docker-entrypoint.sh Outdated
Comment thread prometheus/Dockerfile Outdated
Comment thread prometheus/docker-entrypoint.sh Outdated
Comment thread prometheus/docker-entrypoint.sh
@yorickdowne
Copy link
Copy Markdown
Collaborator

Works well here, both my global external labels as well as the remote write are preserved. Merging. Thank you!

@yorickdowne yorickdowne merged commit 850cad0 into ethstaker:main Nov 25, 2023
@gitpoap-bot
Copy link
Copy Markdown

gitpoap-bot Bot commented Nov 25, 2023

Congrats, your important contribution to this open-source project has earned you a GitPOAP!

GitPOAP: 2023 eth-docker Contributor:

GitPOAP: 2023 eth-docker Contributor GitPOAP Badge

Head to gitpoap.io & connect your GitHub account to mint!

Learn more about GitPOAPs here.

@aliask aliask deleted the monitoring-compatibility branch November 25, 2023 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants