Plat 594/ Replicate prometheus + add smtp env vars for grafana #169
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
marrouchi
force-pushed
the
PLAT-594/add-smtp-env-vars
branch
from
fb0440a
to
bfd2c25
Compare
michaelloosen
suggested changes
Nov 2, 2022
marrouchi
force-pushed
the
PLAT-594/add-smtp-env-vars
branch
from
c084744
to
36061de
Compare
marrouchi
changed the title
nour-borgi
approved these changes
Nov 21, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
1- In order to enable Grafana email alerts we need to enable SMTP and add support for configuring SMTP credentials.
2- Removed the "depends_on" property since it's ignored by docker swarm.
3- Expose grafana on port 3000 instead of 3030.
4- 2x replicas of prometheus : Both prometheus servers will scrape the same endpoints and thus (more or less) have the same data. I don't think we can use docker swarm replication though as they will need to be separate services so that Grafana only talks to one one of them else Grafana might see inconsistent data on subsequent queries (due to round robining). So we will need to define a service like prometheus and promenteus-backup as services with the exact some docker config. Then if we lose the node promenteus-main is on we can manually switch Grafana to promenteus-backup and still see historical data. The idea not to have 3 copies of Prometheus is just to unburden the metrics exporter endpoints and if we lose 2 nodes the whole cluster is stuffed anyways.
JIRA : https://jembiprojects.jira.com/browse/PLAT-594