Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up an AlertManager with slack receiver support #317

Merged
merged 76 commits into from
Oct 22, 2020

Conversation

willgraf
Copy link
Contributor

@willgraf willgraf commented Apr 1, 2020

This PR udpates the prometheus stack versions to the latest before migrating to prometheus-community, and installs/configures the prometheus alert manager using three new environment variables:

  • ALERTMANAGER_INSTALLED: Whether to install the Prometheus Alert Manager during prometheus-operator installation, defaults to true.
  • KUBE_PROMETHEUS_ALERT_MANAGER_SLACK_WEBHOOK_URL: The Slack webhook URL to send Slack alerts. This is required to integrate with slack.
  • KUBE_PROMETHEUS_ALERT_MANAGER_SLACK_CHANNEL: The Slack channel to send the alerts to, defaults to #alerts.

Additionally, many custom alerts have been added thanks to awesome Prometheus alerts.

dylanbannon and others added 23 commits March 4, 2020 11:37
Lots of stuff in this squashed commit:

* Integration tests for cluster deployment on GKE were added. Part of this process was paring down a minimal build environment with 1) a minimal set of environmental variables, 2) a permanent service account with a minimal set of permissions, and 3) a VM environment with a minimal set of dependencies (see `.travis/install_script.sh`).

* Integration testing utilizes as much production code as possible (almost total overlap).

* All testing is currently done in TravisCI and has been optimized to minimize build time.

* Integration tests only run on merges into master, but can be triggered at any time by placing [build-integration-tests] in a commit message.

* Testing of ELK-enabled clusters is currently turned off, due to the potential for dangling resources upon cluster failure. ELK-enabled integration tests can be triggered by including both [build-integration-tests] and [test-elk] in the same commit message.

* The project's pull request template now "requires" integration tests to be run, although it's up to the reviewer to enforce this requirement.

Minor:

* Added CONF_PATH_PREFIX variable for compatibility between production environment and testing VMs.

* Replaced shopt calls with a new system for toggling ELK deployment. (ELK_DEPLOYMENT_TOGGLE = "" by default and any other value turns ELK deployment on.)
* remove chart quotes and set all image.pullPolicy to IfNotPresent

* remove pullPolicy from helmfile, no need to override by default
* remove numbers and update version

* bold headline in rst

* copyedited for clarity and  correctness

* fix formatting

* change dev content to note

* change formatting in Usage

* update license badge with correct target

* update README cmd version to 1.1.0

* crop images to be the same size
* use `gomplate` for templating `hpa.yaml`.

* change reference in the docs as well.

* remove unused `GPU_MAX_TIMES_X` and `GPU_MAX_DIVIDED_BY_X` env vars.
… of kiosk (#295)

* Change all references to vanvalenlab/kiosk to vanvalenlab/kiosk-console.

* Change run command from `kiosk` to `kiosk-console`.

* Update architecture diagram to include kiosk-console.

* Clarify differences between DeepCell Kiosk and kiosk-console.
The previous chart version is 1.1.0, which I think was intended to be the app version instead.
* change GPU_MIN_NODES to GPU_NODE_MIN_SIZE and GPU_MAX_NODES to GPU_NODE_MAX_SIZE
* Change default value for GCP_PREDICTION_GPU_TYPE to nvidia-tesla-t4

* GPU only needs to be in limits, not requests
* remove serviceAccount default value from helmfile

* update image to 0.4.1

* reduce CPU request to 50m

* unstringify helmfile
* Upgrade redis-janitor image to 0.3.1
* Unstringify helmfile
* Remove unused serviceAccount in helmfile
Includes dataLabel field in the backend API predict spec
* Update prometheus-operator, prometheus-adapter, and promethes-redis-exporter helm charts and remove stale default values

* Relabel redis-exporter with `deployment=$QUEUE-consumer` and change key to be `$QUEUE`

* Rename zip-consumer to segmentation-zip-consumer to match labels.

* Using .75 instead of .9 for backoff coefficient.
* Updating openvpn means updating its documentation as well!
Catch stable branch up to master
Co-Authored-By: willgraf <7930703+willgraf@users.noreply.github.com>
Co-Authored-By: MekWarrior <moen.erick@gmail.com>
Co-Authored-By: willgraf <7930703+willgraf@users.noreply.github.com>
Co-Authored-By: MekWarrior <moen.erick@gmail.com>
@willgraf willgraf added enhancement New feature or request wip labels Apr 1, 2020
@willgraf willgraf requested a review from MekWarrior April 1, 2020 00:03
@willgraf willgraf changed the base branch from stable to benchmarks May 22, 2020 22:26
@willgraf willgraf changed the base branch from benchmarks to stable May 22, 2020 22:28
dylanbannon and others added 2 commits May 22, 2020 17:16
Lots of stuff in this squashed commit:

* Integration tests for cluster deployment on GKE were added. Part of this process was paring down a minimal build environment with 1) a minimal set of environmental variables, 2) a permanent service account with a minimal set of permissions, and 3) a VM environment with a minimal set of dependencies (see `.travis/install_script.sh`).

* Integration testing utilizes as much production code as possible (almost total overlap).

* All testing is currently done in TravisCI and has been optimized to minimize build time.

* Integration tests only run on merges into master, but can be triggered at any time by placing [build-integration-tests] in a commit message.

* Testing of ELK-enabled clusters is currently turned off, due to the potential for dangling resources upon cluster failure. ELK-enabled integration tests can be triggered by including both [build-integration-tests] and [test-elk] in the same commit message.

* The project's pull request template now "requires" integration tests to be run, although it's up to the reviewer to enforce this requirement.

Minor:

* Added CONF_PATH_PREFIX variable for compatibility between production environment and testing VMs.

* Replaced shopt calls with a new system for toggling ELK deployment. (ELK_DEPLOYMENT_TOGGLE = "" by default and any other value turns ELK deployment on.)
* remove chart quotes and set all image.pullPolicy to IfNotPresent

* remove pullPolicy from helmfile, no need to override by default
@willgraf willgraf removed the wip label Oct 19, 2020
@willgraf willgraf merged commit 6380eb0 into stable Oct 22, 2020
@willgraf willgraf deleted the willgraf/alertmanager branch October 22, 2020 18:05
willgraf added a commit that referenced this pull request Dec 16, 2020
* Support TLS traffic with cert-manager. (#357)

* Fix frontend ingress issue when no hosts are provided. (#381)

* Template frontend ingress annotations using `CERTIFICATE_MANAGER_ENABLED` (#383)

* Create tf-serving configuration files using an initContainer. (#382)

* Fix whitespace issue in tasks/Makefile.kubectl (#386)

* Bump openvpn to 4.2.3 (#385)

* Upgrade certificate manager to version 1.0.3 (#384)

* Add screenshot of successfully created cluster to docs. (#388)

* Set up an AlertManager with slack receiver support (#317)

* Install procps to give access to sysctl. (#390)

* Migrate CI/CD from TravisCI to GitHub Actions (#394)

* Change the redis helm chart repo to bitnami (#393)

* Upgrade tf-serving chart to 0.3.0 for application version 0.4.0 (#392)

* Move the frontend HPA definition into the helm chart. (#395)

* Move the tf-serving HPA into the helm chart. (#396)

* Move redis-consumer HPA into the helm chart. (#397)

* Remove deprecated and unused charts (#398)

* Migrate stable helm chart repo to archived URL. (#399)

* Destroy the secret and remove the key from the DNS solver SA in a new task: `gke/destroy/certificate-manager-secret` (fixes #391).

* Use GCP_SERVICE_ACCOUNT for DNS resolution (#401)

* Clean up docs and test them with new GitHub Action workflow (#402)

* Add code-formatted filename to list of files to change (#403)

* Update ELK stack helmfiles (#380)

* Move the prometheus-redis-exporter script to a chart using incubator/raw. (#405)

* Use `kubectl del pvc` instead of deleting all pds with the cluster name. (#406)

* Update helmfile defaults for faster helm deployments. (#404)

* Skip gke/destroy/node-pools during cluster teardown. (#407)

* Update docs to reflect the pending 1.4.0 release. (#408)

* Bump redis-consumer version to 0.8.3 (#409)

* Run integration tests on all PRs to master OR if they have the commit message. (#411)

* Remove helm defaults for ELK helmfiles (#413)

Co-authored-by: Morgan Schwartz <msschwartz21@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants