-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Set up an AlertManager with slack receiver support #317
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Lots of stuff in this squashed commit: * Integration tests for cluster deployment on GKE were added. Part of this process was paring down a minimal build environment with 1) a minimal set of environmental variables, 2) a permanent service account with a minimal set of permissions, and 3) a VM environment with a minimal set of dependencies (see `.travis/install_script.sh`). * Integration testing utilizes as much production code as possible (almost total overlap). * All testing is currently done in TravisCI and has been optimized to minimize build time. * Integration tests only run on merges into master, but can be triggered at any time by placing [build-integration-tests] in a commit message. * Testing of ELK-enabled clusters is currently turned off, due to the potential for dangling resources upon cluster failure. ELK-enabled integration tests can be triggered by including both [build-integration-tests] and [test-elk] in the same commit message. * The project's pull request template now "requires" integration tests to be run, although it's up to the reviewer to enforce this requirement. Minor: * Added CONF_PATH_PREFIX variable for compatibility between production environment and testing VMs. * Replaced shopt calls with a new system for toggling ELK deployment. (ELK_DEPLOYMENT_TOGGLE = "" by default and any other value turns ELK deployment on.)
* remove chart quotes and set all image.pullPolicy to IfNotPresent * remove pullPolicy from helmfile, no need to override by default
* remove numbers and update version * bold headline in rst * copyedited for clarity and correctness * fix formatting * change dev content to note * change formatting in Usage * update license badge with correct target * update README cmd version to 1.1.0 * crop images to be the same size
* use `gomplate` for templating `hpa.yaml`. * change reference in the docs as well. * remove unused `GPU_MAX_TIMES_X` and `GPU_MAX_DIVIDED_BY_X` env vars.
… of kiosk (#295) * Change all references to vanvalenlab/kiosk to vanvalenlab/kiosk-console. * Change run command from `kiosk` to `kiosk-console`. * Update architecture diagram to include kiosk-console. * Clarify differences between DeepCell Kiosk and kiosk-console.
The previous chart version is 1.1.0, which I think was intended to be the app version instead.
* change GPU_MIN_NODES to GPU_NODE_MIN_SIZE and GPU_MAX_NODES to GPU_NODE_MAX_SIZE
* Change default value for GCP_PREDICTION_GPU_TYPE to nvidia-tesla-t4 * GPU only needs to be in limits, not requests
* remove serviceAccount default value from helmfile * update image to 0.4.1 * reduce CPU request to 50m * unstringify helmfile
* Upgrade redis-janitor image to 0.3.1 * Unstringify helmfile * Remove unused serviceAccount in helmfile
Includes dataLabel field in the backend API predict spec
* Update prometheus-operator, prometheus-adapter, and promethes-redis-exporter helm charts and remove stale default values * Relabel redis-exporter with `deployment=$QUEUE-consumer` and change key to be `$QUEUE` * Rename zip-consumer to segmentation-zip-consumer to match labels. * Using .75 instead of .9 for backoff coefficient.
* Updating openvpn means updating its documentation as well!
Catch stable branch up to master
Co-Authored-By: willgraf <7930703+willgraf@users.noreply.github.com> Co-Authored-By: MekWarrior <moen.erick@gmail.com>
Co-Authored-By: willgraf <7930703+willgraf@users.noreply.github.com> Co-Authored-By: MekWarrior <moen.erick@gmail.com>
Lots of stuff in this squashed commit: * Integration tests for cluster deployment on GKE were added. Part of this process was paring down a minimal build environment with 1) a minimal set of environmental variables, 2) a permanent service account with a minimal set of permissions, and 3) a VM environment with a minimal set of dependencies (see `.travis/install_script.sh`). * Integration testing utilizes as much production code as possible (almost total overlap). * All testing is currently done in TravisCI and has been optimized to minimize build time. * Integration tests only run on merges into master, but can be triggered at any time by placing [build-integration-tests] in a commit message. * Testing of ELK-enabled clusters is currently turned off, due to the potential for dangling resources upon cluster failure. ELK-enabled integration tests can be triggered by including both [build-integration-tests] and [test-elk] in the same commit message. * The project's pull request template now "requires" integration tests to be run, although it's up to the reviewer to enforce this requirement. Minor: * Added CONF_PATH_PREFIX variable for compatibility between production environment and testing VMs. * Replaced shopt calls with a new system for toggling ELK deployment. (ELK_DEPLOYMENT_TOGGLE = "" by default and any other value turns ELK deployment on.)
* remove chart quotes and set all image.pullPolicy to IfNotPresent * remove pullPolicy from helmfile, no need to override by default
same functionality, fewer warnings.
The current version (8.12.3) has issues with helm3. The upgrade is the final version published in helm/charts.
More helpful Slack messages.
…v var is not empty.
CPUThrottlingHigh and AggregatedAPIDown have known issues making them spammy.
PrometheusTemplateTextExpansionFailures and PrometheusAlertmanagerNotificationFailing
Using cloudposse defaults.
Using cloudposse settings.
willgraf
added a commit
that referenced
this pull request
Dec 16, 2020
* Support TLS traffic with cert-manager. (#357) * Fix frontend ingress issue when no hosts are provided. (#381) * Template frontend ingress annotations using `CERTIFICATE_MANAGER_ENABLED` (#383) * Create tf-serving configuration files using an initContainer. (#382) * Fix whitespace issue in tasks/Makefile.kubectl (#386) * Bump openvpn to 4.2.3 (#385) * Upgrade certificate manager to version 1.0.3 (#384) * Add screenshot of successfully created cluster to docs. (#388) * Set up an AlertManager with slack receiver support (#317) * Install procps to give access to sysctl. (#390) * Migrate CI/CD from TravisCI to GitHub Actions (#394) * Change the redis helm chart repo to bitnami (#393) * Upgrade tf-serving chart to 0.3.0 for application version 0.4.0 (#392) * Move the frontend HPA definition into the helm chart. (#395) * Move the tf-serving HPA into the helm chart. (#396) * Move redis-consumer HPA into the helm chart. (#397) * Remove deprecated and unused charts (#398) * Migrate stable helm chart repo to archived URL. (#399) * Destroy the secret and remove the key from the DNS solver SA in a new task: `gke/destroy/certificate-manager-secret` (fixes #391). * Use GCP_SERVICE_ACCOUNT for DNS resolution (#401) * Clean up docs and test them with new GitHub Action workflow (#402) * Add code-formatted filename to list of files to change (#403) * Update ELK stack helmfiles (#380) * Move the prometheus-redis-exporter script to a chart using incubator/raw. (#405) * Use `kubectl del pvc` instead of deleting all pds with the cluster name. (#406) * Update helmfile defaults for faster helm deployments. (#404) * Skip gke/destroy/node-pools during cluster teardown. (#407) * Update docs to reflect the pending 1.4.0 release. (#408) * Bump redis-consumer version to 0.8.3 (#409) * Run integration tests on all PRs to master OR if they have the commit message. (#411) * Remove helm defaults for ELK helmfiles (#413) Co-authored-by: Morgan Schwartz <msschwartz21@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR udpates the prometheus stack versions to the latest before migrating to prometheus-community, and installs/configures the prometheus alert manager using three new environment variables:
ALERTMANAGER_INSTALLED
: Whether to install the Prometheus Alert Manager during prometheus-operator installation, defaults to true.KUBE_PROMETHEUS_ALERT_MANAGER_SLACK_WEBHOOK_URL
: The Slack webhook URL to send Slack alerts. This is required to integrate with slack.KUBE_PROMETHEUS_ALERT_MANAGER_SLACK_CHANNEL
: The Slack channel to send the alerts to, defaults to#alerts
.Additionally, many custom alerts have been added thanks to awesome Prometheus alerts.