Add new menu option for default settings with 4 GPUs #333
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I have successfully created and destroyed a cluster in this branch. |
MekWarrior
reviewed
Apr 24, 2020
MekWarrior
reviewed
Apr 24, 2020
MekWarrior
reviewed
Apr 24, 2020
willgraf
added a commit
that referenced
this pull request
May 22, 2020
* Feature/cicd (#268) * Set all custom charts image.pullPolicy to IfNotPresent (#258) * setting TRANSLATE_COLON_NOTATION=false by default (#289) * Update Getting Started (#287) * Update PULL_REQUEST.md for grammar (#292) * Use gomplate to template patches/hpa.yaml. (#293) * default account has 100 firewalls, not 200. (#297) * Update all documentation and links to reference kiosk-console instead of kiosk (#295) * Use yq and helmfile build to dynamically deploy helm charts based on release name. (#300) * Upgrade the openvpn chart to latest 4.2.1. (#301) * Change CLUSTER in Makefile to kiosk-console to fix binary name issue. (#302) * update raw.gif and tracked.gif with new nearly perfect gif (#303) * Update default values for tf-serving (#306) * Update Redis to the latest helm chart before they migrate to bitnami (#307) * Update autoscaler to 0.4.1 (#308) * Update redis-janitor to 0.3.1 (#309) * Update frontend to 0.4.1. (#310) * Update OpenVPN command for version 4.2.1 (#313) * Upgrade consumers to 0.5.1 and update models to DeepWatershed. (#311) * Set no-appendfsync-on-rewrite=yes to prevent Redis latency issues during AOF fsync (#316) * Install yq in install_script.sh (#319) * Use 4 random digits for cluster names. (#318) * update to latest version of the frontend (#322) * Change default consumer machine type to n1-standard-2 (#323) * Upgrade benchmarking to 0.2.4 and fix for Deep Watershed models (#324) * Use GRAFANA_PASSWORD env var to override the default grafana password. (#325) * Update Getting Started docs with new user feedback (#321) * Add basic unit tests (#326) * Use the docker container to run integration tests. (#327) * Warn users if bucket's region and cluster's region do not match (#329) * Bump benchmarking to latest 0.2.5 release (#331) * Add Logo Banner and Update README (#332) * Add new menu option for default settings with 4 GPUs (#333) * Update HPA target to 2 keys per zip consumer pod. (#334) * Bump consumers to version 0.5.2 (#336) * Update consumer and benchmarking versions (#337) * Bump redis-janitor to 0.3.2 to fix empty key bug. (#339) * bump benchmarking to 0.3.1 to fix No route to host bug. (#341) * Allow users to select which zone(s) to deploy the cluster (#340) * Pin KUBERNETES_VERSION to 1.14. (#346) * Fix bug indexing into last array element of valid_zones. (#348) * Fix logs to indicate finality and be less redundant. (#351) * If KUBERNETES_VERSION is 1.14, warn user of potential future version removal (#352) Co-authored-by: dylanbannon <dylanbannon5@gmail.com> Co-authored-by: MekWarrior <moen.erick@gmail.com>
willgraf
added a commit
that referenced
this pull request
May 23, 2020
* add new menu option for 4 GPU default (with n1-standard-4 default node type) * add troubleshooting section for Prometheus OOMKilled issue. * update the confirm display to include node information. * update docs with new menu options.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new Default configuration option for 4 GPUs. This sets the default node type to
n1-standard-4to help remove issues where Prometheus will get OOMKilled because the node it lives on does not have enough memory.n1-standard-4has shown to work for high-volume jobs with up to 8 GPUs. (Fixes #296).The troubleshooting section has also been updated to assist users in understanding this issue and how to resolve it.
Here is what the new menu options look like: