Skip to content
This repository was archived by the owner on May 6, 2020. It is now read-only.

fix(scheduler): when user asks for limits beyond their allowance then error out faster#975

Merged
helgi merged 1 commit intodeis:masterfrom
helgi:failed_scheduling
Aug 12, 2016
Merged

fix(scheduler): when user asks for limits beyond their allowance then error out faster#975
helgi merged 1 commit intodeis:masterfrom
helgi:failed_scheduling

Conversation

@helgi
Copy link
Copy Markdown
Contributor

@helgi helgi commented Aug 11, 2016

This detect errors from the pods event stream by turning the image error handling code into something slightly more generic so we can expand it over time if need be

Events:
  FirstSeenLastSeenCountFromSubobjectPathTypeReasonMessage
  ------------------------------------------------------------
  4m1m2{default-scheduler }WarningFailedSchedulingpod (gaslit-joyrider-cmd-2657344266-tv41n) failed to fit in any node
fit failure on node (ip-172-20-0-109.us-west-2.compute.internal): Node didn't have enough resource: CPU, requested: 2000000, used: 520, capacity: 2000

I made the "failed deploy, going back to old release" contain more contextual information. This can be good but may also get to be busy... Could change it so we log the additional information but do not return it via the API

Testing Instructions

  1. Create a Deis Cluster
  2. Register an app
  3. deis pull deis/example-go
  4. deis limits:set -c cmd=2000 (this should fail)

Look at the message that comes back - For now it will not look good since the CLI or SDK (or go?) are not dealing with newlines

@helgi helgi added this to the v2.4 milestone Aug 11, 2016
@helgi helgi self-assigned this Aug 11, 2016
@helgi helgi force-pushed the failed_scheduling branch 2 times, most recently from a7bee19 to 5c52275 Compare August 12, 2016 16:11
@codecov-io
Copy link
Copy Markdown

codecov-io commented Aug 12, 2016

Current coverage is 85.57% (diff: 38.46%)

Merging #975 into master will increase coverage by 0.16%

@@             master       #975   diff @@
==========================================
  Files            28         28          
  Lines          3117       3119     +2   
  Methods           0          0          
  Messages          0          0          
  Branches        531        533     +2   
==========================================
+ Hits           2662       2669     +7   
+ Misses          309        299    -10   
- Partials        146        151     +5   

Powered by Codecov. Last update 7f19c48...63c5932

… error out faster

This detect errors from the pods event stream

Events:
  FirstSeenLastSeenCountFromSubobjectPathTypeReasonMessage
  ------------------------------------------------------------
  4m1m2{default-scheduler }WarningFailedSchedulingpod (gaslit-joyrider-cmd-2657344266-tv41n) failed to fit in any node
fit failure on node (ip-172-20-0-109.us-west-2.compute.internal): Node didn't have enough resource: CPU, requested: 2000000, used: 520, capacity: 2000
@helgi helgi force-pushed the failed_scheduling branch from 1f42fa7 to 63c5932 Compare August 12, 2016 17:00
@mboersma
Copy link
Copy Markdown
Member

$ deis limits:set -c cmd=20000 -a inward-inventor
Applying limits... Error: Unknown Error (400): {"detail":"(app::deploy): There was a problem while deploying v3 of inward-inventor-cmd. Going back to the previous release. Additional information:\npod (inward-inventor-cmd-3948341660-3fasy) failed to fit in any node\nfit failure on node (gke-mboersma-default-pool-f041a048-xxs8): Insufficient CPU\nfit failure on node (gke-mboersma-default-pool-f041a048-crig): Insufficient CPU\nfit failure on node (gke-mboersma-default-pool-f041a048-mlza): Insufficient CPU\n"}

It's pretty verbose, but informative.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants