New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubernetes-worker Juju charm: "pause" action can't succeed in default configuration #44392

Closed
fghaas opened this Issue Apr 12, 2017 · 4 comments

Comments

Projects
None yet
2 participants
@fghaas

fghaas commented Apr 12, 2017

Is this a request for help?
No

What keywords did you search in Kubernetes issues before filing this one?
kubernetes-worker charm

Is this a BUG REPORT or FEATURE REQUEST?
Bug Report

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.3", GitCommit:"029c3a408176b55c30846f0faedf56aae5992e9b", GitTreeState:"clean", BuildDate:"2017-02-23T22:48:32Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.3", GitCommit:"029c3a408176b55c30846f0faedf56aae5992e9b", GitTreeState:"clean", BuildDate:"2017-02-23T22:28:16Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration: Libvirt VMs
  • OS: Ubuntu 16.04
  • Kernel: 4.4.0-21-generic
  • Install tools: Juju 2.1.2-xenial-amd64, cs:~containers/kubernetes-worker-14

What happened:
juju run-action kubernetes-worker/0 pause leads to immediate action failure. This action runs kubectl drain for the node that kubernetes-worker/0 is deployed to, which is bound to fail:

kubectl drain daisy
node "daisy" cordoned
error: pods with local storage (use --delete-local-data to override): monitoring-influxdb-grafana-v4-6218d

This leaves the node in a cordoned, undrained state.

What you expected to happen:
One of three options:

  • The pause action should come with a parameter enabling kubectl drain --delete-local-data, or
  • CDK should not be deploying monitoring-influxdb-grafana (if other options are available), or
  • CDK should automatically reconfigure monitoring-influxdb-grafana to run off a Ceph RBD volume if kubernetes-master is linked to ceph-mon.

How to reproduce it (as minimally and precisely as possible):

  • Deploy CDK.
  • Run juju run-action kubernetes-worker/0 pause and note the action UUID
  • Run juju show-action-status on that UUID
  • Observe the action's failed state:
actions:
- id: ac1875a0-319b-47c1-8f18-52b02b313bb2
  status: failed
  unit: kubernetes-worker/0

@fghaas fghaas changed the title from kubernetes-worker charm: "pause" action can't succeed in default configuration to kubernetes-worker Juju charm: "pause" action can't succeed in default configuration Apr 12, 2017

@lazypower

This comment has been minimized.

Show comment
Hide comment
@lazypower

lazypower Apr 25, 2017

Member

@fghaas Good find on this one. I had taken a stab at this before but i've since lost the PR and didn't track it to completion.

I'll circle back on this one this week and add a force option to pass --delete-local-data for the immediate resolve, and evaluate the suggestions for a more robust solution moving forward.

Member

lazypower commented Apr 25, 2017

@fghaas Good find on this one. I had taken a stab at this before but i've since lost the PR and didn't track it to completion.

I'll circle back on this one this week and add a force option to pass --delete-local-data for the immediate resolve, and evaluate the suggestions for a more robust solution moving forward.

@fghaas

This comment has been minimized.

Show comment
Hide comment
@fghaas

fghaas Apr 25, 2017

@chuckbutler Got it, thanks for the update!

fghaas commented Apr 25, 2017

@chuckbutler Got it, thanks for the update!

lazypower added a commit to lazypower/kubernetes that referenced this issue Apr 25, 2017

Closes kubernetes#44392
Fix the pause action with regard to the new behavior where
--delete-local-data=false by default. Historically --force was all that
was required, this flag has changed to be more descriptive of the
actions it's taking.

k8s-merge-robot added a commit that referenced this issue Apr 26, 2017

Merge pull request #44931 from chuckbutler/worker-pause-action
Automatic merge from submit-queue (batch tested with PRs 44931, 44808)

Closes #44392

**What this PR does / why we need it**:

Fix the pause action with regard to the new behavior where
--delete-local-data=false by default. Historically --force was all that
was required, this flag has changed to be more descriptive of the
actions it's taking.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #44392


**Release note**:

```release-note
Added support to the pause action in the kubernetes-worker charm for new flag --delete-local-data
```
@lazypower

This comment has been minimized.

Show comment
Hide comment
@lazypower

lazypower Apr 26, 2017

Member

@fghaas fix is committed and will go with our next release, projected next Thursday. It won't succeed by default as wildly deleting data is something the user should be concerned about and make the call. But the charm action now has the appropriate flags and better-documented action description.

juju actions --format=yaml --schema kubernetes-worker
Member

lazypower commented Apr 26, 2017

@fghaas fix is committed and will go with our next release, projected next Thursday. It won't succeed by default as wildly deleting data is something the user should be concerned about and make the call. But the charm action now has the appropriate flags and better-documented action description.

juju actions --format=yaml --schema kubernetes-worker
@fghaas

This comment has been minimized.

Show comment
Hide comment
@fghaas

fghaas Apr 26, 2017

@chuckbutler Thanks! And I do agree about the cautious default.

fghaas commented Apr 26, 2017

@chuckbutler Thanks! And I do agree about the cautious default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment