Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NO FailedPostStartHook Error Message After Upgrade to 1.7.8 #54671

Closed
lovejoy opened this issue Oct 27, 2017 · 13 comments · Fixed by #54739
Closed

NO FailedPostStartHook Error Message After Upgrade to 1.7.8 #54671

lovejoy opened this issue Oct 27, 2017 · 13 comments · Fixed by #54739
Labels
area/kubelet kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@lovejoy
Copy link
Contributor

lovejoy commented Oct 27, 2017

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened:
NO FailedPostStartHook Error Message After Upgrade to 1.7.8
When we use 1.6.3 , We can get message like Error syncing pod, skipping: failed to "StartContainer" for "xxxx" with PostStart handler: command '/xxxxxx' exited with 1: Hook do errexit, this is stderr : "PostStart Hook Failed" From Event, But After Upgrade to 1.7.8 ,the FailedPostStartHook Event always have no message

What you expected to happen:
we need the message to find out the poststarthook failed reason

How to reproduce it (as minimally and precisely as possible):
Use PostStartHook And Fail It

Anything else we need to know?:

a02f10f
pkg/kubelet/kuberuntime/kuberuntime_container.go
line 123 ,dont delete it

Environment:

  • Kubernetes version (use kubectl version):
    Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.8", GitCommit:"bc6162cc70b4a39a7f39391564e0dd0be60b39e9", GitTreeState:"clean", BuildDate:"2017-10-05T06:54:19Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.8", GitCommit:"bc6162cc70b4a39a7f39391564e0dd0be60b39e9", GitTreeState:"clean", BuildDate:"2017-10-05T06:35:40Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):CentOS Linux 7 (Core)
  • Kernel (e.g. uname -a):4.13.7-1.el7.elrepo.x86_64
  • Install tools:
  • Others:
@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Oct 27, 2017
@k8s-github-robot k8s-github-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Oct 27, 2017
@lovejoy
Copy link
Contributor Author

lovejoy commented Oct 27, 2017

/area kubelet
/sig node

@k8s-ci-robot k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. labels Oct 27, 2017
@lovejoy
Copy link
Contributor Author

lovejoy commented Oct 27, 2017

@derekwaynecarr

@k8s-github-robot k8s-github-robot removed the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Oct 27, 2017
@xiangpengzhao
Copy link
Contributor

I think this was fixed from 1.8.
see: 9fcc25d

@lovejoy
Copy link
Contributor Author

lovejoy commented Oct 27, 2017

@xiangpengzhao
the error message is in handlerErr ,so 9fcc25d didnot fix this

@OJezu
Copy link

OJezu commented Oct 27, 2017

I've just updated to kubernetes 1.8 and still no error message in kubectl describe pod <pod>

However, I've never seen it there, as I've been using k8s for a few days, some of which were spent on searching how to debug hooks...

Anyway client and server version:

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T19:48:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4", GitTreeState:"dirty", BuildDate:"2017-10-17T15:09:55Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Server is running as a part of minikube

@xiangpengzhao
Copy link
Contributor

@lovejoy @OJezu thanks for pointing it out! I send PR #54739. PTAL.

@lovejoy
Copy link
Contributor Author

lovejoy commented Oct 30, 2017

@xiangpengzhao We also require the error message included in the Event, Would you please add it?

@xiangpengzhao
Copy link
Contributor

xiangpengzhao commented Oct 30, 2017

Seems like the handlerErr is already included in the msg according to m.runner.Run implementation.

https://github.com/kubernetes/kubernetes/blob/3dfaa086b39ac18ae44dfc1614350f7eb1565fa4/pkg/kubelet/kuberuntime/kuberuntime_container.go#L161

@lovejoy
Copy link
Contributor Author

lovejoy commented Oct 30, 2017

the m.runner.Run in master seems same as release-1.7 ,And I found the msg is empty in release-1.7,so the handlerErr may not included in msg.

@xiangpengzhao
Copy link
Contributor

ping @sjenning and @derekwaynecarr for confirmation. thanks!

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 28, 2018
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 27, 2018
huawei-k8s-bot pushed a commit to huawei-cloudnative/kubernetes that referenced this issue Feb 27, 2018
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add FailedPostStartHook error message.

**What this PR does / why we need it**:

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes kubernetes#54671

**Special notes for your reviewer**:
/cc @derekwaynecarr 
cc @lovejoy @OJezu

**Release note**:

```release-note
NONE
```
@lovejoy
Copy link
Contributor Author

lovejoy commented Jun 4, 2018

JUST FOR SOMEBODY WHO CARE ABOUT THE PostStartHook Event Message
"the msg is empty " have really already been fixed in #56959
It's in release 1.7.12 but it seems the release note didn't mention this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/kubelet kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants