Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rpc error: code = Unknown desc = docker does not support reopening container log files #23

Open
dyrnq opened this issue May 31, 2022 · 10 comments
Open

rpc error: code = Unknown desc = docker does not support reopening container log files #23

dyrnq opened this issue May 31, 2022 · 10 comments

Comments

@dyrnq
Copy link
Owner

dyrnq commented May 31, 2022 

May 31 10:43:19 master1 kubelet[61217]: E0531 10:43:19.425540   61217 remote_runtime.go:1134] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:19 master1 kubelet[61217]: E0531 10:43:19.425653   61217 container_log_manager.go:259] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log\": failed to reopen container log \"449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:29 master1 kubelet[61217]: E0531 10:43:29.477431   61217 remote_runtime.go:1134] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:29 master1 kubelet[61217]: E0531 10:43:29.477512   61217 container_log_manager.go:259] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log\": failed to reopen container log \"449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:39 master1 kubelet[61217]: E0531 10:43:39.523688   61217 remote_runtime.go:1134] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:39 master1 kubelet[61217]: E0531 10:43:39.523790   61217 container_log_manager.go:259] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log\": failed to reopen container log \"449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:49 master1 kubelet[61217]: E0531 10:43:49.573024   61217 remote_runtime.go:1134] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:49 master1 kubelet[61217]: E0531 10:43:49.573139   61217 container_log_manager.go:259] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log\": failed to reopen container log \"449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:59 master1 kubelet[61217]: E0531 10:43:59.617391   61217 remote_runtime.go:1134] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
May 31 10:43:59 master1 kubelet[61217]: E0531 10:43:59.617466   61217 container_log_manager.go:259] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log\": failed to reopen container log \"449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_e98f687a9c48a16fc91b3507b08b875c/kube-apiserver/0.log" containerID="449256e146e311c00812a4fbdf7e5329ee4792633f4559846ee7318e07d0cfb9"
@dyrnq
Copy link
Owner Author

dyrnq commented Oct 13, 2022 

Oct 13 10:37:44 master1 kubelet[33321]: E1013 10:37:44.923134   33321 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log\": failed to reopen container log \"2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:37:55 master1 kubelet[33321]: E1013 10:37:55.003191   33321 remote_runtime.go:1145] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:37:55 master1 kubelet[33321]: E1013 10:37:55.005104   33321 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log\": failed to reopen container log \"2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:38:05 master1 kubelet[33321]: E1013 10:38:05.053292   33321 remote_runtime.go:1145] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:38:05 master1 kubelet[33321]: E1013 10:38:05.053382   33321 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log\": failed to reopen container log \"2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:38:15 master1 kubelet[33321]: E1013 10:38:15.154938   33321 remote_runtime.go:1145] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:38:15 master1 kubelet[33321]: E1013 10:38:15.155004   33321 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log\": failed to reopen container log \"2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:38:25 master1 kubelet[33321]: E1013 10:38:25.245699   33321 remote_runtime.go:1145] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:38:25 master1 kubelet[33321]: E1013 10:38:25.245759   33321 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log\": failed to reopen container log \"2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:38:35 master1 kubelet[33321]: E1013 10:38:35.296500   33321 remote_runtime.go:1145] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"
Oct 13 10:38:35 master1 kubelet[33321]: E1013 10:38:35.297509   33321 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log\": failed to reopen container log \"2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-apiserver-master1_670e05977ae8dbc1e7bd49144daac8c8/kube-apiserver/0.log" containerID="2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1"

@dyrnq
Copy link
Owner Author

dyrnq commented Oct 13, 2022

xref Mirantis/cri-dockerd#35

@dyrnq
Copy link
Owner Author

dyrnq commented Oct 13, 2022 

du -ch /var/lib/docker/containers/*/*-json.log
8.0K	/var/lib/docker/containers/054dd623cd6bf7adafe2ef798f52c3784211a57037e98377c0e3831e5eb43a8c/054dd623cd6bf7adafe2ef798f52c3784211a57037e98377c0e3831e5eb43a8c-json.log
0	/var/lib/docker/containers/06c5f7e3816b0ca0b4c320fd888fc5104fc039bd7627bf856e513bb20b7374eb/06c5f7e3816b0ca0b4c320fd888fc5104fc039bd7627bf856e513bb20b7374eb-json.log
8.0K	/var/lib/docker/containers/19eeca71f931c6c0f05fe4a3be7935e419dc1b17fff50fb9ad034faebf77bcd7/19eeca71f931c6c0f05fe4a3be7935e419dc1b17fff50fb9ad034faebf77bcd7-json.log
4.0K	/var/lib/docker/containers/1d13a725e32e9570ed9d183c1cffb010c9e2dba3728b9b8999e29ad461d6679b/1d13a725e32e9570ed9d183c1cffb010c9e2dba3728b9b8999e29ad461d6679b-json.log
8.0K	/var/lib/docker/containers/222330711eb536ea5a2497bb660331a0afcfb5d2141e1a088f8243b3fccde1da/222330711eb536ea5a2497bb660331a0afcfb5d2141e1a088f8243b3fccde1da-json.log
0	/var/lib/docker/containers/27a488e2adcebf74573c4fc7f2450bf3dc4954921e094323744d2e13585ff3bd/27a488e2adcebf74573c4fc7f2450bf3dc4954921e094323744d2e13585ff3bd-json.log
0	/var/lib/docker/containers/28ea42f81748823f91c30979bddd9717593a751dabc8454986561f903ef6e532/28ea42f81748823f91c30979bddd9717593a751dabc8454986561f903ef6e532-json.log
49M	/var/lib/docker/containers/2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1/2cf170c66ff939bd79f82cac0c0d9f755ee3560802da9e303b98cb256b2aaca1-json.log
4.0K	/var/lib/docker/containers/3408e75fb16fab729b57a3ebbd593c62e1ee1b86b564d141033dc894458a7743/3408e75fb16fab729b57a3ebbd593c62e1ee1b86b564d141033dc894458a7743-json.log
0	/var/lib/docker/containers/3da81688306de6afb19af07f02a1bb0860fbaa3b62c938e55554789a99d98a19/3da81688306de6afb19af07f02a1bb0860fbaa3b62c938e55554789a99d98a19-json.log
72K	/var/lib/docker/containers/498638d10c26bc0447e290655e814de4c32d259ec497bd06a656a910516fc86f/498638d10c26bc0447e290655e814de4c32d259ec497bd06a656a910516fc86f-json.log
0	/var/lib/docker/containers/598f7a4d0aa1741891b2bdf1dca80fc25f78b96543bb7e78e095915b86e80511/598f7a4d0aa1741891b2bdf1dca80fc25f78b96543bb7e78e095915b86e80511-json.log
4.0K	/var/lib/docker/containers/718384359f6e03cbe58a1ad02261aae691d6459dd42608814136f5ace6622ead/718384359f6e03cbe58a1ad02261aae691d6459dd42608814136f5ace6622ead-json.log
0	/var/lib/docker/containers/9495438663cd2bbb77f6f58dc816b79d2a6db7ae32e0aef43ba21d86d844238a/9495438663cd2bbb77f6f58dc816b79d2a6db7ae32e0aef43ba21d86d844238a-json.log
0	/var/lib/docker/containers/a720299627230f7361c4362d7807c744820ec3dd76108de7315137a325bc9d25/a720299627230f7361c4362d7807c744820ec3dd76108de7315137a325bc9d25-json.log
0	/var/lib/docker/containers/b36a9e80278c5556c460b09db4928ec2e487d6bd2514b52f8317b05adae7ff33/b36a9e80278c5556c460b09db4928ec2e487d6bd2514b52f8317b05adae7ff33-json.log
328K	/var/lib/docker/containers/bc55d8f8d66b7de59790043a4ba23ff7093c5e20b350c06f08e7e4a5e65cb0b9/bc55d8f8d66b7de59790043a4ba23ff7093c5e20b350c06f08e7e4a5e65cb0b9-json.log
276K	/var/lib/docker/containers/c81d8adbc8c6584764787b5b49fc1296265eab305a6fa57ce178eafc8ed12364/c81d8adbc8c6584764787b5b49fc1296265eab305a6fa57ce178eafc8ed12364-json.log
0	/var/lib/docker/containers/cb54179e8d951108eb0b9122b7eca28ba6e8a14fb775f1c7bdc91cb88fe6bdb3/cb54179e8d951108eb0b9122b7eca28ba6e8a14fb775f1c7bdc91cb88fe6bdb3-json.log
0	/var/lib/docker/containers/e01d8a93392dc19d2cddc3bd225a01b243710e0a44bffc1d8d03ac12d4d1ff8d/e01d8a93392dc19d2cddc3bd225a01b243710e0a44bffc1d8d03ac12d4d1ff8d-json.log
6.8M	/var/lib/docker/containers/e82d09e2764e826849ff7827c4572085bb5157270a228b300563b1524bc0e330/e82d09e2764e826849ff7827c4572085bb5157270a228b300563b1524bc0e330-json.log
57M	total

@dyrnq
Copy link
Owner Author

dyrnq commented Oct 13, 2022 

docker version
Client:
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 22:56:42 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:45 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309f
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
cri-dockerd --version
cri-dockerd 0.2.6 (d8accf7)
kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.6", GitCommit:"b39bf148cd654599a52e867485c02c4f9d28b312", GitTreeState:"clean", BuildDate:"2022-09-21T13:19:24Z", GoVersion:"go1.18.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.6", GitCommit:"b39bf148cd654599a52e867485c02c4f9d28b312", GitTreeState:"clean", BuildDate:"2022-09-21T13:12:04Z", GoVersion:"go1.18.6", Compiler:"gc", Platform:"linux/amd64"}

@dyrnq
Copy link
Owner Author

dyrnq commented Oct 13, 2022 •
edited
Loading

maybe
kubernetes/kubernetes#59898
kubernetes/kubernetes#110630

@dyrnq
Copy link
Owner Author

dyrnq commented Oct 13, 2022

conflict --container-log-max-size with log-opts in daemon.json
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

--container-log-max-size string     Default: 10Mi
cat /etc/docker/daemon.json 
{
    "log-level": "info",
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "100m",
	"max-file": "5"
    }
}

@dyrnq
Copy link
Owner Author

dyrnq commented Oct 13, 2022

Final solution
pass --container-log-max-size=100Mi to kubelet args and systemctl daemon-reload && systemctl restart kubelet .
let --container-log-max-size same with max-size in log-opts of /etc/docker/daemon.json !

@dyrnq dyrnq closed this as completed Oct 13, 2022
@dyrnq
Copy link
Owner Author

dyrnq commented Oct 13, 2022

Maybe use --feature-gates=CRIContainerLogRotation=false to close this feature.
But CRIContainerLogRotation has removed in https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates-removed/

@dyrnq dyrnq reopened this Oct 13, 2022
@dyrnq
Copy link
Owner Author

dyrnq commented Apr 11, 2023

xref containerd/containerd#4830

@dyrnq
Copy link
Owner Author

dyrnq commented Apr 11, 2023

docker(conf log-opts in daemon.json) ---> containerd (no need config)

kubelet(conf containerLogMaxSize and containerLogMaxFiles in KubeletConfiguration ) ---> containerd (no need config)

nerdctl run (pass --log-opt) ---> containerd (no need config)
eg

nerdctl run --log-driver=json-file --log-opt=max-size=100m --log-opt=max-file=10 -d --name mynginx nginx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@dyrnq