Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

k3s node becomes unresponsive with error InvalidDiskCapacity #9962

Closed
khalidkhushal opened this issue Apr 17, 2024 · 1 comment
Closed

k3s node becomes unresponsive with error InvalidDiskCapacity #9962

khalidkhushal opened this issue Apr 17, 2024 · 1 comment

Comments

@khalidkhushal
Copy link

Hi,

I have installed k3s cluster and node was running fine but after installing a deployment node became unresponsive with following details.

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests               Limits
  --------           --------               ------
  cpu                10650m (44%)           89600m (373%)
  memory             20870437273600m (62%)  59245382860800m (176%)
  ephemeral-storage  0 (0%)                 0 (0%)
  hugepages-1Gi      0 (0%)                 0 (0%)
  hugepages-2Mi      0 (0%)                 0 (0%)
  nvidia.com/gpu     2                      2
Events:
  Type     Reason                   Age    From     Message
  ----     ------                   ----   ----     -------
  Normal   Starting                 11m    kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      11m    kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  11m    kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 11m    kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      11m    kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  11m    kubelet  Updated Node Allocatable limit across pods
  Normal   NodeAllocatableEnforced  11m    kubelet  Updated Node Allocatable limit across pods
  Warning  InvalidDiskCapacity      11m    kubelet  invalid capacity 0 on image filesystem
  Normal   Starting                 11m    kubelet  Starting kubelet.
  Normal   Starting                 10m    kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      10m    kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  10m    kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 10m    kubelet  Starting kubelet.
  Normal   NodeAllocatableEnforced  10m    kubelet  Updated Node Allocatable limit across pods
  Warning  InvalidDiskCapacity      10m    kubelet  invalid capacity 0 on image filesystem
  Normal   Starting                 10m    kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      10m    kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  10m    kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 10m    kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      10m    kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  10m    kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 10m    kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      10m    kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  10m    kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 10m    kubelet  Starting kubelet.
  Normal   NodeAllocatableEnforced  10m    kubelet  Updated Node Allocatable limit across pods
  Warning  InvalidDiskCapacity      10m    kubelet  invalid capacity 0 on image filesystem
  Normal   Starting                 9m50s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      9m50s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  9m50s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 9m40s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      9m40s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  9m40s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 9m29s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      9m29s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  9m29s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 9m19s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      9m19s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  9m18s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 9m8s   kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      9m8s   kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  9m8s   kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 8m58s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      8m58s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  8m57s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 8m47s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      8m47s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  8m47s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 8m37s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      8m37s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  8m36s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 8m26s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      8m26s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  8m26s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 8m16s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      8m16s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  8m15s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 8m6s   kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      8m6s   kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  8m5s   kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 7m55s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      7m55s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  7m55s  kubelet  Updated Node Allocatable limit across pods
  Normal   NodeAllocatableEnforced  7m44s  kubelet  Updated Node Allocatable limit across pods
  Warning  InvalidDiskCapacity      7m44s  kubelet  invalid capacity 0 on image filesystem
  Normal   Starting                 7m44s  kubelet  Starting kubelet.
  Normal   Starting                 7m34s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      7m34s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  7m33s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 7m23s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      7m23s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  7m23s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 7m13s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      7m13s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  7m12s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 7m2s   kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      7m2s   kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  7m2s   kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 6m51s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      6m51s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  6m51s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 6m41s  kubelet  Starting kubelet.
  Normal   NodeAllocatableEnforced  6m41s  kubelet  Updated Node Allocatable limit across pods
  Warning  InvalidDiskCapacity      6m41s  kubelet  invalid capacity 0 on image filesystem
  Warning  InvalidDiskCapacity      6m30s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  6m30s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 6m30s  kubelet  Starting kubelet.
  Normal   NodeAllocatableEnforced  6m19s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 6m19s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      6m19s  kubelet  invalid capacity 0 on image filesystem
  Normal   Starting                 6m9s   kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      6m9s   kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  6m9s   kubelet  Updated Node Allocatable limit across pods
  Warning  InvalidDiskCapacity      5m58s  kubelet  invalid capacity 0 on image filesystem
  Normal   Starting                 5m58s  kubelet  Starting kubelet.
  Normal   NodeAllocatableEnforced  5m58s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 5m48s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      5m48s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  5m47s  kubelet  Updated Node Allocatable limit across pods
  Normal   NodeAllocatableEnforced  5m37s  kubelet  Updated Node Allocatable limit across pods
  Warning  InvalidDiskCapacity      5m37s  kubelet  invalid capacity 0 on image filesystem
  Normal   Starting                 5m37s  kubelet  Starting kubelet.
  Normal   Starting                 5m27s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      5m27s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  5m26s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 5m16s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      5m16s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  5m16s  kubelet  Updated Node Allocatable limit across pods
  Normal   NodeAllocatableEnforced  5m6s   kubelet  Updated Node Allocatable limit across pods
  Warning  InvalidDiskCapacity      5m6s   kubelet  invalid capacity 0 on image filesystem
  Normal   Starting                 5m6s   kubelet  Starting kubelet.
  Normal   Starting                 4m55s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      4m55s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  4m55s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 4m44s  kubelet  Starting kubelet.
  Normal   NodeAllocatableEnforced  4m44s  kubelet  Updated Node Allocatable limit across pods
  Warning  InvalidDiskCapacity      4m44s  kubelet  invalid capacity 0 on image filesystem
  Normal   Starting                 4m40s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      4m40s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  4m39s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 4m29s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      4m29s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  4m29s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 4m18s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      4m18s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  4m18s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 4m8s   kubelet  Starting kubelet.
  Normal   NodeAllocatableEnforced  4m8s   kubelet  Updated Node Allocatable limit across pods
  Warning  InvalidDiskCapacity      4m8s   kubelet  invalid capacity 0 on image filesystem
  Normal   Starting                 3m57s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      3m57s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  3m57s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 3m47s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      3m47s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  3m47s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 3m36s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      3m36s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  3m36s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 3m26s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      3m26s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  3m25s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 3m15s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      3m15s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  3m14s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 3m4s   kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      3m4s   kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  3m3s   kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 2m53s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      2m53s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  2m53s  kubelet  Updated Node Allocatable limit across pods
  Normal   NodeAllocatableEnforced  2m42s  kubelet  Updated Node Allocatable limit across pods
  Warning  InvalidDiskCapacity      2m42s  kubelet  invalid capacity 0 on image filesystem
  Normal   Starting                 2m42s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      2m31s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  2m31s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 2m31s  kubelet  Starting kubelet.
  Normal   NodeAllocatableEnforced  2m20s  kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 2m20s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      2m20s  kubelet  invalid capacity 0 on image filesystem
  Normal   Starting                 2m10s  kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      2m10s  kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  2m9s   kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 119s   kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      119s   kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  119s   kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 109s   kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      109s   kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  108s   kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 97s    kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      97s    kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  97s    kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 87s    kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      87s    kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  87s    kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 76s    kubelet  Starting kubelet.
  Normal   NodeAllocatableEnforced  76s    kubelet  Updated Node Allocatable limit across pods
  Warning  InvalidDiskCapacity      76s    kubelet  invalid capacity 0 on image filesystem
  Warning  InvalidDiskCapacity      65s    kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  65s    kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 65s    kubelet  Starting kubelet.
  Normal   Starting                 54s    kubelet  Starting kubelet.
  Normal   NodeAllocatableEnforced  54s    kubelet  Updated Node Allocatable limit across pods
  Warning  InvalidDiskCapacity      54s    kubelet  invalid capacity 0 on image filesystem
  Normal   Starting                 44s    kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      44s    kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  44s    kubelet  Updated Node Allocatable limit across pods
  Normal   NodeAllocatableEnforced  33s    kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 33s    kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      33s    kubelet  invalid capacity 0 on image filesystem
  Normal   Starting                 23s    kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      23s    kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  22s    kubelet  Updated Node Allocatable limit across pods
  Warning  InvalidDiskCapacity      12s    kubelet  invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  12s    kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 12s    kubelet  Starting kubelet.
  Normal   NodeAllocatableEnforced  1s     kubelet  Updated Node Allocatable limit across pods
  Normal   Starting                 1s     kubelet  Starting kubelet.
  Warning  InvalidDiskCapacity      1s     kubelet  invalid capacity 0 on image filesystem

cluster is unresponsive and gives following error on kubectl describe node:

Get "https://0.0.0.0:6443/api/v1/nodes?limit=500": dial tcp 0.0.0.0:6443: connect: connection refused - error from a previous attempt: read tcp 127.0.0.1:60672->127.0.0.1:6443: read: connection reset by peer
@brandond
Copy link
Contributor

brandond commented Apr 17, 2024 •
edited
Loading

invalid capacity 0 on image filesystem is thrown every time the node starts, see #1857

The behavior you're describing indicates that the node is crashlooping for some reason. Check the journalctl logs to see why.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
K3s Development
Status: Done Issue
Milestone
No milestone
Development

No branches or pull requests
2 participants
@brandond @khalidkhushal