docker stats stopped working around a week ago

[jammin1911]

Description

was looking at my stats today and saw that docker stats stopped about a week ago. screenshot below

OS / Architecture

linux, x64, ubuntu 22.04

Beszel version

0.12.7

Installation method

Docker

Configuration

server and agent on same host

server compose:
services:
  beszel:
    image: henrygd/beszel
    container_name: beszel
    restart: unless-stopped
#    ports:
#      - 8090:8090
    volumes:
      - /opt/beszel/beszel_data:/beszel_data
    networks:
      static:
        ipv4_address: 192.168.50.11

networks:
  static:
    external: true

agent compose:
services:
  beszel-agent:
    image: henrygd/beszel-agent
    container_name: beszel-agent
    restart: unless-stopped
    network_mode: host
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./beszel_agent_data:/var/lib/beszel-agent
      # monitor other disks / partitions by mounting a folder in /extra-filesystems
      # - /mnt/disk/.beszel:/extra-filesystems/sda1:ro
    environment:
      LISTEN: 45876
      KEY: 'ssh-ed25censored'
      TOKEN: sensored
      HUB_URL: https://beszel.censored.ca

Hub Logs

beszel  | 2025/09/13 03:53:23 Server started at http://0.0.0.0:8090
beszel  | ├─ REST API:  http://0.0.0.0:8090/api/
beszel  | └─ Dashboard: http://0.0.0.0:8090/_/
beszel  | 2025/09/13 05:03:23 Server started at http://0.0.0.0:8090
beszel  | ├─ REST API:  http://0.0.0.0:8090/api/
beszel  | └─ Dashboard: http://0.0.0.0:8090/_/
beszel  | 2025/09/13 05:05:03 Server started at http://0.0.0.0:8090
beszel  | ├─ REST API:  http://0.0.0.0:8090/api/
beszel  | └─ Dashboard: http://0.0.0.0:8090/_/
beszel  | 2025/09/13 05:09:30 attempt to write a readonly database (1544)
beszel  | 2025/09/13 05:09:31 attempt to write a readonly database (1544)
beszel  | 2025/09/13 05:09:31 attempt to write a readonly database (1544)
beszel  | 2025/09/13 05:09:32 attempt to write a readonly database (1544)
beszel  | 2025/09/13 05:09:32 attempt to write a readonly database (1544)
beszel  | 2025/09/13 05:09:34 attempt to write a readonly database (1544)
beszel  | 2025/09/13 05:09:38 attempt to write a readonly database (1544)
beszel  | 2025/09/13 05:09:44 attempt to write a readonly database (1544)
beszel  | 2025/09/13 05:09:57 attempt to write a readonly database (1544)
beszel  | 2025/09/13 05:10:23 attempt to write a readonly database (1544)
beszel  | 2025/09/13 05:10:43 Server started at http://0.0.0.0:8090
beszel  | ├─ REST API:  http://0.0.0.0:8090/api/
beszel  | └─ Dashboard: http://0.0.0.0:8090/_/

Agent Logs

note there is a permission error here because i was messing around trying to fix it. for testing purposes i did chmod 777 on the data directory and it didnt fix the issue with the docker stats.

beszel-agent  | 2025/09/13 05:03:23 WARN Connection closed err=EOF
beszel-agent  | 2025/09/13 05:03:23 WARN Disconnected from hub
beszel-agent  | 2025/09/13 05:03:23 WARN WebSocket connection failed err="unexpected status code: 502"
beszel-agent  | 2025/09/13 05:03:23 INFO Starting SSH server addr=:45876 network=tcp
beszel-agent  | 2025/09/13 05:03:33 INFO WebSocket connected host=beszel.xxx.ca
beszel-agent  | 2025/09/13 05:03:33 INFO Stopping SSH server
beszel-agent  | 2025/09/13 05:05:02 WARN Connection closed err=EOF
beszel-agent  | 2025/09/13 05:05:02 WARN Disconnected from hub
beszel-agent  | 2025/09/13 05:05:02 WARN WebSocket connection failed err="unexpected status code: 502"
beszel-agent  | 2025/09/13 05:05:02 INFO Starting SSH server addr=:45876 network=tcp
beszel-agent  | 2025/09/13 05:05:12 INFO WebSocket connected host=beszel.xxx.ca
beszel-agent  | 2025/09/13 05:05:12 INFO Stopping SSH server
beszel-agent  | 2025/09/13 05:05:18 INFO Shutting down
beszel-agent  | 2025/09/13 05:05:18 WARN Connection closed err="close normal"
beszel-agent  | 2025/09/13 05:05:18 Failed to start server: remove /tmp/beszel_health: no such file or directory
beszel-agent  | 2025/09/13 05:05:18 INFO Data directory path=/var/lib/beszel-agent
beszel-agent  | 2025/09/13 05:05:18 INFO Detected root device name=sda1
beszel-agent  | 2025/09/13 05:05:18 INFO Detected network interface name=eth0 sent=23606823271 recv=19337334269
beszel-agent  | 2025/09/13 05:05:18 INFO Detected network interface name=tailscale0 sent=6462256 recv=5445567
beszel-agent  | 2025/09/13 05:05:18 INFO WebSocket connected host=beszel.xxx.ca
beszel-agent  | 2025/09/13 05:09:30 WARN Connection closed err=EOF
beszel-agent  | 2025/09/13 05:09:30 WARN Disconnected from hub
beszel-agent  | 2025/09/13 05:09:30 WARN WebSocket connection failed err="unexpected status code: 502"
beszel-agent  | 2025/09/13 05:09:30 INFO Starting SSH server addr=:45876 network=tcp
beszel-agent  | 2025/09/13 05:09:39 INFO Shutting down
beszel-agent  | 2025/09/13 05:09:39 INFO Stopping SSH server
beszel-agent  | 2025/09/13 05:09:39 Failed to start server: remove /tmp/beszel_health: no such file or directory
beszel-agent  | 2025/09/13 05:09:39 WARN Data directory not found
beszel-agent  | 2025/09/13 05:09:39 INFO Detected root device name=sda1
beszel-agent  | 2025/09/13 05:09:39 INFO Detected network interface name=eth0 sent=23607546636 recv=19338124528
beszel-agent  | 2025/09/13 05:09:39 INFO Detected network interface name=tailscale0 sent=6462576 recv=5445839
beszel-agent  | 2025/09/13 05:09:44 WARN WebSocket connection failed err="read tcp 66.xxx.xxx:33868->66.xxx.xxx.85:443: i/o timeout"
beszel-agent  | 2025/09/13 05:09:44 INFO Starting SSH server addr=:45876 network=tcp
beszel-agent  | 2025/09/13 05:09:57 WARN WebSocket connection failed err="unexpected status code: 502"
beszel-agent  | 2025/09/13 05:10:09 WARN WebSocket connection failed err="read tcp 66.xxx.xxx.xxx:45472->66.xxxx.xxx.85:443: i/o timeout"
beszel-agent  | 2025/09/13 05:10:17 WARN WebSocket connection failed err="unexpected status code: 502"
beszel-agent  | 2025/09/13 05:10:29 WARN WebSocket connection failed err="read tcp 66.xxx.xxx.5:51176->66.xxx.xxxx.xx:443: i/o timeout"
beszel-agent  | 2025/09/13 05:10:37 WARN WebSocket connection failed err="unexpected status code: 502"
beszel-agent  | 2025/09/13 05:10:38 INFO Shutting down
beszel-agent  | 2025/09/13 05:10:38 INFO Stopping SSH server
beszel-agent  | 2025/09/13 05:10:38 Failed to start server: remove /tmp/beszel_health: no such file or directory
beszel-agent  | 2025/09/13 05:10:38 INFO Data directory path=/var/lib/beszel-agent
beszel-agent  | 2025/09/13 05:10:38 INFO Detected root device name=sda1
beszel-agent  | 2025/09/13 05:10:38 INFO Detected network interface name=eth0 sent=23607735218 recv=19338407323
beszel-agent  | 2025/09/13 05:10:38 INFO Detected network interface name=tailscale0 sent=6462576 recv=5445839
beszel-agent  | 2025/09/13 05:10:41 WARN WebSocket connection failed err="unexpected status code: 502"
beszel-agent  | 2025/09/13 05:10:41 INFO Starting SSH server addr=:45876 network=tcp
beszel-agent  | 2025/09/13 05:10:51 INFO WebSocket connected host=beszel.xxx.ca
beszel-agent  | 2025/09/13 05:10:51 INFO Stopping SSH server

[svenvg93]

You might want you mask the ip's in the logs 😉😅

[jammin1911]

Ty

[henrygd]

Do the Docker stats show if you switch to one hour?

Please provide the output of docker version on the agent machine and check if docker stats works properly.

Add LOG_LEVEL=debug to the agent and it should provide some logging on startup about Docker and if it finds any containers.

[jammin1911]

Thanks for the reply.

Docker version:
root@vps1:~# docker version
Client: Docker Engine - Community
 Version:           28.3.3
 API version:       1.51
 Go version:        go1.24.5
 Git commit:        980b856
 Built:             Fri Jul 25 11:34:09 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.3.3
  API version:      1.51 (minimum version 1.24)
  Go version:       go1.24.5
  Git commit:       bea959c
  Built:            Fri Jul 25 11:34:09 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker stats looks good (I use it faily often for quick reference while in terminal)

I dont think there are any errors in the agent logs after adding log level debug but here they are in case you see anything important:

beszel-agent  | 2025/09/16 18:22:49 DEBUG 0.12.7
beszel-agent  | 2025/09/16 18:22:49 DEBUG Not monitoring ZFS ARC err="open /proc/spl/kstat/zfs/arcstats: no such file or directory"
beszel-agent  | 2025/09/16 18:22:49 DEBUG Disk partitions="[{\"device\":\"/dev/sda1\",\"mountpoint\":\"/etc/resolv.conf\",\"fstype\":\"ext4\",\"opts\":[\"rw\",\"relatime\",\"bind\"]} {\"device\":\"/dev/sda1\",\"mountpoint\":\"/etc/hostname\",\"fstype\":\"ext4\",\"opts\":[\"rw\",\"relatime\",\"bind\"]} {\"device\":\"/dev/sda1\",\"mountpoint\":\"/etc/hosts\",\"fstype\":\"ext4\",\"opts\":[\"rw\",\"relatime\",\"bind\"]} {\"device\":\"/dev/sda1\",\"mountpoint\":\"/var/lib/beszel-agent\",\"fstype\":\"ext4\",\"opts\":[\"rw\",\"relatime\",\"bind\"]}]"
beszel-agent  | 2025/09/16 18:22:49 DEBUG Disk I/O diskstats="map[loop0:{\"readCount\":11,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":14336,\"writeBytes\":0,\"readTime\":0,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":0,\"weightedIO\":0,\"name\":\"loop0\",\"serialNumber\":\"\",\"label\":\"\"} sda:{\"readCount\":6272869,\"mergedReadCount\":1913493,\"writeCount\":8209334,\"mergedWriteCount\":10599272,\"readBytes\":153944874496,\"writeBytes\":209193032704,\"readTime\":841790,\"writeTime\":19240396,\"iopsInProgress\":0,\"ioTime\":8299794,\"weightedIO\":23666843,\"name\":\"sda\",\"serialNumber\":\"\",\"label\":\"\"} sda1:{\"readCount\":6272044,\"mergedReadCount\":1908458,\"writeCount\":8209267,\"mergedWriteCount\":10599226,\"readBytes\":153920186368,\"writeBytes\":209192711168,\"readTime\":841709,\"writeTime\":19240317,\"iopsInProgress\":0,\"ioTime\":8423774,\"weightedIO\":20143053,\"name\":\"sda1\",\"serialNumber\":\"\",\"label\":\"\"} sda14:{\"readCount\":199,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":942080,\"writeBytes\":0,\"readTime\":10,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":6,\"weightedIO\":10,\"name\":\"sda14\",\"serialNumber\":\"\",\"label\":\"\"} sda15:{\"readCount\":256,\"mergedReadCount\":4970,\"writeCount\":2,\"mergedWriteCount\":0,\"readBytes\":12760064,\"writeBytes\":1024,\"readTime\":24,\"writeTime\":1,\"iopsInProgress\":0,\"ioTime\":21,\"weightedIO\":25,\"name\":\"sda15\",\"serialNumber\":\"\",\"label\":\"\"} sda16:{\"readCount\":270,\"mergedReadCount\":65,\"writeCount\":65,\"mergedWriteCount\":46,\"readBytes\":9019904,\"writeBytes\":320512,\"readTime\":31,\"writeTime\":77,\"iopsInProgress\":0,\"ioTime\":84,\"weightedIO\":110,\"name\":\"sda16\",\"serialNumber\":\"\",\"label\":\"\"} sr0:{\"readCount\":171,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":5072896,\"writeBytes\":0,\"readTime\":6,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":2,\"weightedIO\":6,\"name\":\"sr0\",\"serialNumber\":\"\",\"label\":\"\"}]"
beszel-agent  | 2025/09/16 18:22:49 INFO Detected root device name=sda1
beszel-agent  | 2025/09/16 18:22:49 INFO Detected network interface name=eth0 sent=26813509415 recv=20278159341
beszel-agent  | 2025/09/16 18:22:49 INFO Detected network interface name=tailscale0 sent=7678938 recv=6921333
beszel-agent  | 2025/09/16 18:22:49 DEBUG GPU err="no GPU found - install nvidia-smi, rocm-smi, or tegrastats"
beszel-agent  | 2025/09/16 18:22:49 DEBUG No battery found err="Could not retrieve battery info: `Not found`"
beszel-agent  | 2025/09/16 18:22:49 DEBUG Load average 5m=0.11 15m=0.09
beszel-agent  | 2025/09/16 18:22:49 DEBUG Temperature sensors=[]
beszel-agent  | 2025/09/16 18:22:49 DEBUG sysinfo data="{Hostname:vps1 KernelVersion:6.8.0-79-generic Cores:1 Threads:1 CpuModel:AMD Ryzen 9 9950X 16-Core Processor Uptime:1267938 Cpu:100 MemPct:53.24 DiskPct:33.31 Bandwidth:0 AgentVersion:0.12.7 Podman:false GpuPct:0 DashboardTemp:0 Os:0 LoadAvg1:0.1 LoadAvg5:0.11 LoadAvg15:0.09 BandwidthBytes:0 LoadAvg:[0.1 0.11 0.09]}"
beszel-agent  | 2025/09/16 18:22:49 DEBUG System data data="&{Stats:{Cpu:100 MaxCpu:0 Mem:3.82 MemUsed:2.04 MemPct:53.24 MemBuffCache:1.64 MemZfsArc:0 Swap:8 SwapUsed:0.08 DiskTotal:95.82 DiskUsed:31.91 DiskPct:33.31 DiskReadPs:21.18 DiskWritePs:0 MaxDiskReadPs:0 MaxDiskWritePs:0 NetworkSent:0 NetworkRecv:0 MaxNetworkSent:0 MaxNetworkRecv:0 Temperatures:map[] ExtraFs:map[] GPUData:map[] LoadAvg1:0 LoadAvg5:0 LoadAvg15:0 Bandwidth:[0 0] MaxBandwidth:[0 0] LoadAvg:[0.1 0.11 0.09] Battery:[0 0] MaxMem:0} Info:{Hostname:vps1 KernelVersion:6.8.0-79-generic Cores:1 Threads:1 CpuModel:AMD Ryzen 9 9950X 16-Core Processor Uptime:1267938 Cpu:100 MemPct:53.24 DiskPct:33.31 Bandwidth:0 AgentVersion:0.12.7 Podman:false GpuPct:0 DashboardTemp:0 Os:0 LoadAvg1:0.1 LoadAvg5:0.11 LoadAvg15:0.09 BandwidthBytes:0 LoadAvg:[0.1 0.11 0.09]} Containers:[]}"
beszel-agent  | 2025/09/16 18:22:49 DEBUG Containers data="[0xc000156380 0xc000156700 0xc000156a80 0xc000156e00 0xc000157c00 0xc00017c000 0xc000157180 0xc000157500 0xc000157880 0xc00017c0e0 0xc00017c1c0 0xc00017c2a0 0xc00017c380]"
beszel-agent  | 2025/09/16 18:22:49 DEBUG Extra FS data=map[]
beszel-agent  | 2025/09/16 18:22:49 DEBUG Stats data="&{Stats:{Cpu:100 MaxCpu:0 Mem:3.82 MemUsed:2.04 MemPct:53.24 MemBuffCache:1.64 MemZfsArc:0 Swap:8 SwapUsed:0.08 DiskTotal:95.82 DiskUsed:31.91 DiskPct:33.31 DiskReadPs:21.18 DiskWritePs:0 MaxDiskReadPs:0 MaxDiskWritePs:0 NetworkSent:0 NetworkRecv:0 MaxNetworkSent:0 MaxNetworkRecv:0 Temperatures:map[] ExtraFs:map[] GPUData:map[] LoadAvg1:0 LoadAvg5:0 LoadAvg15:0 Bandwidth:[0 0] MaxBandwidth:[0 0] LoadAvg:[0.1 0.11 0.09] Battery:[0 0] MaxMem:0} Info:{Hostname:vps1 KernelVersion:6.8.0-79-generic Cores:1 Threads:1 CpuModel:AMD Ryzen 9 9950X 16-Core Processor Uptime:1267938 Cpu:100 MemPct:53.24 DiskPct:33.31 Bandwidth:0 AgentVersion:0.12.7 Podman:false GpuPct:0 DashboardTemp:0 Os:0 LoadAvg1:0.1 LoadAvg5:0.11 LoadAvg15:0.09 BandwidthBytes:0 LoadAvg:[0.1 0.11 0.09]} Containers:[0xc000156380 0xc000156700 0xc000156a80 0xc000156e00 0xc000157c00 0xc00017c000 0xc000157180 0xc000157500 0xc000157880 0xc00017c0e0 0xc00017c1c0 0xc00017c2a0 0xc00017c380]}"
beszel-agent  | 2025/09/16 18:22:49 INFO WebSocket connected host=beszel.xxxx.ca
beszel-agent  | 2025/09/16 18:22:49 DEBUG Cached data session=fdb97a1a-2e8e-4cff-ad22-89f27da98016```

[jammin1911]

WAIT! When I restarted it to add log_level debug it started working again.........

[henrygd]

I was going to say, looks like the containers are being picked up properly :)

Maybe it was an issue with the socket that couldn't be resolved until restarting. Perhaps you could check logs from the time it went down? Not sure if this will work but worth a try:

docker logs --since "2025-09-06T09:00:00Z" --until "2025-09-09T17:00:00Z" beszel-agent