[Bug]: Docker high CPU ACLKSYNC thread

[ptr727]

Bug description

Been running netdata in docker for a long time, noticed netdata consuming 100%+ CPU.

$ sudo top
top - 09:08:39 up 1 day, 23:43,  2 users,  load average: 4.61, 4.74, 4.92
Tasks: 908 total,   1 running, 907 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.0 us,  1.6 sy,  4.2 ni, 92.4 id,  0.7 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem : 128764.7 total,  35214.1 free,  89119.4 used,   5622.7 buff/cache
MiB Swap:   8192.0 total,   8192.0 free,      0.0 used.  39645.2 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
3160369 root      39  19 1650380 229360  16408 S 103.0   0.2  16:15.90 netdata

Expected behavior

Normal CPU

Steps to reproduce

1. Install netdata in docker on Proxmox
2. netdata consumer 100%+ CPU

Installation method

docker

System info

$ uname -a; grep -HvE "^#|URL" /etc/*release
Linux server-1 6.5.11-7-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.11-7 (2023-12-05T09:44Z) x86_64 GNU/Linux
/etc/os-release:PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
/etc/os-release:NAME="Debian GNU/Linux"
/etc/os-release:VERSION_ID="12"
/etc/os-release:VERSION="12 (bookworm)"
/etc/os-release:VERSION_CODENAME=bookworm
/etc/os-release:ID=debian

Netdata build info

"version":"v1.44.0-217-nightly"

Additional info

_netdata_logs.txt

[ilyam8]

Hey, @ptr727. We need more info.

• Your docker run/compose configuration. Did you just use Recommended Way? If not, please share your config/diff vs recommended way.
• You will need to do more debugging yourself because I can't reproduce using "Steps to reproduce" - see how to do it using htop.
• You haven't provided the build info, but only version - please update.

[ptr727]

Install using Ansible, using the documented method at the time I added netdata to my deployment, that had been working without issue for more than a year:

---
# Install NetData
# https://learn.netdata.cloud/docs/agent/packaging/docker
# https://learn.netdata.cloud/docs/agent/claim#connect-an-agent-running-in-docker
# https://github.com/netdata/netdata

# https://docs.ansible.com/ansible/latest/collections/community/docker/docker_volume_module.html
- name: "Create NetData Volumes"
  docker_volume:
    name: "{{ item }}"
    recreate: "{{ docker_volume_recreate }}"
  with_items:
    - netdata_config
    - netdata_lib
    - netdata_cache

# https://docs.ansible.com/ansible/latest/collections/community/docker/docker_container_module.html
- name: "Install NetData"
  community.docker.docker_container:
    name: netdata
    image: docker.io/netdata/netdata:latest
    pull: yes
    hostname: "netdata"
    domainname: "{{ ansible_domain }}"
    restart_policy: unless-stopped
    capabilities:
      - SYS_PTRACE
    security_opts:
      - apparmor:unconfined
    env:
      TZ: "{{ local_timezone }}"
      DOCKER_USR: "root"
      NETDATA_CLAIM_TOKEN: "{{ netdata_token }}"
      NETDATA_CLAIM_ROOMS: "{{ netdata_rooms }}"
      NETDATA_CLAIM_URL: "{{ netdata_url }}"
    volumes:
      - netdata_config:/etc/netdata
      - netdata_lib:/var/lib/netdata
      - netdata_cache:/var/cache/netdata
      - /etc/passwd:/host/etc/passwd:ro
      - /etc/group:/host/etc/group:ro
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /etc/os-release:/host/etc/os-release:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - name: "{{ docker_local_network }}"
    # published_ports:
    # - 19999:19999 # HTTP
    labels:
      traefik.enable: "true"
      traefik.http.routers.netdata.rule: "Host(`netdata.{{ ansible_domain }}`) || Host(`netdata-{{ ansible_fqdn }}`)"
      traefik.http.routers.netdata.entrypoints: "websecure"
      com.centurylinklabs.watchtower.enable: "true"
    recreate: "{{ docker_container_recreate }}"

How to get build info when running docker, I use what was reported in the docker state, I am using latest tag, that looks to be running nightly builds.

I will investigate more with htop threads when I have some time.

[ptr727]

ACLKSYNC

Node info:

{"fullyLoaded":false,"loaded":true,"id":"6900e68d-8cbc-4412-b513-0a8cfc5cd555","mg":"5e13c6a2-3d69-11eb-ae72-0242ac120016","nd":"6900e68d-8cbc-4412-b513-0a8cfc5cd555","isDeleted":false,"name":"netdata","version":"v1.44.0-227-nightly","ni":0,"labels":{"_os_version":"12 (bookworm)","_os_name":"Debian GNU/Linux","_cloud_instance_region":"unknown","_cloud_instance_type":"unknown","_cloud_provider_type":"unknown","_system_cores":"20","_system_ram_total":"135019528192","_system_disk_space":"198403535773696","_architecture":"x86_64","_system_cpu_freq":"3600000000","_kernel_version":"6.5.11-7-pve","_virtualization":"none","_container":"docker","_container_detection":"dockerenv","_virt_detection":"none","_is_k8s_node":"false","_install_type":"oci","_prebuilt_arch":"x86_64","_aclk_available":"true","_mqtt_version":"5","_aclk_proxy":"none","_aclk_ng_new_cloud_protocol":"true","_is_ephemeral":"false","_has_unstable_connection":"false","_is_parent":"false"},"labelKeys":["_os_version","_os_name","_cloud_instance_region","_cloud_instance_type","_cloud_provider_type","_system_cores","_system_ram_total","_system_disk_space","_architecture","_system_cpu_freq","_kernel_version","_virtualization","_container","_container_detection","_virt_detection","_is_k8s_node","_install_type","_prebuilt_arch","_aclk_available","_mqtt_version","_aclk_proxy","_aclk_ng_new_cloud_protocol","_is_ephemeral","_has_unstable_connection","_is_parent"],"hw":{"architecture":"x86_64","cpuFrequency":"3600000000","cpus":"20","memory":"135019528192","diskSpace":"198403535773696","virtualization":"none","container":"docker"},"os":{"id":"debian","nm":"Debian GNU/Linux","v":"12 (bookworm)","kernel":{"nm":"Linux","v":"6.5.11-7-pve"}},"capabilities":{"proto":{"name":"proto","version":1,"enabled":true},"ml":{"name":"ml","version":1,"enabled":true},"mc":{"name":"mc","version":1,"enabled":true},"ctx":{"name":"ctx","version":1,"enabled":true},"funcs":{"name":"funcs","version":1,"enabled":true},"http_api_v2":{"name":"http_api_v2","version":6,"enabled":true},"health":{"name":"health","version":1,"enabled":true},"req_cancel":{"name":"req_cancel","version":1,"enabled":true}},"state":"reachable","isProtobufCapable":true,"urls":[],"accessCount":0,"lastAccessTime":"","updateSeverity":null,"hasAccessibleData":true,"isLive":true,"nodeStatus":"Live","isOffline":false,"isUnseen":false,"capabilityKeys":["proto","ml","mc","ctx","funcs","http_api_v2","health","req_cancel"],"health":{"status":"initializing"},"isPreferred":true}

[ilyam8]

Ok, the same issue was reported on:

• Discord.
• Forum.

[netdata-community-bot]

This issue has been mentioned on the Netdata Community Forums. There might be relevant details there:

https://community.netdata.cloud/t/high-cpu-use-of-netdata/5087/7