Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: 1.45.3 kernel: [] netdata[]: segfault at 18 ip 00007fd020b3afc4 sp 00007ffd4deb9df8 error 4 in libpthread-2.31.so #17549

Closed
ryanwadc opened this issue Apr 29, 2024 · 4 comments
Closed

[Bug]: 1.45.3 kernel: [] netdata[]: segfault at 18 ip 00007fd020b3afc4 sp 00007ffd4deb9df8 error 4 in libpthread-2.31.so #17549

ryanwadc opened this issue Apr 29, 2024 · 4 comments
Assignees
@stelfrag
Labels
area/daemon area/health bug

Comments

@ryanwadc
Copy link

ryanwadc commented Apr 29, 2024 •
edited by ilyam8

Bug description

Getting kernel segfaults after upgrading from 1.44.3 to 1.45.3 on some of our Ubuntu 20.04 physical hosts.

https://community.netdata.cloud/t/netdata-1-45-3-ubuntu-20-04-kernel-segfault-libpthread/5399

Expected behavior

Software to run similar as previous version.

Steps to reproduce

  1. Upgrade from 1.44.3 to 1.45.3
  2. Keep the same configs
  3. Process restart loops
    ...

Installation method

manual setup of official DEB/RPM packages

System info

Linux 5.4.0-177-generic #197-Ubuntu SMP Thu Mar 28 22:45:47 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
/etc/lsb-release:DISTRIB_ID=Ubuntu
/etc/lsb-release:DISTRIB_RELEASE=20.04
/etc/lsb-release:DISTRIB_CODENAME=focal
/etc/lsb-release:DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"
/etc/os-release:NAME="Ubuntu"
/etc/os-release:VERSION="20.04.6 LTS (Focal Fossa)"
/etc/os-release:ID=ubuntu
/etc/os-release:ID_LIKE=debian
/etc/os-release:PRETTY_NAME="Ubuntu 20.04.6 LTS"
/etc/os-release:VERSION_ID="20.04"
/etc/os-release:VERSION_CODENAME=focal
/etc/os-release:UBUNTU_CODENAME=focal

Netdata build info

Packaging:
    Netdata Version ____________________________________________ : v1.45.3
    Installation Type __________________________________________ : binpkg-deb
    Package Architecture _______________________________________ : x86_64
    Package Distro _____________________________________________ :  
    Configure Options __________________________________________ : dummy-configure-command
Default Directories:
    User Configurations ________________________________________ : /etc/netdata
    Stock Configurations _______________________________________ : /usr/lib/netdata/conf.d
    Ephemeral Databases (metrics data, metadata) _______________ : /var/cache/netdata
    Permanent Databases ________________________________________ : /var/lib/netdata
    Plugins ____________________________________________________ : /usr/libexec/netdata/plugins.d
    Static Web Files ___________________________________________ : /var/lib/netdata/www
    Log Files __________________________________________________ : /var/log/netdata
    Lock Files _________________________________________________ : /var/lib/netdata/lock
    Home _______________________________________________________ : /var/lib/netdata
Operating System:
    Kernel _____________________________________________________ : Linux
    Kernel Version _____________________________________________ : 5.4.0-177-generic
    Operating System ___________________________________________ : Ubuntu
    Operating System ID ________________________________________ : ubuntu
    Operating System ID Like ___________________________________ : debian
    Operating System Version ___________________________________ : 20.04.6 LTS (Focal Fossa)
    Operating System Version ID ________________________________ : none
    Detection __________________________________________________ : /etc/os-release
Hardware:
    CPU Cores __________________________________________________ : 88
    CPU Frequency ______________________________________________ : 3600000000
    RAM Bytes __________________________________________________ : 135059431424
    Disk Capacity ______________________________________________ : 2199291691008
    CPU Architecture ___________________________________________ : x86_64
    Virtualization Technology __________________________________ : none
    Virtualization Detection ___________________________________ : systemd-detect-virt
Container:
    Container __________________________________________________ : none
    Container Detection ________________________________________ : systemd-detect-virt
    Container Orchestrator _____________________________________ : none
    Container Operating System _________________________________ : none
    Container Operating System ID ______________________________ : none
    Container Operating System ID Like _________________________ : none
    Container Operating System Version _________________________ : none
    Container Operating System Version ID ______________________ : none
    Container Operating System Detection _______________________ : none
Features:
    Built For __________________________________________________ : Linux
    Netdata Cloud ______________________________________________ : YES
    Health (trigger alerts and send notifications) _____________ : YES
    Streaming (stream metrics to parent Netdata servers) _______ : YES
    Back-filling (of higher database tiers) ____________________ : YES
    Replication (fill the gaps of parent Netdata servers) ______ : YES
    Streaming and Replication Compression ______________________ : YES (zstd lz4 gzip)
    Contexts (index all active and archived metrics) ___________ : YES
    Tiering (multiple dbs with different metrics resolution) ___ : YES (5)
    Machine Learning ___________________________________________ : YES
Database Engines:
    dbengine ___________________________________________________ : YES
    alloc ______________________________________________________ : YES
    ram ________________________________________________________ : YES
    none _______________________________________________________ : YES
Connectivity Capabilities:
    ACLK (Agent-Cloud Link: MQTT over WebSockets over TLS) _____ : YES
    static (Netdata internal web server) _______________________ : YES
    h2o (web server) ___________________________________________ : YES
    WebRTC (experimental) ______________________________________ : NO
    Native HTTPS (TLS Support) _________________________________ : YES
    TLS Host Verification ______________________________________ : YES
Libraries:
    LZ4 (extremely fast lossless compression algorithm) ________ : YES
    ZSTD (fast, lossless compression algorithm) ________________ : YES
    zlib (lossless data-compression library) ___________________ : YES
    Brotli (generic-purpose lossless compression algorithm) ____ : NO
    protobuf (platform-neutral data serialization protocol) ____ : YES (system)
    OpenSSL (cryptography) _____________________________________ : YES
    libdatachannel (stand-alone WebRTC data channels) __________ : NO
    JSON-C (lightweight JSON manipulation) _____________________ : YES
    libcap (Linux capabilities system operations) ______________ : NO
    libcrypto (cryptographic functions) ________________________ : YES
    libyaml (library for parsing and emitting YAML) ____________ : YES
Plugins:
    apps (monitor processes) ___________________________________ : YES
    cgroups (monitor containers and VMs) _______________________ : YES
    cgroup-network (associate interfaces to CGROUPS) ___________ : YES
    proc (monitor Linux systems) _______________________________ : YES
    tc (monitor Linux network QoS) _____________________________ : YES
    diskspace (monitor Linux mount points) _____________________ : YES
    freebsd (monitor FreeBSD systems) __________________________ : NO
    macos (monitor MacOS systems) ______________________________ : NO
    statsd (collect custom application metrics) ________________ : YES
    timex (check system clock synchronization) _________________ : YES
    idlejitter (check system latency and jitter) _______________ : YES
    bash (support shell data collection jobs - charts.d) _______ : YES
    debugfs (kernel debugging metrics) _________________________ : YES
    cups (monitor printers and print jobs) _____________________ : YES
    ebpf (monitor system calls) ________________________________ : YES
    freeipmi (monitor enterprise server H/W) ___________________ : YES
    nfacct (gather netfilter accounting) _______________________ : YES
    perf (collect kernel performance events) ___________________ : YES
    slabinfo (monitor kernel object caching) ___________________ : YES
    Xen ________________________________________________________ : YES
    Xen VBD Error Tracking _____________________________________ : NO
    Logs Management ____________________________________________ : YES
Exporters:
    AWS Kinesis ________________________________________________ : NO
    GCP PubSub _________________________________________________ : NO
    MongoDB ____________________________________________________ : YES
    Prometheus (OpenMetrics) Exporter __________________________ : YES
    Prometheus Remote Write ____________________________________ : YES
    Graphite ___________________________________________________ : YES
    Graphite HTTP / HTTPS ______________________________________ : YES
    JSON _______________________________________________________ : YES
    JSON HTTP / HTTPS __________________________________________ : YES
    OpenTSDB ___________________________________________________ : YES
    OpenTSDB HTTP / HTTPS ______________________________________ : YES
    All Metrics API ____________________________________________ : YES
    Shell (use metrics in shell scripts) _______________________ : YES
Debug/Developer Features:
    Trace All Netdata Allocations (with charts) ________________ : NO
    Developer Mode (more runtime checks, slower) _______________ : NO

Additional info

Apr 25 00:04:39  netdata[2559809]: SPAWN: Ran out of protocol buffer space.
Apr 25 00:04:39  netdata[2559809]: time=2024-04-25T00:04:39.891-05:00 comm=netdata source=daemon level=alert errno="4, Interrupted system call" tid=2559809 thread=netdata msg="Assertion `SPAWN_
PROT_EXEC_CMD == header->opcode' failed"
Apr 25 00:04:39  netdata[2559809]: /usr/sbin/netdata(+0x4a10b2)[0x563192d760b2]
Apr 25 00:04:39  netdata[2559809]: /usr/sbin/netdata(+0x38f318)[0x563192c64318]
Apr 25 00:04:39  netdata[2559809]: /lib/x86_64-linux-gnu/libuv.so.1(+0x1abb1)[0x7f3e150aebb1]
Apr 25 00:04:39  netdata[2559809]: /lib/x86_64-linux-gnu/libuv.so.1(+0x1b6e8)[0x7f3e150af6e8]
Apr 25 00:04:39  netdata[2559809]: /lib/x86_64-linux-gnu/libuv.so.1(uv__io_poll+0x360)[0x7f3e150b4b90]
Apr 25 00:04:39  netdata[2559809]: /lib/x86_64-linux-gnu/libuv.so.1(uv_run+0x11c)[0x7f3e150a485c]
Apr 25 00:04:39  netdata[2559809]: /usr/sbin/netdata(+0x38f657)[0x563192c64657]
Apr 25 00:04:39  netdata[2559809]: /usr/sbin/netdata(+0x799fb)[0x56319294e9fb]
Apr 25 00:04:39  netdata[2559809]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f3e1495f083]
Apr 25 00:04:39  netdata[2559809]: /usr/sbin/netdata(+0x7dcfe)[0x563192952cfe]
Apr 25 00:04:39  netdata[2559809]: time=2024-04-25T00:04:39.891-05:00 comm=netdata source=daemon level=info tid=2559809 thread=netdata msg="NETDATA SHUTDOWN: initializing shutdown with code 1..."
Apr 25 00:04:39  netdata[2566342]: sh: 1: Syntax error: Unterminated quoted string
Apr 25 00:04:39  netdata[2559799]: time=2024-04-25T00:04:39.893-05:00 comm=netdata source=daemon level=info tid=2559808 thread=DAEMON_SPAWN msg="EOF found in spawn pipe."
Apr 25 00:04:39  netdata[2559799]: time=2024-04-25T00:04:39.897-05:00 comm=netdata source=daemon level=alert tid=2559808 thread=DAEMON_SPAWN msg="Assertion `ret == 0' failed" 
Apr 25 00:04:39  netdata[2559799]: /usr/sbin/netdata(+0x4a10b2)[0x55e5c0aa10b2]
Apr 25 00:04:39  netdata[2559799]: /usr/sbin/netdata(+0x3900a3)[0x55e5c09900a3]
Apr 25 00:04:39  netdata[2559799]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7f0332adf609]
Apr 25 00:04:39  netdata[2559799]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f0332169353]
Apr 25 00:04:39  netdata[2559799]: time=2024-04-25T00:04:39.898-05:00 comm=netdata source=daemon level=info tid=2559808 thread=DAEMON_SPAWN msg="NETDATA SHUTDOWN: initializing shutdown with code 1..."
@ryanwadc ryanwadc added bug needs triage Issues which need to be manually labelled labels Apr 29, 2024
@stelfrag stelfrag self-assigned this Apr 29, 2024
@netdata-community-bot
Copy link

This issue has been mentioned on the Netdata Community Forums. There might be relevant details there:

https://community.netdata.cloud/t/netdata-1-45-3-ubuntu-20-04-kernel-segfault-libpthread/5399/5

@stelfrag
Copy link
Collaborator

Hi @ryanwadc , does the agent still crash when you disable health? (that is setting enabled = no in the [health] section of netdata.conf and restaring the agent) ?

@ryanwadc
Copy link
Author

Hi @stelfrag . Yes, the service stays running with [health] enabled = no.

@stelfrag
Copy link
Collaborator

Hi @stelfrag . Yes, the service stays running with [health] enabled = no.

Thank you, we will be working on a fix

@stelfrag stelfrag mentioned this issue May 2, 2024
Merged
@ilyam8 ilyam8 added area/daemon area/health and removed needs triage Issues which need to be manually labelled labels May 3, 2024
@ilyam8 ilyam8 closed this as completed May 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stelfrag stelfrag

Labels
area/daemon area/health bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ilyam8 @stelfrag @netdata-community-bot @ryanwadc