[Bug]: METRIC: refcount is 0 (zero or negative) during release

[davidcba1]

Bug description

Netdata errors and exits out with this message in this logs. Didn't include anything from previous time stamps

Not sure if its related to this one #16809

Feb 27 23:01:10 test_host netdata[1909088]: Host 'AWS host1' with machine guid '7bdb3c90-d106-11ee-919f-0269db087c85' is obsolete - cleaning up.
Feb 27 23:01:10 test_host netdata[1909088]: Host 'AWS host2' with machine guid '67f378a4-cf90-11ee-a877-06708782dc81' is obsolete - cleaning up.
Feb 27 23:01:10 test_host netdata[1909088]: METRIC: refcount is 0 (zero or negative) during release
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0xbfe50)[0x557699a5de50]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x370f4b)[0x557699d0ef4b]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x371f7d)[0x557699d0ff7d]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x36787d)[0x557699d0587d]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x265c16)[0x557699c03c16]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x266820)[0x557699c04820]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x8b3d9)[0x557699a293d9]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x8cd9c)[0x557699a2ad9c]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x8d6ed)[0x557699a2b6ed]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x265d44)[0x557699c03d44]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x276010)[0x557699c14010]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x8b3d9)[0x557699a293d9]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x8cd9c)[0x557699a2ad9c]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x8d6ed)[0x557699a2b6ed]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x2720cb)[0x557699c100cb]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x269012)[0x557699c07012]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x714c7)[0x557699a0f4c7]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0xcd4e3)[0x557699a6b4e3]
Feb 27 23:01:10 test_host netdata[1909088]: /lib64/libpthread.so.0(+0x81ca)[0x7f465373b1ca]
Feb 27 23:01:10 test_host netdata[1909088]: /lib64/libc.so.6(clone+0x43)[0x7f465233de73]
Feb 27 23:01:10 test_host netdata[1909088]: NETDATA SHUTDOWN: initializing shutdown with code 1...
Feb 27 23:01:10 test_host netdata[1909088]: NETDATA SHUTDOWN: next: create shutdown file
Feb 27 23:01:10 test_host netdata[1909088]: NETDATA SHUTDOWN: in       1 ms, create shutdown file - next: dbengine exit mode
Feb 27 23:01:10 test_host netdata[1909088]: NETDATA SHUTDOWN: in       0 ms, dbengine exit mode - next: close webrtc connections
Feb 27 23:01:10 test_host netdata[1909088]: NETDATA SHUTDOWN: in       1 ms, close webrtc connections - next: disable maintenance, new queries, new web requests, new streaming connections and aclk
Feb 27 23:01:10 test_host netdata[1909088]: NETDATA SHUTDOWN: in       0 ms, disable maintenance, new queries, new web requests, new streaming connections and aclk - next: stop replication, exporters, he>
Feb 27 23:01:10 test_host netdata[1909088]: SERVICE CONTROL: waiting for the following 3 services [ WEB_SERVER HEALTH ] to exit: 'HEALTH' (1909292), 'WEB[1]' (1909297), 'WEB[2]' (1909326)
Feb 27 23:01:10 test_host netdata[1909088]: stopped after 42 connects, 42 disconnects (max concurrent 4), 1631 receptions and 3198 sends
Feb 27 23:01:10 test_host netdata[1909088]: closing all web server sockets...
Feb 27 23:01:10 test_host netdata[1909088]: all static web threads stopped.

Expected behavior

Netdata shouldn't shutdown...

Steps to reproduce

Not sure what causes it.. just leaving it running and it stops working

Installation method

manual setup of official DEB/RPM packages

System info

# uname -a; grep -HvE "^#|URL" /etc/*release
Linux vnl00008362 4.18.0-513.11.1.el8_9.x86_64 #1 SMP Thu Dec 7 03:06:13 EST 2023 x86_64 x86_64 x86_64 GNU/Linux
/etc/os-release:NAME="Red Hat Enterprise Linux"
/etc/os-release:VERSION="8.9 (Ootpa)"
/etc/os-release:ID="rhel"
/etc/os-release:ID_LIKE="fedora"
/etc/os-release:VERSION_ID="8.9"
/etc/os-release:PLATFORM_ID="platform:el8"
/etc/os-release:PRETTY_NAME="Red Hat Enterprise Linux 8.9 (Ootpa)"
/etc/os-release:ANSI_COLOR="0;31"
/etc/os-release:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
/etc/os-release:
/etc/os-release:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
/etc/os-release:REDHAT_BUGZILLA_PRODUCT_VERSION=8.9
/etc/os-release:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
/etc/os-release:REDHAT_SUPPORT_PRODUCT_VERSION="8.9"
/etc/redhat-release:Red Hat Enterprise Linux release 8.9 (Ootpa)
/etc/system-release:Red Hat Enterprise Linux release 8.9 (Ootpa)

Netdata build info

# netdata -W buildinfo
Packaging:
    Netdata Version ____________________________________________ : v1.44.3
    Installation Type __________________________________________ : binpkg-rpm
    Package Architecture _______________________________________ : x86_64
    Package Distro _____________________________________________ :
    Configure Options __________________________________________ :  '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--datadir=/usr/share' '--includedir=/usr/include' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/usr/libexec' '--libdir=/usr/lib' '--with-zlib' '--with-math' '--with-user=netdata' '--disable-dependency-tracking' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'LDFLAGS=-Wl,-z,relro  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' 'CXXFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'PKG_CONFIG_PATH=:/usr/lib/pkgconfig:/usr/share/pkgconfig'
Default Directories:
    User Configurations ________________________________________ : /etc/netdata
    Stock Configurations _______________________________________ : /usr/lib/netdata/conf.d
    Ephemeral Databases (metrics data, metadata) _______________ : /var/cache/netdata
    Permanent Databases ________________________________________ : /var/lib/netdata
    Plugins ____________________________________________________ : /usr/libexec/netdata/plugins.d
    Static Web Files ___________________________________________ : /usr/share/netdata/web
    Log Files __________________________________________________ : /var/log/netdata
    Lock Files _________________________________________________ : /var/lib/netdata/lock
    Home _______________________________________________________ : /var/lib/netdata
Operating System:
    Kernel _____________________________________________________ : Linux
    Kernel Version _____________________________________________ : 4.18.0-513.11.1.el8_9.x86_64
    Operating System ___________________________________________ : Red Hat Enterprise Linux
    Operating System ID ________________________________________ : rhel
    Operating System ID Like ___________________________________ : fedora
    Operating System Version ___________________________________ : 8.9 (Ootpa)
    Operating System Version ID ________________________________ : none
    Detection __________________________________________________ : /etc/os-release
Hardware:
    CPU Cores __________________________________________________ : 4
    CPU Frequency ______________________________________________ : 2095000000
    RAM Bytes __________________________________________________ : 8058548224
    Disk Capacity ______________________________________________ : 183609851904
    CPU Architecture ___________________________________________ : x86_64
    Virtualization Technology __________________________________ : vmware
    Virtualization Detection ___________________________________ : systemd-detect-virt
Container:
    Container __________________________________________________ : none
    Container Detection ________________________________________ : systemd-detect-virt
    Container Orchestrator _____________________________________ : none
    Container Operating System _________________________________ : none
    Container Operating System ID ______________________________ : none
    Container Operating System ID Like _________________________ : none
    Container Operating System Version _________________________ : none
    Container Operating System Version ID ______________________ : none
    Container Operating System Detection _______________________ : none
Features:
    Built For __________________________________________________ : Linux
    Netdata Cloud ______________________________________________ : YES
    Health (trigger alerts and send notifications) _____________ : YES
    Streaming (stream metrics to parent Netdata servers) _______ : YES
    Back-filling (of higher database tiers) ____________________ : YES
    Replication (fill the gaps of parent Netdata servers) ______ : YES
    Streaming and Replication Compression ______________________ : YES (zstd lz4 gzip)
    Contexts (index all active and archived metrics) ___________ : YES
    Tiering (multiple dbs with different metrics resolution) ___ : YES (5)
    Machine Learning ___________________________________________ : YES
Database Engines:
    dbengine ___________________________________________________ : YES
    alloc ______________________________________________________ : YES
    ram ________________________________________________________ : YES
    map ________________________________________________________ : YES
    save _______________________________________________________ : YES
    none _______________________________________________________ : YES
Connectivity Capabilities:
    ACLK (Agent-Cloud Link: MQTT over WebSockets over TLS) _____ : YES
    static (Netdata internal web server) _______________________ : YES
    h2o (web server) ___________________________________________ : YES
    WebRTC (experimental) ______________________________________ : NO
    Native HTTPS (TLS Support) _________________________________ : YES
    TLS Host Verification ______________________________________ : YES
Libraries:
    LZ4 (extremely fast lossless compression algorithm) ________ : YES
    ZSTD (fast, lossless compression algorithm) ________________ : YES
    zlib (lossless data-compression library) ___________________ : YES
    Judy (high-performance dynamic arrays and hashtables) ______ : YES (bundled)
    dlib (robust machine learning toolkit) _____________________ : YES (bundled)
    protobuf (platform-neutral data serialization protocol) ____ : YES (system)
    OpenSSL (cryptography) _____________________________________ : YES
    libdatachannel (stand-alone WebRTC data channels) __________ : NO
    JSON-C (lightweight JSON manipulation) _____________________ : YES
    libcap (Linux capabilities system operations) ______________ : NO
    libcrypto (cryptographic functions) ________________________ : YES
    libm (mathematical functions) ______________________________ : YES
    jemalloc ___________________________________________________ : NO
    TCMalloc ___________________________________________________ : NO
Plugins:
    apps (monitor processes) ___________________________________ : YES
    cgroups (monitor containers and VMs) _______________________ : YES
    cgroup-network (associate interfaces to CGROUPS) ___________ : YES
    proc (monitor Linux systems) _______________________________ : YES
    tc (monitor Linux network QoS) _____________________________ : YES
    diskspace (monitor Linux mount points) _____________________ : YES
    freebsd (monitor FreeBSD systems) __________________________ : NO
    macos (monitor MacOS systems) ______________________________ : NO
    statsd (collect custom application metrics) ________________ : YES
    timex (check system clock synchronization) _________________ : YES
    idlejitter (check system latency and jitter) _______________ : YES
    bash (support shell data collection jobs - charts.d) _______ : YES
    debugfs (kernel debugging metrics) _________________________ : YES
    cups (monitor printers and print jobs) _____________________ : YES
    ebpf (monitor system calls) ________________________________ : YES
    freeipmi (monitor enterprise server H/W) ___________________ : YES
    nfacct (gather netfilter accounting) _______________________ : NO
    perf (collect kernel performance events) ___________________ : YES
    slabinfo (monitor kernel object caching) ___________________ : YES
    Xen ________________________________________________________ : NO
    Xen VBD Error Tracking _____________________________________ : NO
    Logs Management ____________________________________________ : YES
Exporters:
    AWS Kinesis ________________________________________________ : NO
    GCP PubSub _________________________________________________ : NO
    MongoDB ____________________________________________________ : YES
    Prometheus (OpenMetrics) Exporter __________________________ : YES
    Prometheus Remote Write ____________________________________ : YES
    Graphite ___________________________________________________ : YES
    Graphite HTTP / HTTPS ______________________________________ : YES
    JSON _______________________________________________________ : YES
    JSON HTTP / HTTPS __________________________________________ : YES
    OpenTSDB ___________________________________________________ : YES
    OpenTSDB HTTP / HTTPS ______________________________________ : YES
    All Metrics API ____________________________________________ : YES
    Shell (use metrics in shell scripts) _______________________ : YES
Debug/Developer Features:
    Trace All Netdata Allocations (with charts) ________________ : NO
    Developer Mode (more runtime checks, slower) _______________ : NO

Additional info

No response

[vkalintiris]

Not sure what causes it.. just leaving it running and it stops working

Out of curiosity, how often/predictably can you reproduce this?

[davidcba1]

I can't.. i just notice monitoring isn't running and restart it.. I just did this and can see its happened 4 times in the past ~2 weeks. This host retrieves metrics from all the hosts so that's the only difference.

# journalctl -u netdata | grep -C10 -i 'METRIC: refcount is 0 (zero or negative) during release'
Feb 22 22:56:25 test_host netdata[972574]: DBENGINE: recalculating tier 0 retention for 12672 metrics starting with datafile 7052
Feb 22 22:56:25 test_host netdata[972574]: DBENGINE: migrated journal file '/var/cache/netdata/dbengine/journalfile-1-0000007086.njfv2', file size 1144716
Feb 22 22:56:25 test_host netdata[972574]: DBENGINE: updating tier 0 metrics registry retention for 12672 metrics
Feb 22 22:56:25 test_host netdata[972574]: DBENGINE: deleting data file '/var/cache/netdata/dbengine/datafile-1-0000007051.ndf'.
Feb 22 22:56:25 test_host netdata[972574]: DBENGINE: deleting data and journal files to maintain disk quota
Feb 22 22:56:25 test_host netdata[972574]: DBENGINE: deleted journal file "/var/cache/netdata/dbengine/journalfile-1-0000007051.njf".
Feb 22 22:56:25 test_host netdata[972574]: DBENGINE: deleted journal file "/var/cache/netdata/dbengine/journalfile-1-0000007051.njfv2".
Feb 22 22:56:25 test_host netdata[972574]: DBENGINE: deleted data file "/var/cache/netdata/dbengine/datafile-1-0000007051.ndf".
Feb 22 22:56:25 test_host netdata[972574]: DBENGINE: reclaimed 7307372 bytes of disk space.
Feb 22 23:01:20 test_host netdata[972574]: Host 'AWS Host' with machine guid '67f378a4-cf90-11ee-a877-06708782dc81' is obsolete - cleaning up.
Feb 22 23:01:20 test_host netdata[972574]: METRIC: refcount is 0 (zero or negative) during release
Feb 22 23:01:20 test_host netdata[972574]: /usr/sbin/netdata(+0xbfe50)[0x559f6946be50]
Feb 22 23:01:20 test_host netdata[972574]: /usr/sbin/netdata(+0x370f4b)[0x559f6971cf4b]
Feb 22 23:01:20 test_host netdata[972574]: /usr/sbin/netdata(+0x371f7d)[0x559f6971df7d]
Feb 22 23:01:20 test_host netdata[972574]: /usr/sbin/netdata(+0x36787d)[0x559f6971387d]
Feb 22 23:01:20 test_host netdata[972574]: /usr/sbin/netdata(+0x265c16)[0x559f69611c16]
Feb 22 23:01:20 test_host netdata[972574]: /usr/sbin/netdata(+0x266820)[0x559f69612820]
Feb 22 23:01:20 test_host netdata[972574]: /usr/sbin/netdata(+0x8b3d9)[0x559f694373d9]
Feb 22 23:01:20 test_host netdata[972574]: /usr/sbin/netdata(+0x8cd9c)[0x559f69438d9c]
Feb 22 23:01:20 test_host netdata[972574]: /usr/sbin/netdata(+0x8d6ed)[0x559f694396ed]
Feb 22 23:01:20 test_host netdata[972574]: /usr/sbin/netdata(+0x265d44)[0x559f69611d44]
--
Feb 26 03:03:22 test_host netdata[2542786]: Deleting chart 'systemd_insights-client-results.pids_current' ('systemd_insights-client-results.pids_current_3') from disk...
Feb 26 03:03:22 test_host netdata[2542786]: NETDATA SHUTDOWN: in    2803 ms, clean rrdhost database - next: stop aclk threads
Feb 26 03:03:22 test_host netdata[2542786]: NETDATA SHUTDOWN: in       0 ms, stop aclk threads - next: stop all remaining worker threads
Feb 26 03:03:22 test_host netdata[2542786]: NETDATA SHUTDOWN: in       0 ms, stop all remaining worker threads - next: cancel main threads
Feb 26 03:03:22 test_host netdata[2542786]: EXIT: Stopping main thread: DYNCFG
Feb 26 03:03:22 test_host netdata[2542786]: Waiting 1 threads to finish...
Feb 26 03:03:22 test_host netdata[2542786]: cleaning up...
Feb 26 03:03:23 test_host netdata[2542786]: All threads finished.
Feb 26 03:03:23 test_host netdata[2542786]: NETDATA SHUTDOWN: in     100 ms, cancel main threads - next: flush dbengine tiers
Feb 26 03:03:24 test_host netdata[2542786]: NETDATA SHUTDOWN: in    1455 ms, flush dbengine tiers - next: stop collection for all hosts
Feb 26 03:03:24 test_host netdata[2542786]: METRIC: refcount is 0 (zero or negative) during release
Feb 26 03:03:24 test_host netdata[2542786]: /usr/sbin/netdata(+0xbfe50)[0x5585464d1e50]
Feb 26 03:03:24 test_host netdata[2542786]: /usr/sbin/netdata(+0x370f4b)[0x558546782f4b]
Feb 26 03:03:24 test_host netdata[2542786]: /usr/sbin/netdata(+0x371f7d)[0x558546783f7d]
Feb 26 03:03:24 test_host netdata[2542786]: /usr/sbin/netdata(+0x36787d)[0x55854677987d]
Feb 26 03:03:24 test_host netdata[2542786]: /usr/sbin/netdata(+0x265c16)[0x558546677c16]
Feb 26 03:03:24 test_host netdata[2542786]: /usr/sbin/netdata(+0x271f90)[0x558546683f90]
Feb 26 03:03:24 test_host netdata[2542786]: /usr/sbin/netdata(+0x26c30d)[0x55854667e30d]
Feb 26 03:03:24 test_host netdata[2542786]: /usr/sbin/netdata(+0x26c398)[0x55854667e398]
Feb 26 03:03:24 test_host netdata[2542786]: /usr/sbin/netdata(+0x6e569)[0x558546480569]
Feb 26 03:03:24 test_host netdata[2542786]: /usr/sbin/netdata(+0x70bcc)[0x558546482bcc]
--
Feb 26 09:16:38 test_host netdata[1571916]: DBENGINE: indexing file '/var/cache/netdata/dbengine/journalfile-1-0000007582.njfv2': extents 202, metrics 12925, pages 12928
Feb 26 09:16:38 test_host netdata[1571916]: DBENGINE: migrated journal file '/var/cache/netdata/dbengine/journalfile-1-0000007582.njfv2', file size 1144896
Feb 26 09:16:38 test_host netdata[1571916]: DBENGINE: recalculating tier 0 retention for 12733 metrics starting with datafile 7548
Feb 26 09:16:38 test_host netdata[1571916]: DBENGINE: updating tier 0 metrics registry retention for 12733 metrics
Feb 26 09:16:38 test_host netdata[1571916]: DBENGINE: deleting data file '/var/cache/netdata/dbengine/datafile-1-0000007547.ndf'.
Feb 26 09:16:38 test_host netdata[1571916]: DBENGINE: deleting data and journal files to maintain disk quota
Feb 26 09:16:38 test_host netdata[1571916]: DBENGINE: deleted journal file "/var/cache/netdata/dbengine/journalfile-1-0000007547.njf".
Feb 26 09:16:38 test_host netdata[1571916]: DBENGINE: deleted journal file "/var/cache/netdata/dbengine/journalfile-1-0000007547.njfv2".
Feb 26 09:16:38 test_host netdata[1571916]: DBENGINE: deleted data file "/var/cache/netdata/dbengine/datafile-1-0000007547.ndf".
Feb 26 09:16:38 test_host netdata[1571916]: DBENGINE: reclaimed 7325200 bytes of disk space.
Feb 26 09:22:54 test_host netdata[1571916]: METRIC: refcount is 0 (zero or negative) during release
Feb 26 09:22:54 test_host netdata[1571916]: /usr/sbin/netdata(+0xbfe50)[0x557ea5bc2e50]
Feb 26 09:22:54 test_host netdata[1571916]: /usr/sbin/netdata(+0x370f4b)[0x557ea5e73f4b]
Feb 26 09:22:54 test_host netdata[1571916]: /usr/sbin/netdata(+0x371f7d)[0x557ea5e74f7d]
Feb 26 09:22:54 test_host netdata[1571916]: /usr/sbin/netdata(+0x36787d)[0x557ea5e6a87d]
Feb 26 09:22:54 test_host netdata[1571916]: /usr/sbin/netdata(+0x265c16)[0x557ea5d68c16]
Feb 26 09:22:54 test_host netdata[1571916]: /usr/sbin/netdata(+0x266820)[0x557ea5d69820]
Feb 26 09:22:54 test_host netdata[1571916]: /usr/sbin/netdata(+0x8b3d9)[0x557ea5b8e3d9]
Feb 26 09:22:54 test_host netdata[1571916]: /usr/sbin/netdata(+0x8cd9c)[0x557ea5b8fd9c]
Feb 26 09:22:54 test_host netdata[1571916]: /usr/sbin/netdata(+0x8d6ed)[0x557ea5b906ed]
Feb 26 09:22:54 test_host netdata[1571916]: /usr/sbin/netdata(+0x265d44)[0x557ea5d68d44]
--
Feb 27 22:59:10 test_host netdata[1909088]: Deleting chart header file '/var/cache/netdata/systemd_dnf-makecache.throttle_serviced_ops/main.db'.
Feb 27 22:59:10 test_host netdata[1909088]: Deleting dimension file '/var/cache/netdata/systemd_dnf-makecache.throttle_serviced_ops/read.db'.
Feb 27 22:59:10 test_host netdata[1909088]: Deleting dimension file '/var/cache/netdata/systemd_dnf-makecache.throttle_serviced_ops/write.db'.
Feb 27 22:59:10 test_host netdata[1909088]: Deleting empty directory '/var/cache/netdata/systemd_dnf-makecache.throttle_serviced_ops'
Feb 27 22:59:10 test_host netdata[1909088]: Deleting chart 'systemd_dnf-makecache.pids_current' ('systemd_dnf-makecache.pids_current_6') from disk...
Feb 27 22:59:10 test_host netdata[1909088]: Deleting chart header file '/var/cache/netdata/systemd_dnf-makecache.pids_current/main.db'.
Feb 27 22:59:10 test_host netdata[1909088]: Deleting dimension file '/var/cache/netdata/systemd_dnf-makecache.pids_current/pids.db'.
Feb 27 22:59:10 test_host netdata[1909088]: Deleting empty directory '/var/cache/netdata/systemd_dnf-makecache.pids_current'
Feb 27 23:01:10 test_host netdata[1909088]: Host 'AWS Host' with machine guid '7bdb3c90-d106-11ee-919f-0269db087c85' is obsolete - cleaning up.
Feb 27 23:01:10 test_host netdata[1909088]: Host 'AWS host' with machine guid '67f378a4-cf90-11ee-a877-06708782dc81' is obsolete - cleaning up.
Feb 27 23:01:10 test_host netdata[1909088]: METRIC: refcount is 0 (zero or negative) during release
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0xbfe50)[0x557699a5de50]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x370f4b)[0x557699d0ef4b]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x371f7d)[0x557699d0ff7d]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x36787d)[0x557699d0587d]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x265c16)[0x557699c03c16]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x266820)[0x557699c04820]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x8b3d9)[0x557699a293d9]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x8cd9c)[0x557699a2ad9c]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x8d6ed)[0x557699a2b6ed]
Feb 27 23:01:10 test_host netdata[1909088]: /usr/sbin/netdata(+0x265d44)[0x557699c03d44]

[stelfrag]

@davidcba1 Thanks for the report, we are investigating the issue

[hvulin]

Just to add to count I have the same problem:
time=2024-03-13T11:00:51.384+02:00 comm=netdata source=daemon level=alert tid=32420 thread=SERVICE msg="METRIC: refcount is 0 (zero or negative) during release"
/usr/sbin/netdata(+0xc832e)[0x55b35a11e32e]
/usr/sbin/netdata(+0x59b6f)[0x55b35a0afb6f]
/usr/sbin/netdata(+0x367632)[0x55b35a3bd632]
/usr/sbin/netdata(+0x35caa8)[0x55b35a3b2aa8]
/usr/sbin/netdata(+0x2662ba)[0x55b35a2bc2ba]
/usr/sbin/netdata(+0x266d9a)[0x55b35a2bcd9a]
/usr/sbin/netdata(+0x96dd3)[0x55b35a0ecdd3]
/usr/sbin/netdata(+0x9839f)[0x55b35a0ee39f]
/usr/sbin/netdata(+0x98dad)[0x55b35a0eedad]
/usr/sbin/netdata(+0x2663d0)[0x55b35a2bc3d0]
/usr/sbin/netdata(+0x2767f8)[0x55b35a2cc7f8]
/usr/sbin/netdata(+0x96dd3)[0x55b35a0ecdd3]
/usr/sbin/netdata(+0x9839f)[0x55b35a0ee39f]
/usr/sbin/netdata(+0x98dad)[0x55b35a0eedad]
/usr/sbin/netdata(+0x273537)[0x55b35a2c9537]
/usr/sbin/netdata(+0x26a0b6)[0x55b35a2c00b6]
/usr/sbin/netdata(+0x7dc57)[0x55b35a0d3c57]
/usr/sbin/netdata(+0xd5790)[0x55b35a12b790]
/lib64/libpthread.so.0(+0x7ea5)[0x7f7927884ea5]
/lib64/libc.so.6(clone+0x6d)[0x7f7926d8d8dd]

Web component dies and no 19999 port is open although process is still running.

[hugovalente-pm]

I believe this #17239 will address this issue

[ilyam8]

Yes, should be fixed in #17239. We will do a patch release (v1.45.1) later this week.