segfault with streaming to multi-host dbengine instances

[mfundul]

As I'm only using netdata as collector aggregation and polling from prometheus to plot in grafana, I can trigger netdata primary restarts frequently without loosing much data or having gaps in my graphs, and systemd is keeping up with the segfaults when netdata crashes, but it's not as stable as one would like it to be:

That said, I'm also seeing recurrent segfaults in several different threads:

# dmesg -T | egrep -A3 -B3 segfault
[Wed Aug  5 10:20:49 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 10:20:49 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 10:20:49 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 11:00:43 2020] WEB_SERVER[stat[8193]: segfault at 100000001 ip 00000000004358b8 sp 00007f92be07d2e0 error 6 in netdata[400000+188000]
[Wed Aug  5 11:01:25 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 11:01:25 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 11:01:25 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 11:13:09 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 11:13:09 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 11:13:09 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 11:25:57 2020] STREAM_RECEIVER[21617]: segfault at 8 ip 00000000004d6ed5 sp 00007f994f4d7120 error 4 in netdata[400000+188000]
[Wed Aug  5 11:26:40 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 11:26:40 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 11:26:41 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 12:12:14 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 12:12:15 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 12:12:15 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 12:21:27 2020] HEALTH[1880]: segfault at 24320 ip 00000000004e67b0 sp 00007efd42557530 error 6 in netdata[400000+188000]
[Wed Aug  5 12:22:11 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 12:22:11 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 12:22:11 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 12:23:53 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 12:23:53 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 12:23:53 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 12:32:40 2020] PLUGIN[proc][5313]: segfault at 18 ip 00007fb9e256f45b sp 00007fb9df4ae400 error 4 in libc-2.17.so[7fb9e24ef000+1c3000]
[Wed Aug  5 12:33:25 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 12:33:25 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 12:33:26 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 12:35:08 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 12:35:08 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 12:35:08 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 12:43:00 2020] HEALTH[8397]: segfault at 5f2bc051 ip 00000000004e6741 sp 00007fa50c4ec530 error 6 in netdata[400000+188000]
[Wed Aug  5 12:43:47 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 12:43:47 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 12:43:47 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 12:45:29 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 12:45:29 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 12:45:29 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 12:53:16 2020] HEALTH[11360]: segfault at f7a0 ip 00000000004e67b0 sp 00007f1608263530 error 6 in netdata[400000+188000]
[Wed Aug  5 12:54:03 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 12:54:03 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 12:54:04 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 12:55:46 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 12:55:46 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 12:55:46 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 13:02:18 2020] HEALTH[14155]: segfault at 230 ip 00000000004e67b0 sp 00007fc426086530 error 6 in netdata[400000+188000]
[Wed Aug  5 13:03:05 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 13:03:05 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 13:03:06 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 13:04:47 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 13:04:48 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 13:04:48 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 13:12:20 2020] HEALTH[17170]: segfault at 513ef020 ip 00000000004e67b0 sp 00007fbe7b2ad530 error 6 in netdata[400000+188000]
[Wed Aug  5 13:13:08 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 13:13:08 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 13:13:08 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 13:36:34 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 13:36:34 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 13:36:34 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 13:39:18 2020] netdata[26676]: segfault at 31 ip 00007f4a5b56c079 sp 00007fff887d0ba0 error 4 in libc-2.17.so[7f4a5b51f000+1c3000]
[Wed Aug  5 13:39:32 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 13:39:32 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 13:39:32 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 13:51:08 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 13:51:08 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 13:51:08 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 13:51:12 2020] netdata[31711]: segfault at 31 ip 00007f2f38e38079 sp 00007ffc28265370 error 4 in libc-2.17.so[7f2f38deb000+1c3000]
[Wed Aug  5 13:51:16 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 13:51:16 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 13:51:16 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 13:55:16 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 13:55:16 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 13:55:16 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 14:06:34 2020] STREAM_RECEIVER[2943]: segfault at 8 ip 00000000004d6ed5 sp 00007f1164db0120 error 4
[Wed Aug  5 14:07:08 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 14:07:08 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 14:07:09 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 14:08:51 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 14:08:51 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 14:08:51 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 14:21:03 2020] STREAM_RECEIVER[5901]: segfault at 18 ip 00007f0e1cb9d724 sp 00007f0e048a3fe0 error 4 in libc-2.17.so[7f0e1cb1b000+1c3000]
[Wed Aug  5 14:21:38 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 14:21:38 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 14:21:38 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 15:45:46 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 15:45:46 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 15:45:46 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 15:54:34 2020] PLUGIN[diskspac[28592]: segfault at 18 ip 00007f410757b724 sp 00007f4103cb75d0 error 4 in libc-2.17.so[7f41074f9000+1c3000]
[Wed Aug  5 15:55:13 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 15:55:13 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 15:55:13 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 16:04:38 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 16:04:38 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 16:04:38 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 16:22:10 2020] STREAM_RECEIVER[2352]: segfault at 11 ip 00007ff6fd7baf64 sp 00007ff6ebed3b20 error 4 in libc-2.17.so[7ff6fd739000+1c3000]
[Wed Aug  5 16:22:45 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 16:22:45 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 16:22:45 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 16:24:27 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 16:24:27 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 16:24:27 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 16:33:55 2020] HEALTH[6569]: segfault at 1439ea9b0 ip 00000000004e6741 sp 00007f15456d8530 error 6 in netdata[400000+188000]
[Wed Aug  5 16:34:30 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 16:34:30 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 16:34:31 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 17:14:28 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 17:14:28 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 17:14:28 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 17:48:51 2020] HEALTH[18315]: segfault at 6410 ip 00000000004e67b0 sp 00007f0477786530 error 6 in netdata[400000+188000]
[Wed Aug  5 17:49:30 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 17:49:30 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 17:49:31 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 18:48:03 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 18:48:03 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 18:48:03 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 19:05:26 2020] HEALTH[3245]: segfault at 60 ip 00000000004e67b0 sp 00007f7bea94d530 error 6 in netdata[400000+188000]
[Wed Aug  5 19:06:08 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 19:06:08 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 19:06:09 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 19:26:43 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 19:26:43 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 19:26:43 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 19:40:00 2020] PLUGINSD[11731]: segfault at b2bd508a ip 00007f56ed8f442c sp 00007f56e582a540 error 4 in libc-2.17.so[7f56ed874000+1c3000]
[Wed Aug  5 19:40:44 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 19:40:44 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 19:40:45 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 19:59:30 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 19:59:31 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 19:59:31 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 20:07:08 2020] HEALTH[19057]: segfault at 52f97000 ip 00000000004e67b0 sp 00007f8568ccb530 error 6 in netdata[400000+188000]
[Wed Aug  5 20:07:52 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 20:07:52 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 20:07:52 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 20:57:59 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 20:57:59 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 20:57:59 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 21:25:16 2020] STREAM_RECEIVER[5167]: segfault at cf1343b8 ip 00007fcae793e712 sp 00007fcae086dfc0 error 4 in libc-2.17.so[7fcae78bc000+1c3000]
[Wed Aug  5 21:26:06 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 21:26:06 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 21:26:07 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 21:27:49 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 21:27:49 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 21:27:49 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 21:41:36 2020] STREAM_RECEIVER[9063]: segfault at 5f3f0010 ip 00007fc21af0bff0 sp 00007fc211636fc0 error 4 in libc-2.17.so[7fc21ae8a000+1c3000]
[Wed Aug  5 21:42:27 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 21:42:27 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 21:42:27 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 21:44:09 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 21:44:09 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 21:44:09 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 22:08:20 2020] STREAM_RECEIVER[9600]: segfault at 15aa88500 ip 00007fcb850f7712 sp 00007fcb74813fe0 error 4 in libc-2.17.so[7fcb85075000+1c3000]
[Wed Aug  5 22:09:26 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 22:09:26 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 22:09:26 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 22:45:32 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 22:45:32 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 22:45:32 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 22:58:30 2020] HEALTH[25799]: segfault at 299 ip 00000000004e6741 sp 00007f17cc328530 error 6 in netdata[400000+188000]
[Wed Aug  5 22:59:24 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 22:59:24 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 22:59:24 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Wed Aug  5 23:32:44 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 23:32:44 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 23:32:44 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Wed Aug  5 23:54:42 2020] STREAM_RECEIVER[5031]: segfault at 100000011 ip 00007fefcd248ff0 sp 00007fefc3172fc0 error 4 in libc-2.17.so[7fefcd1c7000+1c3000]
[Wed Aug  5 23:55:39 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Wed Aug  5 23:55:39 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Wed Aug  5 23:55:40 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 01:02:25 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 01:02:26 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 01:02:26 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 01:16:38 2020] STREAM_RECEIVER[29686]: segfault at 5f2c7c92 ip 00007f003cb00ff0 sp 00007f002aa1afc0 error 4 in libc-2.17.so[7f003ca7f000+1c3000]
[Thu Aug  6 01:17:13 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 01:17:13 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 01:17:14 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 01:21:49 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 01:21:50 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 01:21:50 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 01:24:22 2020] STREAM_RECEIVER[31575]: segfault at 9597f050 ip 00007f004f739712 sp 00007f0039c45000 error 4 in libc-2.17.so[7f004f6b7000+1c3000]
[Thu Aug  6 01:24:58 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 01:24:58 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 01:24:59 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 01:26:40 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 01:26:41 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 01:26:41 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 01:39:18 2020] STREAM_RECEIVER[934]: segfault at 5f7a95cd ip 00007f2908e5aff0 sp 00007f28f3d6cb20 error 4 in libc-2.17.so[7f2908dd9000+1c3000]
[Thu Aug  6 01:39:54 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 01:39:54 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 01:39:55 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 01:41:37 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 01:41:37 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 01:41:37 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 01:50:04 2020] HEALTH[4210]: segfault at 11a30 ip 00000000004e67b0 sp 00007f9dfa069530 error 6
[Thu Aug  6 01:50:04 2020] PLUGIN[proc][4199]: segfault at 9 ip 00007f9e02934419 sp 00007f9dff873400 error 4
[Thu Aug  6 01:50:04 2020]  in libc-2.17.so[7f9e028b4000+1c3000]

[Thu Aug  6 01:50:04 2020]  in netdata[400000+188000]
--
[Thu Aug  6 04:04:47 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 04:04:47 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 04:04:47 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 04:15:14 2020] STREAM_RECEIVER[6104]: segfault at afc76218 ip 00007ffabf99c712 sp 00007ffab90ccfc0 error 4 in libc-2.17.so[7ffabf91a000+1c3000]
[Thu Aug  6 04:15:55 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 04:15:55 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 04:15:55 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 04:17:38 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 04:17:38 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 04:17:38 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 04:26:35 2020] HEALTH[9771]: segfault at 299 ip 00000000004e6741 sp 00007fc30072d530 error 6 in netdata[400000+188000]
[Thu Aug  6 04:27:17 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 04:27:17 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 04:27:18 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 04:29:00 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 04:29:00 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 04:29:00 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 04:43:12 2020] HEALTH[12726]: segfault at 2080 ip 00000000004e67b0 sp 00007fd6d6f93530 error 6 in netdata[400000+188000]
[Thu Aug  6 04:43:54 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 04:43:54 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 04:43:55 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 04:45:37 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 04:45:37 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 04:45:37 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 04:57:49 2020] PLUGINSD[apps][16594]: segfault at 8 ip 00000000004d6ed5 sp 00007f21c17195b0 error 4 in netdata[400000+188000]
[Thu Aug  6 04:58:31 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 04:58:31 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 04:58:32 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 05:00:14 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 05:00:14 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 05:00:14 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 05:25:41 2020] STREAM_RECEIVER[20177]: segfault at 1f1 ip 0000000000442e6e sp 00007f27fefe0d00 error 4 in netdata[400000+188000]
[Thu Aug  6 05:26:25 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 05:26:25 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 05:26:26 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 05:28:08 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 05:28:08 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 05:28:08 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 05:28:31 2020] STREAM_RECEIVER[25700]: segfault at 8 ip 00000000004d6ed5 sp 00007f5709474120 error 4 in netdata[400000+188000]
[Thu Aug  6 05:29:15 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 05:29:15 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 05:29:16 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 05:30:58 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 05:30:58 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 05:30:58 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 05:58:49 2020] WEB_SERVER[stat[26862]: segfault at 18 ip 00007f917cf59724 sp 00007f9172e89270 error 4 in libc-2.17.so[7f917ced7000+1c3000]
[Thu Aug  6 05:59:36 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 05:59:36 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 05:59:36 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 06:01:18 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 06:01:18 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 06:01:18 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 06:10:45 2020] WEB_SERVER[stat[997]: segfault at 0 ip 00000000004358b8 sp 00007f78ba7b0340 error 6 in netdata[400000+188000]
[Thu Aug  6 06:11:34 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 06:11:34 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 06:11:34 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 06:38:48 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 06:38:48 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 06:38:48 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 06:49:25 2020] HEALTH[11563]: segfault at 4321d13d63e ip 0000000000410000 sp 00007f393b17d528 error 6 in netdata[400000+188000]
[Thu Aug  6 06:50:14 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 06:50:14 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 06:50:14 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 06:51:56 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 06:51:57 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 06:51:57 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 06:59:09 2020] STREAM_RECEIVER[15775]: segfault at 15353b990 ip 00007fef531f6724 sp 00007fef4791cfe0 error 4 in libc-2.17.so[7fef53174000+1c3000]
[Thu Aug  6 06:59:58 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 06:59:58 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 06:59:59 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 07:01:41 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 07:01:41 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 07:01:41 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 07:10:08 2020] HEALTH[18342]: segfault at 158f3ec90 ip 00000000004e6741 sp 00007f1a69427530 error 6 in netdata[400000+188000]
[Thu Aug  6 07:10:58 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 07:10:58 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 07:10:59 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 07:36:56 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 07:36:56 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 07:36:56 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 07:51:34 2020] PLUGIN[cgroups][27310]: segfault at 100000011 ip 00007f3450b20ff0 sp 00007f344ca59f90 error 4 in libc-2.17.so[7f3450a9f000+1c3000]
[Thu Aug  6 07:52:56 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 07:52:56 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 07:52:56 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 08:14:36 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 08:14:37 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 08:14:37 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 08:29:44 2020] PLUGIN[proc][19692]: segfault at 100000011 ip 00000000004b03f8 sp 00007fe5d96c5170 error 4 in netdata[400000+187000]
[Thu Aug  6 08:29:44 2020] PLUGINSD[apps][19717]: segfault at 100000001 ip 0000000100000001 sp 00007fe5d56bf5f8 error 14


[Thu Aug  6 08:30:18 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 08:32:00 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 08:32:00 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 08:32:00 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 08:43:22 2020] STREAM_RECEIVER[23931]: segfault at 89 ip 00007f01ac2bc274 sp 00007f01991da200 error 4 in libc-2.17.so[7f01ac240000+1c3000]
[Thu Aug  6 08:43:57 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 08:43:57 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 08:43:57 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
--
[Thu Aug  6 08:45:39 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 08:45:39 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 08:45:39 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_1 doesn't exist.
[Thu Aug  6 08:56:36 2020] WEB_SERVER[stat[27342]: segfault at 8 ip 00007f3a104e1112 sp 00007f3a01d26300 error 4 in libz.so.1.2.7[7f3a104de000+15000]
[Thu Aug  6 08:57:10 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
[Thu Aug  6 08:57:10 2020] Event kprobes/p_netdata_tcp_cleanup_rbuf_0 doesn't exist.
[Thu Aug  6 08:57:11 2020] Event kprobes/p_netdata_vfs_write_0 doesn't exist.
# dmesg -T | egrep -A3 -B3 segfault | egrep -c segfault
131
# uname -a
Linux localhost 3.10.0-1127.13.1.el7.x86_64 #1 SMP Tue Jun 23 15:46:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

# yum update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.mirror.nucleus.be
 * epel: d2lzkl7pfhq30w.cloudfront.net
 * extras: centos.quelquesmots.fr
 * updates: miroir.univ-paris13.fr
No packages marked for update
# yum upgrade
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.mirrors.proxad.net
 * epel: d2lzkl7pfhq30w.cloudfront.net
 * extras: centos.quelquesmots.fr
 * updates: miroir.univ-paris13.fr
No packages marked for update
#
# cat /etc/centos-release
CentOS Linux release 7.8.2003 (Core)
# netdata -version
netdata v1.23.2-131-nightly

Originally posted by @marcelmfs in #9627 (comment)

[mfundul]

@marcelmfs I've created a new issue for you as this is not related to the original issue. It appears there is some memory corruption from crashes at malloc() or free() sites.

from the user's error.log:

*** Error in `/usr/sbin/netdata': free(): invalid pointer: 0x00000000606cbddc ***

and

*** Error in `/usr/sbin/netdata': malloc(): memory corruption (fast): 0x000000001da2b8d0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f3e4)[0x7f071d0333e4]
/lib64/libc.so.6(+0x82b20)[0x7f071d036b20]
/lib64/libc.so.6(__libc_malloc+0x4c)[0x7f071d0396fc]
/lib64/libc.so.6(__strdup+0x1a)[0x7f071d040afa]
/usr/sbin/netdata[0x41c509]
/usr/sbin/netdata[0x4a9743]
/usr/sbin/netdata[0x4a9af9]
/usr/sbin/netdata[0x46e37c]
/usr/sbin/netdata[0x49753e]
/usr/sbin/netdata[0x4d84bf]
/usr/sbin/netdata[0x4ef4fd]
/usr/sbin/netdata[0x4effa8]
/usr/sbin/netdata[0x443cc8]
/lib64/libpthread.so.0(+0x7ea5)[0x7f071d389ea5]
/lib64/libc.so.6(clone+0x6d)[0x7f071d0b28dd]
======= Memory map: ========
00400000-00587000 r-xp 00000000 08:01 25527512                           /usr/sbin/netdata
00787000-00788000 r--p 00187000 08:01 25527512                           /usr/sbin/netdata
00788000-0079c000 rw-p 00188000 08:01 25527512                           /usr/sbin/netdata
0079c000-008b4000 rw-p 00000000 00:00 0 
00f0f000-5b283000 rw-p 00000000 00:00 0                                  [heap]
7f0703b2d000-7f0703b42000 r-xp 00000000 08:01 26471856                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f0703b42000-7f0703d41000 ---p 00015000 08:01 26471856                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f0703d41000-7f0703d42000 r--p 00014000 08:01 26471856                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f0703d42000-7f0703d43000 rw-p 00015000 08:01 26471856                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f0703d43000-7f0703d44000 ---p 00000000 00:00 0 
7f0703d44000-7f0704544000 rw-p 00000000 00:00 0 
7f0704544000-7f0704545000 ---p 00000000 00:00 0 
7f0704545000-7f0704d45000 rw-p 00000000 00:00 0 
7f0704d45000-7f0704d46000 ---p 00000000 00:00 0 
7f0704d46000-7f0705546000 rw-p 00000000 00:00 0 
7f0705546000-7f070554c000 r-xp 00000000 08:01 25167150                   /usr/lib64/libnss_dns-2.17.so
7f070554c000-7f070574b000 ---p 00006000 08:01 25167150                   /usr/lib64/libnss_dns-2.17.so
7f070574b000-7f070574c000 r--p 00005000 08:01 25167150                   /usr/lib64/libnss_dns-2.17.so
7f070574c000-7f070574d000 rw-p 00006000 08:01 25167150                   /usr/lib64/libnss_dns-2.17.so
7f070574d000-7f070574e000 ---p 00000000 00:00 0 
7f070574e000-7f0705f4e000 rw-p 00000000 00:00 0 
7f0705f4e000-7f0705f4f000 ---p 00000000 00:00 0 
7f0705f4f000-7f070674f000 rw-p 00000000 00:00 0 
7f070674f000-7f0706750000 ---p 00000000 00:00 0 
7f0706750000-7f0706f50000 rw-p 00000000 00:00 0 
7f0706f50000-7f0706f51000 ---p 00000000 00:00 0 
7f0706f51000-7f0707751000 rw-p 00000000 00:00 0 
7f0707751000-7f0707752000 ---p 00000000 00:00 0 
7f0707752000-7f0707f52000 rw-p 00000000 00:00 0 
7f0707f52000-7f0707f53000 ---p 00000000 00:00 0 
7f0707f53000-7f0708753000 rw-p 00000000 00:00 0 
7f0708753000-7f0708754000 ---p 00000000 00:00 0 
7f0708754000-7f0708f54000 rw-p 00000000 00:00 0 
7f0708f54000-7f0708f55000 ---p 00000000 00:00 0 
7f0708f55000-7f0709755000 rw-p 00000000 00:00 0 
7f0709755000-7f0709756000 ---p 00000000 00:00 0 
7f0709756000-7f0709f56000 rw-p 00000000 00:00 0 
7f0709f56000-7f0709f57000 ---p 00000000 00:00 0 
7f0709f57000-7f070a757000 rw-p 00000000 00:00 0 
7f070a757000-7f070a758000 ---p 00000000 00:00 0 
7f070a758000-7f070af58000 rw-p 00000000 00:00 0 
7f070af58000-7f070af59000 ---p 00000000 00:00 0 
7f070af59000-7f070b759000 rw-p 00000000 00:00 0 
7f070b759000-7f070b75a000 ---p 00000000 00:00 0 
7f070b75a000-7f070bf5a000 rw-p 00000000 00:00 0 
7f070bf5a000-7f070bf5b000 ---p 00000000 00:00 0 
7f070bf5b000-7f070c75b000 rw-p 00000000 00:00 0 
7f070c75b000-7f070c75c000 ---p 00000000 00:00 0 
7f070c75c000-7f070cf5c000 rw-p 00000000 00:00 0 
7f070cf5c000-7f070cf5d000 ---p 00000000 00:00 0 
7f070cf5d000-7f070d75d000 rw-p 00000000 00:00 0 
7f070d75d000-7f070d75e000 ---p 00000000 00:00 0 
7f070d75e000-7f070df5e000 rw-p 00000000 00:00 0 
7f070df5e000-7f070df5f000 ---p 00000000 00:00 0 
7f070df5f000-7f070e75f000 rw-p 00000000 00:00 0 
7f070e75f000-7f070e760000 ---p 00000000 00:00 0 
7f070e760000-7f070ef60000 rw-p 00000000 00:00 0 
7f070ef60000-7f070ef61000 ---p 00000000 00:00 0 
7f070ef61000-7f070f761000 rw-p 00000000 00:00 0 
7f070f761000-7f070f762000 ---p 00000000 00:00 0 
7f070f762000-7f070ff62000 rw-p 00000000 00:00 0 
7f070ff62000-7f070ff63000 ---p 00000000 00:00 0 
7f070ff63000-7f0710763000 rw-p 00000000 00:00 0 
7f0710763000-7f0710764000 ---p 00000000 00:00 0 
7f0710764000-7f0710f64000 rw-p 00000000 00:00 0 
7f0710f64000-7f0710f65000 ---p 00000000 00:00 0 
7f0710f65000-7f0711765000 rw-p 00000000 00:00 0 
7f0711765000-7f0711766000 ---p 00000000 00:00 0 
7f0711766000-7f0711f66000 rw-p 00000000 00:00 0 
7f0711f66000-7f0711f67000 ---p 00000000 00:00 0 
7f0711f67000-7f0712767000 rw-p 00000000 00:00 0 
7f0712767000-7f0712768000 ---p 00000000 00:00 0 
7f0712768000-7f0712f68000 rw-p 00000000 00:00 0 
7f0712f68000-7f0712f69000 ---p 00000000 00:00 0 
7f0712f69000-7f0713769000 rw-p 00000000 00:00 0 
7f0713769000-7f071376a000 ---p 00000000 00:00 0 
7f071376a000-7f0713f6a000 rw-p 00000000 00:00 0 
7f0713f6a000-7f0713f6b000 ---p 00000000 00:00 0 
7f0713f6b000-7f071476b000 rw-p 00000000 00:00 0 
7f071476b000-7f071476c000 ---p 00000000 00:00 0 
7f071476c000-7f0714f6c000 rw-p 00000000 00:00 0 
7f0714f6c000-7f0714f6d000 ---p 00000000 00:00 0 
7f0714f6d000-7f071576d000 rw-p 00000000 00:00 0 
7f071576d000-7f071576e000 ---p 00000000 00:00 0 
7f071576e000-7f0715f6e000 rw-p 00000000 00:00 0 
7f0715f6e000-7f0715f6f000 ---p 00000000 00:00 0 
7f0715f6f000-7f071676f000 rw-p 00000000 00:00 0 
7f071676f000-7f0716770000 ---p 00000000 00:00 0 
7f0716770000-7f0716f70000 rw-p 00000000 00:00 0 
7f0716f70000-7f0716f71000 ---p 00000000 00:00 0 
7f0716f71000-7f0717771000 rw-p 00000000 00:00 0 
7f0717771000-7f0717772000 ---p 00000000 00:00 0 
7f0717772000-7f0717f72000 rw-p 00000000 00:00 0 
7f0717f72000-7f0717f73000 ---p 00000000 00:00 0 
7f0717f73000-7f0718773000 rw-p 00000000 00:00 0 
7f0718773000-7f0718774000 ---p 00000000 00:00 0 
7f0718774000-7f0718f74000 rw-p 00000000 00:00 0 
7f0718f74000-7f0718f75000 ---p 00000000 00:00 0 
7f0718f75000-7f0719775000 rw-p 00000000 00:00 0 
7f0719775000-7f0719776000 ---p 00000000 00:00 0 
7f0719776000-7f0719f76000 rw-p 00000000 00:00 0 
7f0719f76000-7f0719f77000 ---p 00000000 00:00 0 
7f0719f77000-7f071a777000 rw-p 00000000 00:00 0 
7f071a777000-7f071a778000 ---p 00000000 00:00 0 
7f071a778000-7f071af78000 rw-p 00000000 00:00 0 
7f071af78000-7f071af79000 ---p 00000000 00:00 0 
7f071af79000-7f071b779000 rw-p 00000000 00:00 0 
7f071b779000-7f071b785000 r-xp 00000000 08:01 25747649                   /usr/lib64/libnss_files-2.17.so
7f071b785000-7f071b984000 ---p 0000c000 08:01 25747649                   /usr/lib64/libnss_files-2.17.so
7f071b984000-7f071b985000 r--p 0000b000 08:01 25747649                   /usr/lib64/libnss_files-2.17.so
7f071b985000-7f071b986000 rw-p 0000c000 08:01 25747649                   /usr/lib64/libnss_files-2.17.so
7f071b986000-7f071b98c000 rw-p 00000000 00:00 0 
7f071b98c000-7f071b9ec000 r-xp 00000000 08:01 25167288                   /usr/lib64/libpcre.so.1.2.0
7f071b9ec000-7f071bbec000 ---p 00060000 08:01 25167288                   /usr/lib64/libpcre.so.1.2.0
7f071bbec000-7f071bbed000 r--p 00060000 08:01 25167288                   /usr/lib64/libpcre.so.1.2.0
7f071bbed000-7f071bbee000 rw-p 00061000 08:01 25167288                   /usr/lib64/libpcre.so.1.2.0
7f071bbee000-7f071bc12000 r-xp 00000000 08:01 25166666                   /usr/lib64/libselinux.so.1
7f071bc12000-7f071be11000 ---p 00024000 08:01 25166666                   /usr/lib64/libselinux.so.1
7f071be11000-7f071be12000 r--p 00023000 08:01 25166666                   /usr/lib64/libselinux.so.1
7f071be12000-7f071be13000 rw-p 00024000 08:01 25166666                   /usr/lib64/libselinux.so.1
7f071be13000-7f071be15000 rw-p 00000000 00:00 0 
7f071be15000-7f071be2b000 r-xp 00000000 08:01 25751317                   /usr/lib64/libresolv-2.17.so
7f071be2b000-7f071c02b000 ---p 00016000 08:01 25751317                   /usr/lib64/libresolv-2.17.so
7f071c02b000-7f071c02c000 r--p 00016000 08:01 25751317                   /usr/lib64/libresolv-2.17.so
7f071c02c000-7f071c02d000 rw-p 00017000 08:01 25751317                   /usr/lib64/libresolv-2.17.so
7f071c02d000-7f071c02f000 rw-p 00000000 00:00 0 
7f071c02f000-7f071c032000 r-xp 00000000 08:01 25167287                   /usr/lib64/libkeyutils.so.1.5
7f071c032000-7f071c231000 ---p 00003000 08:01 25167287                   /usr/lib64/libkeyutils.so.1.5
7f071c231000-7f071c232000 r--p 00002000 08:01 25167287                   /usr/lib64/libkeyutils.so.1.5
7f071c232000-7f071c233000 rw-p 00003000 08:01 25167287                   /usr/lib64/libkeyutils.so.1.5
7f071c233000-7f071c241000 r-xp 00000000 08:01 25868687                   /usr/lib64/libkrb5support.so.0.1
7f071c241000-7f071c441000 ---p 0000e000 08:01 25868687                   /usr/lib64/libkrb5support.so.0.1
7f071c441000-7f071c442000 r--p 0000e000 08:01 25868687                   /usr/lib64/libkrb5support.so.0.1
7f071c442000-7f071c443000 rw-p 0000f000 08:01 25868687                   /usr/lib64/libkrb5support.so.0.1
7f071c443000-7f071c474000 r-xp 00000000 08:01 25167414                   /usr/lib64/libk5crypto.so.3.1
7f071c474000-7f071c673000 ---p 00031000 08:01 25167414                   /usr/lib64/libk5crypto.so.3.1
7f071c673000-7f071c675000 r--p 00030000 08:01 25167414                   /usr/lib64/libk5crypto.so.3.1
7f071c675000-7f071c676000 rw-p 00032000 08:01 25167414                   /usr/lib64/libk5crypto.so.3.1
7f071c676000-7f071c679000 r-xp 00000000 08:01 25867425                   /usr/lib64/libcom_err.so.2.1
7f071c679000-7f071c878000 ---p 00003000 08:01 25867425                   /usr/lib64/libcom_err.so.2.1
7f071c878000-7f071c879000 r--p 00002000 08:01 25867425                   /usr/lib64/libcom_err.so.2.1
7f071c879000-7f071c87a000 rw-p 00003000 08:01 25867425                   /usr/lib64/libcom_err.so.2.1
7f071c87a000-7f071c953000 r-xp 00000000 08:01 25167420                   /usr/lib64/libkrb5.so.3.3
7f071c953000-7f071cb52000 ---p 000d9000 08:01 25167420                   /usr/lib64/libkrb5.so.3.3
7f071cb52000-7f071cb60000 r--p 000d8000 08:01 25167420                   /usr/lib64/libkrb5.so.3.3
7f071cb60000-7f071cb63000 rw-p 000e6000 08:01 25167420                   /usr/lib64/libkrb5.so.3.3
7f071cb63000-7f071cbad000 r-xp 00000000 08:01 25867587                   /usr/lib64/libgssapi_krb5.so.2.2
7f071cbad000-7f071cdad000 ---p 0004a000 08:01 25867587                   /usr/lib64/libgssapi_krb5.so.2.2
7f071cdad000-7f071cdae000 r--p 0004a000 08:01 25867587                   /usr/lib64/libgssapi_krb5.so.2.2
7f071cdae000-7f071cdb0000 rw-p 0004b000 08:01 25867587                   /usr/lib64/libgssapi_krb5.so.2.2
7f071cdb0000-7f071cdb2000 r-xp 00000000 08:01 25733994                   /usr/lib64/libdl-2.17.so
7f071cdb2000-7f071cfb2000 ---p 00002000 08:01 25733994                   /usr/lib64/libdl-2.17.so
7f071cfb2000-7f071cfb3000 r--p 00002000 08:01 25733994                   /usr/lib64/libdl-2.17.so
7f071cfb3000-7f071cfb4000 rw-p 00003000 08:01 25733994                   /usr/lib64/libdl-2.17.so
7f071cfb4000-7f071d177000 r-xp 00000000 08:01 25166779                   /usr/lib64/libc-2.17.so
7f071d177000-7f071d377000 ---p 001c3000 08:01 25166779                   /usr/lib64/libc-2.17.so
7f071d377000-7f071d37b000 r--p 001c3000 08:01 25166779                   /usr/lib64/libc-2.17.so
7f071d37b000-7f071d37d000 rw-p 001c7000 08:01 25166779                   /usr/lib64/libc-2.17.so
7f071d37d000-7f071d382000 rw-p 00000000 00:00 0 
7f071d382000-7f071d399000 r-xp 00000000 08:01 25751315                   /usr/lib64/libpthread-2.17.so
7f071d399000-7f071d598000 ---p 00017000 08:01 25751315                   /usr/lib64/libpthread-2.17.so
7f071d598000-7f071d599000 r--p 00016000 08:01 25751315                   /usr/lib64/libpthread-2.17.so
7f071d599000-7f071d59a000 rw-p 00017000 08:01 25751315                   /usr/lib64/libpthread-2.17.so
7f071d59a000-7f071d59e000 rw-p 00000000 00:00 0 
7f071d59e000-7f071d5a8000 r-xp 00000000 08:01 25169794                   /usr/lib64/libjson-c.so.2.0.1
7f071d5a8000-7f071d7a7000 ---p 0000a000 08:01 25169794                   /usr/lib64/libjson-c.so.2.0.1
7f071d7a7000-7f071d7a8000 r--p 00009000 08:01 25169794                   /usr/lib64/libjson-c.so.2.0.1
7f071d7a8000-7f071d7a9000 rw-p 0000a000 08:01 25169794                   /usr/lib64/libjson-c.so.2.0.1
7f071d7a9000-7f071d7f7000 r-xp 00000000 08:01 25346756                   /usr/lib64/libJudy.so.1.0.3
7f071d7f7000-7f071d9f7000 ---p 0004e000 08:01 25346756                   /usr/lib64/libJudy.so.1.0.3
7f071d9f7000-7f071d9f8000 r--p 0004e000 08:01 25346756                   /usr/lib64/libJudy.so.1.0.3
7f071d9f8000-7f071d9f9000 rw-p 0004f000 08:01 25346756                   /usr/lib64/libJudy.so.1.0.3
7f071d9f9000-7f071da0d000 r-xp 00000000 08:01 25168784                   /usr/lib64/liblz4.so.1.7.5
7f071da0d000-7f071dc0c000 ---p 00014000 08:01 25168784                   /usr/lib64/liblz4.so.1.7.5
7f071dc0c000-7f071dc0d000 r--p 00013000 08:01 25168784                   /usr/lib64/liblz4.so.1.7.5
7f071dc0d000-7f071dc0e000 rw-p 00014000 08:01 25168784                   /usr/lib64/liblz4.so.1.7.5
7f071dc0e000-7f071dc3b000 r-xp 00000000 08:01 25166788                   /usr/lib64/libuv.so.1.0.0
7f071dc3b000-7f071de3b000 ---p 0002d000 08:01 25166788                   /usr/lib64/libuv.so.1.0.0
7f071de3b000-7f071de3c000 r--p 0002d000 08:01 25166788                   /usr/lib64/libuv.so.1.0.0
7f071de3c000-7f071de3d000 rw-p 0002e000 08:01 25166788                   /usr/lib64/libuv.so.1.0.0
7f071de3d000-7f071de3e000 rw-p 00000000 00:00 0 
7f071de3e000-7f071de42000 r-xp 00000000 08:01 25867453                   /usr/lib64/libuuid.so.1.3.0
7f071de42000-7f071e041000 ---p 00004000 08:01 25867453                   /usr/lib64/libuuid.so.1.3.0
7f071e041000-7f071e042000 r--p 00003000 08:01 25867453                   /usr/lib64/libuuid.so.1.3.0
7f071e042000-7f071e043000 rw-p 00004000 08:01 25867453                   /usr/lib64/libuuid.so.1.3.0
7f071e043000-7f071e0aa000 r-xp 00000000 08:01 25167434                   /usr/lib64/libssl.so.1.0.2k
7f071e0aa000-7f071e2aa000 ---p 00067000 08:01 25167434                   /usr/lib64/libssl.so.1.0.2k
7f071e2aa000-7f071e2ae000 r--p 00067000 08:01 25167434                   /usr/lib64/libssl.so.1.0.2k
7f071e2ae000-7f071e2b5000 rw-p 0006b000 08:01 25167434                   /usr/lib64/libssl.so.1.0.2k
7f071e2b5000-7f071e4eb000 r-xp 00000000 08:01 25167432                   /usr/lib64/libcrypto.so.1.0.2k
7f071e4eb000-7f071e6eb000 ---p 00236000 08:01 25167432                   /usr/lib64/libcrypto.so.1.0.2k
7f071e6eb000-7f071e707000 r--p 00236000 08:01 25167432                   /usr/lib64/libcrypto.so.1.0.2k
7f071e707000-7f071e714000 rw-p 00252000 08:01 25167432                   /usr/lib64/libcrypto.so.1.0.2k
7f071e714000-7f071e718000 rw-p 00000000 00:00 0 
7f071e718000-7f071e72d000 r-xp 00000000 08:01 25167200                   /usr/lib64/libz.so.1.2.7
7f071e72d000-7f071e92c000 ---p 00015000 08:01 25167200                   /usr/lib64/libz.so.1.2.7
7f071e92c000-7f071e92d000 r--p 00014000 08:01 25167200                   /usr/lib64/libz.so.1.2.7
7f071e92d000-7f071e92e000 rw-p 00015000 08:01 25167200                   /usr/lib64/libz.so.1.2.7
7f071e92e000-7f071ea2f000 r-xp 00000000 08:01 25734005                   /usr/lib64/libm-2.17.so
7f071ea2f000-7f071ec2e000 ---p 00101000 08:01 25734005                   /usr/lib64/libm-2.17.so
7f071ec2e000-7f071ec2f000 r--p 00100000 08:01 25734005                   /usr/lib64/libm-2.17.so
7f071ec2f000-7f071ec30000 rw-p 00101000 08:01 25734005                   /usr/lib64/libm-2.17.so
7f071ec30000-7f071ec52000 r-xp 00000000 08:01 25197364                   /usr/lib64/ld-2.17.so
7f071ee0e000-7f071ee13000 rw-p 00000000 00:00 0 
7f071ee1b000-7f071ee4a000 rw-p 00000000 00:00 0 
7f071ee4a000-7f071ee51000 rw-p 00000000 00:00 0 
7f071ee51000-7f071ee52000 r--p 00021000 08:01 25197364                   /usr/lib64/ld-2.17.so
7f071ee52000-7f071ee53000 rw-p 00022000 08:01 25197364                   /usr/lib64/ld-2.17.so
7f071ee53000-7f071ee54000 rw-p 00000000 00:00 0 
7fff78812000-7fff78833000 rw-p 00000000 00:00 0                          [stack]
7fff78936000-7fff78938000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

[mfundul]

@marcelmfs can you recompile the netdata agent with debug information so that we can figure out what's going on?

You can replace the existing installation by reinstalling from the git master branch as follows:

1. Download the source code:

git clone https://github.com/netdata/netdata.git

2. Install the address sanitizer to catch the memory corruption:

yum install libasan

3. Reinstall the netdata agent with the required debug flags:

sudo CFLAGS="-O0 -ggdb -fno-omit-frame-pointer -fsanitize=address -fstack-protector-all -DNETDATA_INTERNAL_CHECKS=1 -DNETDATA_VERIFY_LOCKS=1" ./netdata-installer.sh

This way we should be able to get useful information in var/log/netdata/error.log or via journalctl.

[marcelmfs]

@mfundul It's done, as soon as I detect it again I'll paste the log section here.

[marcelmfs]

@mfundul it happened just now:

2020-08-10 13:10:17: netdata ERROR : HEALTH : INTERNAL ERROR: requested to step above RRDR size for chart 'cgroup_deploy_container_name_sanitized_service_1.net_eth0'
=================================================================
==23740== ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6004007379d8 at pc 0x63c6e9 bp 0x7f53dde54300 sp 0x7f53dde542f0
READ of size 8 at 0x6004007379d8 thread T15 (HEALTH)
    #0 0x63c6e8 (/usr/sbin/netdata+0x63c6e8)
    #1 0x63deaa (/usr/sbin/netdata+0x63deaa)
    #2 0x60d72a (/usr/sbin/netdata+0x60d72a)
    #3 0x610bdf (/usr/sbin/netdata+0x610bdf)
    #4 0x612f7e (/usr/sbin/netdata+0x612f7e)
    #5 0x5e4072 (/usr/sbin/netdata+0x5e4072)
    #6 0x629bc1 (/usr/sbin/netdata+0x629bc1)
    #7 0x7f53eb5ab867 (/usr/lib64/libasan.so.0.0.0+0x19867)
    #8 0x7f53e9cebea4 (/usr/lib64/libpthread-2.17.so+0x7ea4)
    #9 0x7f53e9a148dc (/usr/lib64/libc-2.17.so+0xfe8dc)
0x6004007379d8 is located 0 bytes to the right of 8-byte region [0x6004007379d0,0x6004007379d8)
allocated by thread T15 (HEALTH) here:
    #0 0x7f53eb5a7ff5 (/usr/lib64/libasan.so.0.0.0+0x15ff5)
    #1 0x666a35 (/usr/sbin/netdata+0x666a35)
    #2 0x6112bf (/usr/sbin/netdata+0x6112bf)
    #3 0x60ca03 (/usr/sbin/netdata+0x60ca03)
    #4 0x610bdf (/usr/sbin/netdata+0x610bdf)
    #5 0x612f7e (/usr/sbin/netdata+0x612f7e)
    #6 0x5e4072 (/usr/sbin/netdata+0x5e4072)
    #7 0x629bc1 (/usr/sbin/netdata+0x629bc1)
    #8 0x7f53eb5ab867 (/usr/lib64/libasan.so.0.0.0+0x19867)
Thread T15 (HEALTH) created by T0 here:
    #0 0x7f53eb59ca0a (/usr/lib64/libasan.so.0.0.0+0xaa0a)
    #1 0x629d8a (/usr/sbin/netdata+0x629d8a)
    #2 0x687dca (/usr/sbin/netdata+0x687dca)
    #3 0x7f53e9938554 (/usr/lib64/libc-2.17.so+0x22554)
Shadow bytes around the buggy address:
  0x0c01000deee0: fa fa fd fd fa fa fd fa fa fa fd fd fa fa fd fd
  0x0c01000deef0: fa fa 05 fa fa fa fd fd fa fa fd fd fa fa fd fd
  0x0c01000def00: fa fa fd fd fa fa 00 04 fa fa 00 01 fa fa fd fd
  0x0c01000def10: fa fa fa fa fa fa fa fa fa fa fd fd fa fa fd fd
  0x0c01000def20: fa fa 00 fa fa fa fd fd fa fa 00 00 fa fa fa fa
=>0x0c01000def30: fa fa fa fa fa fa fa fa fa fa 00[fa]fa fa fd fd
  0x0c01000def40: fa fa 00 fa fa fa fa fa fa fa 00 05 fa fa fa fa
  0x0c01000def50: fa fa fa fa fa fa fa fa fa fa fd fd fa fa fd fd
  0x0c01000def60: fa fa fa fa fa fa fa fa fa fa 02 fa fa fa fa fa
  0x0c01000def70: fa fa fa fa fa fa fd fd fa fa fa fa fa fa fa fa
  0x0c01000def80: fa fa fa fa fa fa fa fa fa fa 05 fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:     fa
  Heap righ redzone:     fb
  Freed Heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==23740== ABORTING
EOF found in spawn pipe.
Shutting down spawn server event loop.
Shutting down spawn server loop complete.
2020-08-10 13:10:17: apps.plugin FATAL : apps.plugin : Cannot write to a pipe # : Success

2020-08-10 13:10:17: go.d INFO: main[main] received terminated signal (15). Terminating...

I'll update here as they keep coming.

[mfundul]

@marcelmfs that's great! Since the address sanitizer output is kind of useless for some reason in CentOS 7 can you send the netdata binary /usr/sbin/netdata to markos@netdata.cloud so that I can run gdb against it?

[mfundul]

Thank you @marcelmfs , I need your help a little bit more unfortunately. Since the program crashed in debug codepath, I'd like to see where the problem lies in production. So can you reinstall with somewhat different arguments? It should be:

sudo CFLAGS="-O0 -ggdb -fno-omit-frame-pointer -fsanitize=address -fstack-protector-all" ./netdata-installer.sh

Edit: Scratch that, I think we have enough information as it is.

[mfundul]

heap-buffer-overflow on:

0x63c6e8 is in rrdr_line_init (web/api/queries/query.c:360).
355         #ifdef NETDATA_INTERNAL_CHECKS
356
357         if(unlikely(rrdr_line >= r->n))
358             error("INTERNAL ERROR: requested to step above RRDR size for chart '%s'", r->st->name);
359
360         if(unlikely(r->t[rrdr_line] != 0 && r->t[rrdr_line] != t))
361             error("INTERNAL ERROR: overwriting the timestamp of RRDR line %zu from %zu to %zu, of chart '%s'", (size_t)rrdr_line, (size_t)r->t[rrdr_line], (size_t)t, r->st->name);
362
363         #endif
364   

0x63deaa is in do_dimension_fixedstep.41667 (web/api/queries/query.c:599).
594                 r->internal.grouping_add(r, value);
595                 values_in_group++;
596                 db_points_read++;
597   
598                 if(unlikely(values_in_group == group_size)) {
599                     rrdr_line = rrdr_line_init(r, now, rrdr_line);
600   
601                     if(unlikely(!min_date)) min_date = now;
602                     max_date = now;
603   

0x60d72a is in rrd2rrdr_fixedstep (web/api/queries/query.c:1075).
1070            r->od[c] |= RRDR_DIMENSION_SELECTED;
1071  
1072            // reset the grouping for the new dimension
1073            r->internal.grouping_reset(r);
1074  
1075            do_dimension_fixedstep(
1076                    r
1077                    , points_wanted
1078                    , rd
1079                    , c

0x610bdf is in rrd2rrdr (web/api/queries/query.c:1580).
1575                                rrdr_convert_before_after_to_absolute(&after_requested, &before_requested, rrd_update_every,
1576                                                                      first_entry_t, last_entry_t);
1577                    }
1578                    freez(region_info_array);
1579                }
1580                return rrd2rrdr_fixedstep(st, points_requested, after_requested, before_requested, group_method,
1581                                          resampling_time_requested, options, dimensions, rrd_update_every,
1582                                          first_entry_t, last_entry_t, absolute_period_requested);
1583            } else {
1584                if (rrd_update_every != (uint16_t)max_interval) {

0x612f7e is in rrdset2value_api_v1 (web/api/formatters/rrd2json.c:72).
67              , uint32_t options
68              , time_t *db_after
69              , time_t *db_before
70              , int *value_is_null
71      ) {
72          RRDR *r = rrd2rrdr(st, points, after, before, group_method, group_time, options, dimensions);
73          if(!r) {
74              if(value_is_null) *value_is_null = 1;
75              return HTTP_RESP_INTERNAL_SERVER_ERROR;
76          }

0x5e4072 is in health_main (health/health.c:721).
716
717                     if (unlikely(RRDCALC_HAS_DB_LOOKUP(rc))) {
718                         /* time_t old_db_timestamp = rc->db_before; */
719                         int value_is_null = 0;
720
721                         int ret = rrdset2value_api_v1(rc->rrdset, NULL, &rc->value, rc->dimensions, 1, rc->after,
722                                                       rc->before, rc->group, 0, rc->options, &rc->db_after,
723                                                       &rc->db_before, &value_is_null
724                         );
725

Allocated by:

0x60ca03 is in rrd2rrdr_fixedstep (web/api/queries/query.c:979).
974   
975         // -------------------------------------------------------------------------
976         // initialize our result set
977         // this also locks the chart for us
978   
979         RRDR *r = rrdr_create(st, points_wanted);
980         if(unlikely(!r)) {
981             #ifdef NETDATA_INTERNAL_CHECKS
982             error("INTERNAL CHECK: Cannot create RRDR for %s, after=%u, before=%u, duration=%u, points=%ld", st->id, (uint32_t)after_wanted, (uint32_t)before_wanted, (uint32_t)duration, points_wanted);
983             #endif

0x610bdf is in rrd2rrdr (web/api/queries/query.c:1580).
1575                                rrdr_convert_before_after_to_absolute(&after_requested, &before_requested, rrd_update_every,
1576                                                                      first_entry_t, last_entry_t);
1577                    }
1578                    freez(region_info_array);
1579                }
1580                return rrd2rrdr_fixedstep(st, points_requested, after_requested, before_requested, group_method,
1581                                          resampling_time_requested, options, dimensions, rrd_update_every,
1582                                          first_entry_t, last_entry_t, absolute_period_requested);
1583            } else {
1584                if (rrd_update_every != (uint16_t)max_interval) {

0x612f7e is in rrdset2value_api_v1 (web/api/formatters/rrd2json.c:72).
67              , uint32_t options
68              , time_t *db_after
69              , time_t *db_before
70              , int *value_is_null
71      ) {
72          RRDR *r = rrd2rrdr(st, points, after, before, group_method, group_time, options, dimensions);
73          if(!r) {
74              if(value_is_null) *value_is_null = 1;
75              return HTTP_RESP_INTERNAL_SERVER_ERROR;
76          }

0x5e4072 is in health_main (health/health.c:721).
716   
717                     if (unlikely(RRDCALC_HAS_DB_LOOKUP(rc))) {
718                         /* time_t old_db_timestamp = rc->db_before; */
719                         int value_is_null = 0;
720   
721                         int ret = rrdset2value_api_v1(rc->rrdset, NULL, &rc->value, rc->dimensions, 1, rc->after,
722                                                       rc->before, rc->group, 0, rc->options, &rc->db_after,
723                                                       &rc->db_before, &value_is_null
724                         );
725   

[marcelmfs]

Edit: Scratch that, I think we have enough information as it is.

OK, just to let you know that with those build options my monitoring broke altogether. None of my custom metrics were being plotted. As a matter of fact not even my central netdata instance was plotting correctly.

With the first build, I was seeing the following log entries:

2020-08-10 08:58:01: netdata INFO  : HEALTH : INTERNAL ERROR: rrd2rrdr() on netdata.dbengine_global_errors update every 1 with unaligned grouping sum (group: 600, resampling_time: 0, resampling_group: 1), after (got: 1597049280, want: 1597049280, req: 1597049279, db: 1597049279), before (got: 1597049879, want: 1597049879, req: 1597049879, db: 1597049879), duration (got: 600, want: 600, req: 600, db: 601), points (got: 0, want: 1, req: 1, db: 3996), got 'points' is not wanted 'points'
2020-08-10 08:58:01: netdata INFO  : HEALTH : INTERNAL ERROR: rrd2rrdr() on netdata.dbengine_global_errors update every 1 with unaligned grouping sum (group: 600, resampling_time: 0, resampling_group: 1), after (got: 1597049280, want: 1597049280, req: 1597049279, db: 1597049279), before (got: 1597049879, want: 1597049879, req: 1597049879, db: 1597049879), duration (got: 600, want: 600, req: 600, db: 601), points (got: 0, want: 1, req: 1, db: 3996), got 'points' is not wanted 'points'

[mfundul]

Edit: Scratch that, I think we have enough information as it is.

OK, just to let you know that with those build options my monitoring broke altogether. None of my custom metrics were being plotted. As a matter of fact not even my central netdata instance was plotting correctly.

With the first build, I was seeing the following log entries:

2020-08-10 08:58:01: netdata INFO  : HEALTH : INTERNAL ERROR: rrd2rrdr() on netdata.dbengine_global_errors update every 1 with unaligned grouping sum (group: 600, resampling_time: 0, resampling_group: 1), after (got: 1597049280, want: 1597049280, req: 1597049279, db: 1597049279), before (got: 1597049879, want: 1597049879, req: 1597049879, db: 1597049879), duration (got: 600, want: 600, req: 600, db: 601), points (got: 0, want: 1, req: 1, db: 3996), got 'points' is not wanted 'points'
2020-08-10 08:58:01: netdata INFO  : HEALTH : INTERNAL ERROR: rrd2rrdr() on netdata.dbengine_global_errors update every 1 with unaligned grouping sum (group: 600, resampling_time: 0, resampling_group: 1), after (got: 1597049280, want: 1597049280, req: 1597049279, db: 1597049279), before (got: 1597049879, want: 1597049879, req: 1597049879, db: 1597049879), duration (got: 600, want: 600, req: 600, db: 601), points (got: 0, want: 1, req: 1, db: 3996), got 'points' is not wanted 'points'

Those options shouldn't have broken anything, that's very strange... In any case, you can reinstall the official build via the official method to get back to where you were. If something has gotten very wrong and you can't seemingly bring monitoring up for any reason, a safe hint is to delete the cached metrics in /var/cache/netdata/dbengine/* but this means you delete all metric history.

[marcelmfs]

My history is being persisted long term in prometheus, so no harm done in deleting the cache.

Actually, my stop action in the systemd unit deliberately removes the cache every time a restart is executed.

[mfundul]

@marcelmfs if you see this problem again feel free to comment in this issue so that we can reopen it if needed.