Inbound packets dropped ratio constantly exceeds alarm limit

[photis]

This issue is similar to #4536, which was closed, so I hope I'm doing the right thing filing a new issue. I'm seeing a large number of packets dropped too.

RX packets:11791910 errors:0 dropped:76076 overruns:0 frame:0

Netdata constantly sounds the alarm because my ratio of dropped packages for the last 10 minutes exceeds 0.1%

I found this on the Intahrnet:

Beginning with kernel 2.6.37, it has been changed the meaning of dropped packet count. 
Before, dropped packets was most likely due to an error. Now, the rx_dropped counter 
shows statistics for dropped frames because of:

Softnet backlog full
Bad / Unintended VLAN tags
Unknown / Unregistered protocols
IPv6 frames when the server is not configured for IPv6
[...]

If the rx_dropped counter stops incrementing while tcpdump is running; then it is more 
than likely showing drops because of the reasons listed earlier.

Meanwhile ifconfig has this to say:

$ sudo ifconfig eth0
eth0      Link encap:Ethernet  HWaddr f4:4d:30:6d:93:92
          inet addr:192.168.1.201  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::f64d:30ff:fe6d:9392/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:11791910 errors:0 dropped:76076 overruns:0 frame:0
          TX packets:17464164 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2459415409 (2.2 GiB)  TX bytes:17603379495 (16.3 GiB)
          Interrupt:20 Memory:aa100000-aa120000

But ethtool finds no fault:

$ sudo ethtool -S eth0 | grep rx
     rx_packets: 11788295
     rx_bytes: 2458306572
     rx_broadcast: 139088
     rx_multicast: 129038
     rx_errors: 0
     rx_length_errors: 0
     rx_over_errors: 0
     rx_crc_errors: 0
     rx_frame_errors: 0
     rx_no_buffer_count: 0
     rx_missed_errors: 0
     rx_long_length_errors: 0
     rx_short_length_errors: 0
     rx_align_errors: 0
     rx_flow_control_xon: 0
     rx_flow_control_xoff: 0
     rx_csum_offload_good: 11637919
     rx_csum_offload_errors: 6
     rx_header_split: 0
     alloc_rx_buff_failed: 0
     rx_smbus: 0
     rx_dma_failed: 0
     rx_hwtstamp_cleared: 0

Would it be best to just disable this alarm, or is there indeed cause for concern and should I try to dig deeper? Thanks.

Netdata version: v1.12.0-rc2-33-g54beb88 - commit: 4505c9f
Debian Linux 8
Linux 3.16.0-7-686-pae on i686
Intel(R) Core(TM) i3-5010U CPU @ 2.10GHz, 4 cores

[Ferroin]

Based on what you've shown, that works out to about 0.6% of all received packets since the last time that system rebooted being dropped. For anything above about 0.1%, you generally want to figure out why they're dropped, because even though it's usually benign, it's still wasting network bandwidth and processing time.

Unfortunately, I'm not quite sure where to start on this one. Checking if tcpdump or some other packet capture program that can put the interface in promiscuous mode makes the drop counter stop increasing would be useful, but you've got a low enough percentage of drops that it would be hard to be certain that it's not just luck that caused them to stop. Double-checking the configuration for all the systems on that network would be useful too, as stuff like this is usually a result of some system not being properly configured.

[cakrit]

For me, tcpdump did make the packet drops disappear, as @Ferroin said. I went through https://serverfault.com/questions/780195/dropped-packets-in-all-linux-and-unix looking for an answer.
I remember that a few weeks ago I had used tcdump and traced the issue I saw to broadcast packets coming from my router every 12sec, like the following:

11:32:36.879101 d8:b6:b7:d9:fc:74 (oui Unknown) > Broadcast, ethertype Unknown (0x88e1), length 60: 
11:32:36.879101 d8:b6:b7:d9:fc:74 (oui Unknown) > Broadcast, ethertype Unknown (0x88e1), length 60: 
        0x0000:  0000 a000 b052 0000 0000 0000 0000 0000  .....R..........
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............
11:32:36.879108 d8:b6:b7:d9:fc:74 (oui Unknown) > Broadcast, ethertype Unknown (0x88e1), length 60: 
        0x0000:  0000 a000 0000 16e8 0600 0000 0000 0000  ................
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............

They don't bother me, so I didn't look any further.

[meguroyama]

I had to increase the net.core.netdev_budget to 100k for the issue to go away.

[chrisspiegl]

I am experiencing this as well. The machine is relatively low traffic but shows (what looks like one dropped package) dropped packages every 30 seconds (see screenshot).

It recently got to the point of being a constant alert and I am wondering where that is coming from. I tried doing some research about it and also found the above mentioned serverfault article but don't really know how to move forward now.

Is this something that is internal to Netdata, system updates, or changes in how things are reported? Do we need to adjust in some way?

I am running everything on Ubuntu with the netdata version v1.12.0-22-nightly.

[cakrit]

Is this something that is internal to Netdata, system updates, or changes in how things are reported? Do we need to adjust in some way?

Netdata is telling you the truth. I haven't seen anyone requesting that we adjust the alarm threshholds in the stock configuration. A tcpdump should tell you where it's coming from, it looks like you have a very similar situation to mine.

[chrisspiegl]

@cakrit Thank you, I have tried looking into those things. I'm just struggling to figure out what it actually is. When I am running the tcpdump those 30 second package drops go away.

I also tried disabling the IPv6 on the server and that didn't change a thing. So there is that. I might just endup disabling the alert if I can't figure it out myself 🙈.

[Ferroin]

@chrisspiegl If using tcpdump reliably makes the drops disappear, then the packets getting dropped are probably not actually intended for your system. Assuming it's not a router, check the destination MAC addresses and IP addresses for the packets recorded by tcpdump and make sure they all match up with the interface they're coming in over. I suspect you'll see some that don't match, and those are almost certainly what's getting dropped.

[chrisspiegl]

@Ferroin thank you for that explanation. It makes sense now and I understand it better. I will investigate and contact my hoster if it continues.

[TomVanBerlo]

We have the same issue and after some investigation broadcast and multicast packets with vlan tags that were not configured on our interface were the cause of the dropped packets. Tcpdump made the drops disappear. What can be done to fix this? I can silence the alarm but if there are dropped packets intended for the interface I won't be notified.

root@ubuntu:~# date; cat /proc/net/dev; sleep 5; date; cat /proc/net/dev;
Fri Jul 26 09:47:56 CEST 2019
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
  ens3: 3137648   20127    0 3377    0     0          0         0  5155300    3582    0    0    0     0       0          0
    lo: 2488228    5932    0    0    0     0          0         0  2488228    5932    0    0    0     0       0          0
Fri Jul 26 09:48:01 CEST 2019
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
  ens3: 3141292   20161    0 3389    0     0          0         0  5159130    3590    0    0    0     0       0          0
    lo: 2495523    5947    0    0    0     0          0         0  2495523    5947    0    0    0     0       0          0

This is on a test machine with low network traffic but it generates a critical alarm. net_packets.ens3.inbound_packets_dropped_ratio = 17.7%.
We are concerned that we might receive these alarms when our systems are hardly used.