significantly increased management traffic after upgrade to Gluon v2017.1.x

[rotanid]

this week we upgraded around 75% of our nodes to Gluon v2017.1.8, while about 20% were using this version already.
before, the nodes were running Gluon v2016.2.7.
(we still have ~20 nodes running Gluon v2016.1.x - i have no control over those, no autoupdate and no SSH)
in total, we have about 540 online nodes.

starting with the upgrade, the mgmt_tx of a single idle node without clients increased from 153 kbit/s to 254 kbit/s (by 66%) and the mgmt_rx from 57 kbit/s to 90 kbit/s (58%)

see the attached graph which shows the increased traffic with lowered spikes over all nodes, also the single node graph. the "all nodes" graph is capped at 1000 kbit/s to be able to see something useful, the spikes would make that impossible without the cap.

@ecsv also reported similar data from his community.

[ecsv]

It is currently unknown why this happens. I have two main suspects

• batman-adv sends more ogms for an unknown reason
• the changes in the wireless layer increased the reliability of broadcasts (which are used to transport OGMs)

The effect can only be seen when a significant portion nodes are updated from 2016.2.x to 2017.1.x

Regarding the batman-adv first point: Freifunk Vogtland increased its batman-adv version during the update from 2017.0.1+norebroadcast to 2018.1+maint+norebroadcast patches. Rotanid's update changed batman-adv from 2016.2.x+maint patches+norebroadcast to 2017.2+maint patches. I would therefore guess that the culprit (when batman-adv is the reason) has to be searched between batman-adv v2017.0.1..v2017.2

---

Maybe somebody is able to reproduce the problem and can install a modified gluon 2017.1.8 version on a lot of nodes in his network. It would be good when this person can then check whether returning to batman-adv 2016.2.x (from gluon 2016.2.x) reduces the number of mgmt overhead again. The three patches to revert to the older batman-adv version can be found in the branch https://github.com/FreifunkVogtland/gluon/commits/batadv/2016.2.x

[rotanid]

FYI, my "spikes" are back (like before the upgrade) - the management traffic remains at the higher level

[lephisto]

I think this also applies to v2018.1. Segmentation done with v2018.1 Rollout. Half the Nodes but increased management Traffic.

Update:

This describes it quite well. It is a node with nearly no Client Traffic. Orginator Interval has not changed..(5s)

[rotanid]

after updating almost all nodes (over 500) to Gluon v2018.1.x with batman-adv 2018.1, mgmt_tx increased by ~10kbit/s and mgmt_rx by ~40kbit/s, so it increased again, but not nearly as much as last time.

[neocturne]

@ecsv Have there been any new insights regarding this issue?

[ecsv]

At least I received no info and I also cannot provide any

[awlx]

We (Ffmuc) also see a heavy increase of mgmt traffic after updating from gluon v2016.2.7 to gluon2018.2.1.

We have about 1500 nodes and our mgmt traffic nearly doubled.

[neocturne]

@T-X Is it possible that this is related to the removal of the no_rebroadcast hack from our batman-adv?

[T-X]

@NeoRaider: Hm, no, the no_rebroadcast patch should only have touched layer 2 broadcast frames, but not mgmt traffic (OGMs). The no_rebroadcast patch was only applied on "batadv_send_outstanding_bcast_packet()".

Similarly, the new, automatic approach in batman-adv is only applied to layer 2 broadcast frames and to BATMAN_V OGMs. But not BATMAN_IV.

@awlx: Urgh, yeah, that graph looks ugly... Is that a capture from a Freifunk router or from a gateway? Would it be possible for you to provide a capture from the VPN interface, filtered by OGMs (with wireshark or tshark you should be able to filter by batadv packet type)?

[T-X]

Also, do you have link to the a site/domain config used before and after the update, @awlx?

[krombel]

Hi @T-X, we as ffmuc had the following configs:

• before: v2018.1
• after: v2019.0.1

In the hope to reduce that traffic we then disabled ULA and ibss in the hope it would help but the benefits of v2019.0.2 where quite minimal to not observable.

Concerning the graph @awlx posted: It was from our grafana: The average reported mgmt traffic of all nodes collected via respondd. So it was only observable for us as the release got rolled out to more nodes...

Concerning a capture: I will try to do a capture on the vpn interface. Do you have a link or a simple command I can use to create it?

[T-X]

Hi @krombel,

You can filter just for the mgmt (== OGMs, originator messages) with this command:

$ tcpdump -i mesh-vpn0 'ether proto 0x4305 and ether[14] = 0x00 and ether[15] = 0x0f' -w /tmp/mgmt.cap

The capture filter here means:

• ether proto 0x4305: The batman-adv ethernet frame type
• ether[14] = 0x00: The batman-adv packet type for OGMs (v1) as defined here for the first byte after the ethernet frame header as defined here
• ether[15] = 0x0f: The batman-adv compatibility version number (15, used since 2014) which is defined here for the second byte after the ethernet frame header as defined here

(An equivalent display filter for tshark or Wireshark would be eth.type == 0x4305 && batadv.batman.packet_type == 0x00 && batadv.iv_ogm.version == 15 in case you might be more familiar with those tools.)