Wi-Fi Proxy ARP breaks poorly implemented DHCP clients

[Ambr051us]

Describe the bug

The "Proxy ARP" feature of hostapd (tested with wpad-wolfssl and wpad-mbedtls) breaks DHCP duplicate address detection on some clients. DHCP clients are supposed to send out an "ARP Who Has" message with the IP address offered to them. IF this so-called "Gratuitous ARP request" gets a response, that means the offered address is already in use and the client sends a DHCPDECLINE. There are some poor implementations out there (many of which are closed-source of course), which send a DHCPACK very quickly after the gratuitous ARP request, maybe even before it. (I assume that hostapd only starts answering ARP requests on behalf of clients once it has seen a DHCPACK or some ARP packet indicating the address being in use, so the bug technically happens on the clients here). This can lead to hostapd answering the gratuitous ARP causing the client to decline the address it just ACK-ed and try again.

I'm opening this because it's easier to fix hostapd than all the bad DHCP implementations that are already out in the wild. What hostapd does (sending an ARP response on behalf of a client to the same client) is not violating any spec strictly speaking, but it would be sane not to do. I'm not familiar with the hostapd codebase and not even sure whether it has an actively maintained upstream (Wikipedia lists 3 actively maintained implementations). I've run into the issue using OpenWRT so I came here, I hope someone knows where the "if requesting MAC address matches the proxy response's MAC address, don't send the proxy response" condition can be added to the code.

OpenWrt version

r23630-842932a63d

OpenWrt release

23.05.2

OpenWrt target/subtarget

mediatek/mt7622

Device

Xiaomi Redmi Router AX6S

Image kind

Official downloaded image

Steps to reproduce

1. Enable 'Proxy ARP' on an SSID, preferably on 2.4 GHz (I've never seen this happen on 5 GHz). Have a DHCP server running in the network somewhere (having it on a separate device from the one running the wireless AP probably increases the chance of seeing the bug)
2. Connect a client to the SSID, which has a "poor" implementation of DHCP. TP-Link Tapo cameras are a known example, possibly also iPhone 8 and older Apple products.
3. Observe the infinitely repeating DHCPDISCOVER->DHCPOFFER->DHCPACK->DHCPDECLINE cycle with a packet sniffer or in the DHCP server's log

Actual behaviour

An infinitely repeating cycle of DHCPDISCOVER -> DHCPOFFER -> DHCPACK -> Proxy_ARP response to gratuitous request -> DHCPDECLINE

Expected behaviour

Never answering ARP requests with the same address as the requester

Additional info

No response

Diffconfig

No response

Terms

• I am reporting an issue for OpenWrt, not an unsupported fork.

[rany2]

I've observed it only on devices that use the official SDK for ESP8266. I think clients should do ICMP ping not ARP who has, its a flaw in their dhcp client implementation.

Edit: to clarify, I think it's mentioned RFC... the client isn't following the spec properly

[brada4]

how do they find dst mac for ping without who-has?
Gratuitous ARP != ARP request, it is kind of arping tells all subnet i have this mac for that ip from now on. Like arping or similar in ha cluster suites.
please randomize mac-s involved and make a pcap. It is hard to g€t what is happening. If device chokes on own arp announcment reflected back it is defective and needs fixed. Or use static IPs. Or disable arp proxy though anyone in same network can knock device out at will.

[Ambr051us]

Sorry, I had a wrong understanding of Gratuitous ARP, thanks for clearing that up @brada4
The client device indeed "chokes on its own ARP announcement reflected back at it." You are absolutely correct in that this is defective behavior and needs to be fixed (DHCPDENY after DHCPACK is not allowed AFAIK). My problem is that a locked-down proprietary device will never get fixed, and I've personally seen 3 different ones exhibit this faulty behavior. The only viable path for improvement seems to be the open source component, since that can actually be changed, unlike the faulty clients. And the workaround that would allow Proxy ARP to coexist with these faulty clients is conceptually quite simple: don't reply to grat. ARP announcements / arping broadcast frames with unicast frames addressed to a host that's itself known to have the IP address in question. Basically "if (outgoing_arp_reply_frame.payload_ip_addr == proxy_arp_map_mac_to_ip(outgoing_arp_reply_frame.destination_mac_addr)) drop(outgoing_arp_reply)"
Now of course I'm not expecting to be serviced by the OSS community like some sort of Karen, just trying to put the concept out there. If I can find the time to acquaint myself with the dev environment of hostapd (which to me appears to be quite a bit more obscure than the average open source project, so some pointers would be very much appreciated), I'll do my best to submit a PR myself.

[rany2]

@Ambr051us I suggest configuring your DHCP server to always give those faulty devices a random IPv4 address as a workaround and a really short lease.

[brada4]

Just disable ARP proxy as it reflects back gratuitous responses. Does not protect against those spoofed.

[Ambr051us]

@rany2 that would make it worse actually. The devices give up and settle for what they think is a duplicate IP address after about 10 minutes of discover-offer-ack-deny ping-pong. Making them renew the address they worked so hard to accept would be detrimental.
@brada4 Yeah, that's how my "2.4GHz IoT crap" SSID is operating right now, with Proxy ARP turned off. 20 or so thermometers, radiator valves, light bulbs, etc. are making more noise (i.e. channel airtime utilization) in the 2.4GHz band than a 4K video stream does on a WiFi 6 channel. Proxy ARP makes it measurably better, especially in terms of battery life. But now that I'm typing this out, I've just realized that I can simply make a separate "even crappier 2.4 GHz IoT crap" SSID (:facepalm:) with Proxy ARP turned off, and have most of the stuff still benefit from being left alone with ARP chatter.
Sorry for wasting your time I guess, and thanks for helping me figure this out :D

[brada4]

Connect unencrypted device and observe traffic in the air. Unless it is clear what type of atypical ARP message locks up IoT it is impossible to fix. Once you have that message there is a slim hope to filter out killer packet with arptables+client isolation till hostapd gets improved. You will need 1x1 2.4ghz card on same channel in monitor mode. Like oldest laptop you are able to power on.
EDIT obviously you dont want any physical function attached to ESP32 for this test.

[nbd168]

Please test if this kernel patch helps: https://nbd.name/p/0d9379f6

[rany2]

@nbd168 I confirm the issue is resolved on my ESP8266 devices that have this problem. Thank you for fixing this!

[Ambr051us]

I'll definitely check too when I get home from work

[Ambr051us]

@nbd168 I ended up solving my problem by adding netfilter rules with the specific MAC addresses of the 2 misbehaving IP cameras that I own. Thanks @brada4 for pointing out that arptables exists :D
Being able to apply this fix through UCI without dealing with the hassle of compiling a custom kernel was too enticing. But it's awesome to see idea getting picked up by linux-netdev/testing, hats off for that to everyone involved 🙇‍♂️

[brada4]

You need to introduce new not-inet AF table then drop output response arp daddr eq ether daddr. It travels outside scope of fw4 inet table.

[Ambr051us]

I can't seem to get nft to accept any non-const expression on the right hand side of the eq operator, and the nftables wiki only lists examples where some packet attribute is matched against a hard-coded constant. So I added arp daddr ether <known address> arp saddr ether <known address> for both specific devices. Is there some syntax I'm unaware of that would allow a general saddr == daddr match?

[nbd168]

Please also test the latest version of the patch that I pushed to OpenWrt. Compared to the earlier version, it fixes an issue with neigh entry refcounting.

[rany2]

@nbd168 Works fine, I already tested the patchwork version as well. rany2/openwrt@b4e9e23