[Regression][WPA2 Enterprise] WPA2 Enterprise connection fail (IDFGH-5866) #7565

PaulFreund · 2021-09-15T21:35:25Z

Hello,

our application has been working fine for over a year now. We used ESP IDF tags/v4.1 and now wanted to upgrade to tags/v4.3 (stable). The lowest tested version was tags/v4.1.2 which also was not able to connect to a WPA2 Enterprise network that works with the previous version of ESP IDF. I will try to provide more addditional information but only have limited time available. Is this a known regression?

I already tried the patch from this issue: #7384 and also disabling mbedTLS in WPA supplicant options.

negativekelvin · 2021-09-16T04:06:09Z

Have you tried latest release/v4.3 branch instead of the tag

PaulFreund · 2021-09-16T05:14:23Z

I have by now tried:
tags/v4.1 - WPA2Enterprise works but I can't use because of pSRAM issues
tags/v4.1.1 - WPA2Enterprise Broken
tags/v4.1.2 - WPA2Enterprise Broken
tags/v4.3 - WPA2Enterprise Broken
master - WPA2Enterprise Broken

I will test release/v4.3 as soon as possible

PaulFreund · 2021-09-16T05:42:37Z

Update:

release/v4.3 - WPA2Enterprise Broken

Little background, I helped debug WPA2E in the past in #2152 and #2324 and it was working without issues since then. Now there are a lot of devices relying on that functionality in a remote location so I have only limited testing capabilities

kapilkedawat · 2021-09-16T06:36:08Z

Hi @PaulFreund ,

Can you please try on the latest master once? Which patch from #7384 have you tried since there are multiple discussions in that?

Also please provide following:

Radius server details
Auth method used
SDK config

AxelLin · 2021-09-16T06:47:10Z

I have by now tried:
tags/v4.1 - WPA2Enterprise works but I can't use because of pSRAM issues
tags/v4.1.1 - WPA2Enterprise Broken

Can you bisect to find bad commit? (This is probably the fastest way to find the problem)
The changes between v4.1 and v4.1.1 in wpa_supplicats:
$ git shortlog v4.1..v4.1.1 --no-merges components/wpa_supplicant/
Angus Gratton (1):
wpa_supplicant: Fix failure to link under some circumstances

GOPTIONS\pfrost (1):
Reduce log level of hexdumps to verbose

Hrudaynath Dhabe (2):
wpa_supplicant: Fix configurable debug log feature's warning issue
wpa_supplicant: Minor bugfix with wpa_supplicant debug logs.

Jiang Jiang Jian (2):
Merge branch 'bugfix/clean_wpa2_wifi_deinit' into 'master'
Merge branch 'bugfix/wps_fail_reason_code' into 'master'

Nachiket Kukade (5):
wpa_supplicant: Allow NULL-padded WPS attributes
wpa_supplicant: Fix WPA3 and WPA2 transition related failures
wpa_supplicant: Increase PMK Lifetime to a very high value
espnow/pmf: Implement ESPNOW + PMF Co-existance
esp_wifi: Update wifi lib

kapil.gupta (6):
wpa_supplicant: WPS Inter operatability Fixes
wpa_supplicant: Support for mbedtls tls handshake
wpa_supplicant: Fix invalid pointer deference and memleak
wpa_supplicant: Fix IOT issue with latest freeradius
ESP-WIFI: Optimize 4way handshake failure time
wpa_supplicant: Fix null pointer dereference if eap init failed

ronghulin (1):
bugfix: fix softap mode wpa memory leak

PaulFreund · 2021-09-16T06:55:24Z

Hi @kapilkedawat

I already tried the latest master yesterday evening (CEST) which did not work.

We don't use client certificates and Radius server is part of Cisco infrastructure.
I didn't set it explicitly but in the past it was PEAP with MSCHAPv2
sdkconfig.txt

kapilkedawat · 2021-09-16T07:37:01Z

Hi @PaulFreund , Just to reconfirm both e8360fe and #7384 (comment) were present in master when you tested this?

Also is it possible to share sniffer capture and logs of the issue? Please note in case mbedTLS is enabled, Please enable logs for both mbedTLS and wpa_supplicant.

AxelLin · 2021-09-16T07:52:22Z

Hi @PaulFreund , Just to reconfirm both e8360fe and #7384 (comment) were present in master when you tested this?

I think it's not related unless v4.1 includes the fix of #7384 (comment)

PaulFreund · 2021-09-16T08:07:41Z

@kapilkedawat I was definitely on master which included your commit and I think I also tested with the fix enabled but not 100% sure. Unfortuantely I don't have physical access to the devices or can read serial output. I'm getting all log output by intercepting vsprintf and sending it to an InfluxDB.

I'm fine tuning the log output to not overwhelm the network (some devices are also connected via Ethernet) and it looks like the error is an assoc expire. I'll try to get more relevant output.

kapilkedawat · 2021-09-16T08:52:24Z

@PaulFreund If you are sure about the issue after disabling mbedTLS in tags/v4.1.1.

Can you please try following:

Revert this commit : wpa_supplicant: Fix IOT issue with latest freeradius
Disable mbedTLS in wpa_supplicant

and see if that helps?

Also are you aware of the TLS version cisco infra is using? Again, does the remote setup has any system which can help you to capture the packets over the air?

PaulFreund · 2021-09-16T09:58:25Z

I just tested with:

50b4cf9 (wpa_supplicant: Fix IOT issue with latest freeradius)
59347d6 (the commit before that)

I also tried before with tags/v4.3 and mbedTLS disabled but I will retest with that version

We should be able to extract information like the TLS version from the Posts I linked above, the infrastructure has not changed since then, I assume TLSv1 (there should even be some captures IIRC)

Small update: 0c13662 is my last tested working commit so far, I'm trying to do a binary search

kapilkedawat · 2021-09-16T10:27:05Z

I just tested with:

50b4cf9 (wpa_supplicant: Fix IOT issue with latest freeradius)

59347d6 (the commit before that)

Both didn't work(MbedTLS was disabled in supplicant config)?

I also tried before with tags/v4.3 and mbedTLS disabled but I will retest with that version

We should be able to extract information like the TLS version from the Posts I linked above, the infrastructure has not changed since then, I assume TLSv1 (there should even be some captures IIRC)

Even if you have older captures please share in case nothing has changed on the server side.

Small update: 0c13662 is my last tested working commit so far, I'm trying to do a binary search

Thanks, we will take a look internally as well.

PaulFreund · 2021-09-16T10:38:58Z

Sorry yes, both didn't work and CONFIG_WPA_MBEDTLS_CRYPTO is still unset (disabled in menuconfig)

I'm now at 573f5de and it does NOT work so it must be between 0c13662 and this one

PaulFreund · 2021-09-16T11:58:16Z

Okay I found the offending commit:

77eb201 - WORKING
e365d1f - NOT WORKING

Which means e365d1f is the commit that brought the issue. In addition I get

E (36877) wpa: Method private structure allocated failure

with this commit which was not present before

PaulFreund · 2021-09-16T12:21:56Z

The allocation failure only happens if CONFIG_WPA_MBEDTLS_CRYPTO is unset.

If MBEDTLS in supplicant is enabled it also does NOT work but the allocation failure does not happen.

Just to clarify,

in 77eb201 WPA2 Enterprise works both with MBEDTLS enabled and disabled
in e365d1f WPA2 Enterprise does not work both with MBEDTLS enabled and disabled

kapilkedawat · 2021-09-16T13:28:46Z

@PaulFreund Can you please share partition.csv file? We are trying to check this with the exact config on windows and linux radius servers.

PaulFreund · 2021-09-16T14:29:48Z

# Name,   Type, SubType, Offset,  Size, Flags
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs,      data, nvs,     0x9000,  0x4000,
otadata,  data, ota,     0xd000,  0x2000
phy_init, data, phy,     0xf000,  0x1000,
ota_0,    app,  ota_0,   0x10000, 0x1F0000
ota_1,    app,  ota_1,   0x200000,0x1F0000
# We currently waste 16000 bytes of data because of alignment

Sure :) Thank you for working on it, this is the content

kapilkedawat · 2021-09-16T16:32:02Z

@PaulFreund Unfortunately this is working for us with hostapd/freeradius.
From your linked issues logs, I can see TLS version was used as TLSv1 but since those are very old logs not sure this was due to esp devices or the server itself doesn't support new protocols.

A fresh capture/serial logs will help to determine the issue faster, Is there any chance we can get them?

PaulFreund · 2021-09-17T06:12:07Z

ESPIDFv4.3_mbedTLS_Enabled.csv

here is a first (anonymised) log of v4.3 with mbedTLS enabled and debug output of wpa and wifi

kapilkedawat · 2021-09-17T07:11:01Z

@PaulFreund I took a look at the logs, Can you please retry with the latest master? Please make sure d3a42d7 and patch mentioned in #7384 (comment) both are present? Please capture the logs again.

PaulFreund · 2021-09-17T08:17:48Z

I will do as soon as possible. Meanwhile, here is the log with mbedTLS disabled in wpa supplicant
ESPIDFv4.3_mbedTLS_Disabled.csv

PaulFreund · 2021-09-17T08:55:28Z

Here is the log of the current master with d3a42d7 included and the fix from the comment. Mbedtls is enabled in supplicant settings
ESPIDFv220590d599_mbedTLS_Enabled.csv

kapilkedawat · 2021-09-17T09:12:02Z

Hi @PaulFreund , do you use make instead of cmake by any chance?

PaulFreund · 2021-09-17T09:12:33Z

Hi, yes I still have to use make because of some legacy components

kapilkedawat · 2021-09-17T09:15:22Z

Can you please try this change when mbedTLS disabled? (CONFIG_SHA256 flag is missing in cflags)

diff --git a/components/wpa_supplicant/component.mk b/components/wpa_supplicant/component.mk
index c81e71993be..f3fe1a67c82 100644
--- a/components/wpa_supplicant/component.mk
+++ b/components/wpa_supplicant/component.mk
@@ -35,7 +35,7 @@ ifneq ($(CONFIG_WPA_11KV_SUPPORT), y)
src/esp_supplicant/esp_scan.o
endif

-CFLAGS += -DCONFIG_DPP -DCONFIG_IEEE80211W -DESP_SUPPLICANT -DIEEE8021X_EAPOL -DEAP_PEER_METHOD -DEAP_TLS -DEAP_TTLS -DEAP_PEAP -DEAP_MSCHAPv2 -DUSE_WPA2_TASK -DCONFIG_WPS2 -DCONFIG_WPS_PIN -DUSE_WPS_TASK -DESPRESSIF_USE -DESP32_WORKAROUND -DCONFIG_ECC -DCONFIG_WNM -D__ets__ -Wno-strict-aliasing
+CFLAGS += -DCONFIG_DPP -DCONFIG_IEEE80211W -DESP_SUPPLICANT -DIEEE8021X_EAPOL -DEAP_PEER_METHOD -DEAP_TLS -DEAP_TTLS -DEAP_PEAP -DEAP_MSCHAPv2 -DUSE_WPA2_TASK -DCONFIG_WPS2 -DCONFIG_WPS_PIN -DUSE_WPS_TASK -DESPRESSIF_USE -DESP32_WORKAROUND -DCONFIG_ECC -DCONFIG_WNM -D__ets__ -Wno-strict-aliasing -DCONFIG_SHA256

ifdef CONFIG_ESP32_WIFI_ENABLE_WPA3_SAE
CFLAGS += -DCONFIG_WPA3_SAE

PaulFreund · 2021-09-17T09:26:19Z

Yes that works with current master!

PaulFreund · 2021-09-17T09:36:48Z

I just tried to compile with mbedtls in current master but I get linking errors

/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(esp_wpa_main.o):(.literal.wpa_sta_disconnected_cb+0x0): undefined reference to `esp_wpa3_free_sae_data'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(esp_wpa_main.o):(.literal.esp_supplicant_init+0x48): undefined reference to `esp_wifi_register_wpa3_cb'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(esp_wpa_main.o): in function `wpa_sta_disconnected_cb':
/home/User/ApplicationName/esp-idf/components/wpa_supplicant/esp_supplicant/src/esp_wpa_main.c:221: undefined reference to `esp_wpa3_free_sae_data'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(esp_wpa_main.o): in function `esp_supplicant_init':
/home/User/ApplicationName/esp-idf/components/wpa_supplicant/esp_supplicant/src/esp_wpa_main.c:270: undefined reference to `esp_wifi_register_wpa3_cb'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(sha1-prf.o):(.literal.sha1_prf+0x0): undefined reference to `forced_memzero'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(sha1-prf.o): in function `sha1_prf':
/home/User/ApplicationName/esp-idf/components/wpa_supplicant/src/crypto/sha1-prf.c:64: undefined reference to `forced_memzero'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(sha256-prf.o): in function `sha256_prf_bits':
/home/User/ApplicationName/esp-idf/components/wpa_supplicant/src/crypto/sha256-prf.c:105: undefined reference to `forced_memzero'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(sha1-tlsprf.o): in function `tls_prf_sha1_md5':
/home/User/ApplicationName/esp-idf/components/wpa_supplicant/src/crypto/sha1-tlsprf.c:95: undefined reference to `forced_memzero'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/esp-idf/components/wpa_supplicant/src/crypto/sha1-tlsprf.c:95: undefined reference to `forced_memzero'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(sha1-tlsprf.o):/home/User/ApplicationName/esp-idf/components/wpa_supplicant/src/crypto/sha1-tlsprf.c:96: more undefined references to `forced_memzero' follow
collect2: error: ld returned 1 exit status
make: *** [/home/User/ApplicationName/esp-idf/make/project.mk:575: /home/User/ApplicationName/build/ApplicationName.elf] Error 1

kapilkedawat · 2021-09-17T10:02:46Z

I don't see any linking issue here, maybe retry after cleaning older project files? That may happen when os_xtensa.o is generated from the old config.

PaulFreund · 2021-09-17T10:10:32Z

Okay here is a small summary WITH the patch you posted here #7565 (comment), WITHOUT the patch from this comment #7384 (comment) and built with make:

ESP IDF version	WPA Supplicant MBEDTLS enabled	WPA2 Enterprise works	Others
master	true	-	Linker errors
master	false	true
tags/v4.3	true	false
tags/v4.3	false	true

PaulFreund · 2021-09-17T10:12:31Z

I always deleted my ./espressif and build directory before starting a new test to not get any old state. But maybe in that test the sdkconfig was slightly different.

Anyway, I think I can work with v4.3 + mbedtls disabled + your patch for now. Would be interresting why mbedtls does not work though

PaulFreund · 2021-09-17T10:17:53Z

@kapilkedawat Thank you so much! You saved my weekend :) 👍

PaulFreund · 2021-09-17T10:20:39Z

Because the application is quite universal I have to do a lot of testing before doing a release. Is it possible to backport this fix to release/v4.3?

kapilkedawat · 2021-09-17T10:22:09Z

Hi @PaulFreund , for the mbedTLS part, Is it possible for you to share this packet?

wpa: SSL: 146 bytes left to be sent out (of total 146 bytes) Excel line : 516, seems like this is causing AP to send a fail message.

Also if the server is reporting some error, please let me know.

Yes, we will backport this till v4.0 and all versions will have this fix.

PaulFreund · 2021-09-17T10:55:42Z

I will put it on my todo list, is it enough to enable verbose for wpa to get the data?

kapilkedawat · 2021-09-17T11:06:11Z

Yes, Please set both supplicant and MbedTLS to verbose level.

Alvin1Zhang · 2021-10-13T02:46:06Z

Thanks for reporting, fix on master branch is available at 71a5003.

AxelLin · 2021-10-22T02:41:30Z

I will put it on my todo list, is it enough to enable verbose for wpa to get the data?

@PaulFreund
Any update with mbedTLS enabled?

AxelLin · 2021-11-27T06:04:15Z

@PaulFreund

Since it was working in 77eb201, this looks like a regression.
Any chance to provide the log with mbedTLS enabled?

Alvin1Zhang · 2022-01-05T06:30:21Z

@PaulFreund Thanks for reporting, would you please help share if any further updates? Thanks.

Alvin1Zhang · 2022-04-19T01:42:27Z

Thanks for reporting, will close due to short of feedback, feel free to reopen with more updates.

espressif-bot added the Status: Opened Issue is new label Sep 15, 2021

github-actions bot changed the title ~~[Regression][WPA2 Enterprise] WPA2 Enterprise connection fail~~ [Regression][WPA2 Enterprise] WPA2 Enterprise connection fail (IDFGH-5866) Sep 15, 2021

espressif-bot added Resolution: NA Issue resolution is unavailable Status: Done Issue is done internally Resolution: Done Issue is done internally and removed Status: Opened Issue is new Resolution: NA Issue resolution is unavailable labels Sep 30, 2021

Alvin1Zhang added the Awaiting Response awaiting a response from the author label Jan 5, 2022

Alvin1Zhang closed this as completed Apr 19, 2022

[Regression][WPA2 Enterprise] WPA2 Enterprise connection fail (IDFGH-5866) #7565

[Regression][WPA2 Enterprise] WPA2 Enterprise connection fail (IDFGH-5866) #7565

Comments

PaulFreund commented Sep 15, 2021

negativekelvin commented Sep 16, 2021

PaulFreund commented Sep 16, 2021

PaulFreund commented Sep 16, 2021 • edited

kapilkedawat commented Sep 16, 2021 • edited

AxelLin commented Sep 16, 2021

PaulFreund commented Sep 16, 2021

kapilkedawat commented Sep 16, 2021 • edited

AxelLin commented Sep 16, 2021

PaulFreund commented Sep 16, 2021

kapilkedawat commented Sep 16, 2021

PaulFreund commented Sep 16, 2021 • edited

kapilkedawat commented Sep 16, 2021

PaulFreund commented Sep 16, 2021

PaulFreund commented Sep 16, 2021

PaulFreund commented Sep 16, 2021 • edited

kapilkedawat commented Sep 16, 2021

PaulFreund commented Sep 16, 2021

kapilkedawat commented Sep 16, 2021

PaulFreund commented Sep 17, 2021

kapilkedawat commented Sep 17, 2021

PaulFreund commented Sep 17, 2021

PaulFreund commented Sep 17, 2021

kapilkedawat commented Sep 17, 2021

PaulFreund commented Sep 17, 2021

kapilkedawat commented Sep 17, 2021

PaulFreund commented Sep 17, 2021

PaulFreund commented Sep 17, 2021

kapilkedawat commented Sep 17, 2021

PaulFreund commented Sep 17, 2021

PaulFreund commented Sep 17, 2021

PaulFreund commented Sep 17, 2021

PaulFreund commented Sep 17, 2021

kapilkedawat commented Sep 17, 2021 • edited

PaulFreund commented Sep 17, 2021

kapilkedawat commented Sep 17, 2021

Alvin1Zhang commented Oct 13, 2021

AxelLin commented Oct 22, 2021

AxelLin commented Nov 27, 2021

Alvin1Zhang commented Jan 5, 2022

Alvin1Zhang commented Apr 19, 2022

PaulFreund commented Sep 16, 2021 •

edited

kapilkedawat commented Sep 16, 2021 •

edited

kapilkedawat commented Sep 16, 2021 •

edited

PaulFreund commented Sep 16, 2021 •

edited

PaulFreund commented Sep 16, 2021 •

edited

kapilkedawat commented Sep 17, 2021 •

edited