wifi: EAP re-authentication does not work (IDFGH-345) #2324
Comments
FayeY
changed the title
|
Hi @mikaelkanstrup this is because our enterprise is a unique task, after you call esp_wifi_sta_wpa2_ent_enable(), the wpa_enterprise task create, after finish the EAP step, the task will delete itself. However, the task would create automatic when you re-authentice, you need enable the task at first then connect. |
|
@mikaelkanstrup Sorry, the task wouldn't create itself when you re-connect. You need enable the task at first by call esp_wifi_sta_wpa2_ent_enable() and then connect |
|
@XinDeng11 Thanks for the sharing the info! Though I'm not sure how to treat your answer. Do you believe this is an application side error or esp-idf internal error? For me this is an esp-idf internal error. The steps to make the initial connection already involves calling esp_wifi_sta_wpa2_ent_enable and then wifi_connect. You have the example code so just check the sources if anything is unclear there. After the connection has been established there's no way for application side to know when the network will initiate an EAP re-authentication. You say the unique task to perform EAP steps is deleted after finishing the initial authentication. This sounds wrong. Some task should still be running and listen for EAP Request Identity data packets from the network that then starts the EAP re-authentication procedure. All of this I would expect is handled internally. BTW are planning on making any of this open source? I would be happy to help out investigating problems in this area but without source code I can only guess based on the behavior seen. |
|
We need discuss it internal at first |
|
@XinDeng11 Please have a look at the attached files in computer-reauth-success.tar.gz These logs (airsniffer logs, wpa_supplicant and hostapd) show my computer successfully keeping its wifi connection through 5 reauthentications. Compare that with the earlier logs from ESP32 where the connection drops when re-authentication times out. To be able to decrypt all traffic you wil need to configure Wireshark with the following keys: That's six different keys as after initial key negotiation the keys are re-negotiated another 5 times. |
|
I was told that this issue continues through email, so I am going to close it. Please feel free to reopen, if I misunderstood it. Thanks. |
|
@FayeY @XinDeng11 I think this issue should be kept open as EAP re-authentication is still not working. We've discussed it via email and we've received information about a possible workaround (i.e. by re-connecting successfully after disconnect). The workaround definitely makes this issue less critical so let's continue the discussion here. Keeping it open also show other users of ESP-IDF that this is still an issue. |
ghost
mentioned this issue
|
Hi, implementing reauth would help me a lot (I can stop setting up a parallel infrastructure without WPA2 Enterprise), is this something you are working on? |
ghost
mentioned this issue
projectgus
changed the title
|
This issue is still present on latest master as of today. Are there any plans to support this? |
|
This bugfix is internally ready-to be merged and should be available on master soon. We will update once done. |
@sagb2015 Ok that's really nice to hear! I'd gladly support testing it in our environment if you want some extra verification before merging. |
|
@mikaelkanstrup Thanks for offering the help, but it needs both IDF and wifi-lib patches and compatible versions. The changes should be available soon (less than a week time). You can test it then and provide the feedback. |
|
@sagb2015 Ok sure. I'll try it out once available on master and report back. |
|
Apologies for the delay! This has been fixed by 3b606aa. Please help to test. |
|
Closing this issue. Feel free to open if you still face it. |
|
Thanks for implementing this! Just wanted to confirm that reauth now works. In my test setup I've just completed ~50 successful re-authentications without dropping wifi connection. |
|
Thank you for your help in testing! |
|
Unfortunately it does not seem to fix the problem with the Cisco AP's we have here. I try to make a verbose log as soon as possible. (I had >150 disconnects over the weekend) |
|
@PaulFreund I have not been able to properly verify towards Cisco APs due to #3956. That issue makes connection attempts fail very often with recent esp-idf master. Most likely it affects both initial authentication as well as re-authentications. Unfortunately there's at the moment no version where EAP re-auth is supported but the commit causing #3956 is not also integrated. Due to the above issue I've only verified EAP re-auth towards a local test setup running hostapd with an internal radius server. In this setup all works fine. |
|
I now have a log with wifi set to verbose: Was more of the code open sourced now? Would it be possible to help debugging now? By now we have 16 separately managed routers in our building just to cope with this. |
|
@PaulFreund @mikaelkanstrup : We will prioritise #3956 internally and see if there is any impact on this issue. |
|
@PaulFreund Yes, there's some more code open sourced now. And most of the EAP handling is in supplicant. With debug logs enabled you get huge amounts of supplicant logs. Note that debug logs include hexdump of encryption keys if you plan to share any logs. You get a somewhat reasonable level of debug logs for this by leaving the function wpa_hexdump in components/wpa_supplicant/src/utils/wpa_debug.c empty. Like this: And setting Component config>Log output>Default log verbosity>Debug With this you should see the EAP states, eapol frames, 4 way handshake, key installs etc happening. Hopefully this will tell what's wrong. |
|
Here is the log of the reauthentication with disconnect: I think Is the last event before it goes haywire |
|
It appears 2/4 message of 4-way hanshake sent by ESP32 is not received by Cisco AP since it is retrying message 1/4. Is it possible for you to provide a sniffer capture for re-authentication failure case? |
|
At the moment not unfortunately but to me it looks like the AP is actually replying to the 2/4 but with a different payload: Working: Not working: There is a difference of 34 bytes in the key data length coming from the AP and the key info in working case is "Pairwise Install Ack MIC Secure Encr" instead of the non working "Pairwise Ack", which means in key_info are not set. I would still need to read the specification about that points but it seems to me that in one case (the first one) the router wants to install and therefore includes more information in the key data and in the second case it expects the information to be already present. |
|
If I am not mistaken we are in the wpa_sm_rx_eapol function of wpa.c (line 1495) where the output of the log information is wpa_eapol_key_dump (line 1558) and later there is the check starting at line 1658: as WPA_KEY_INFO_KEY_TYPE is set (Pairwise is in the log) and MIC not this should be the reason why wpa jumps back to step 1. |
|
@PaulFreund - In the failure case, the router did not get message 2/4 from ESP32. So it is retrying by sending 1/4 to ESP32 again. "Pairwise Ack" case is actually msg 1 out of 4. To check why 2/4 was not received by router a sniffer capture would have helped a lot. For 4-way handhshake, you can refer to this article. |
|
I want to make sure we are really on the same page, here is a diff of the working case vs the non working case: http://www.mergely.com/i6XsuLvv/ with exactly the same starting point in the sequence but different outcome: Small breakdown of the logs: This should be the AP sending the ANonce, initiating the handshake. "IEEE 802.1X RX" indicates that this comes from the wpa_supplicant receive function which means the data comes from the AP Here the client uses the information to derive the PTK and sends the MIC to the AP (wpa_supplicant_send_2_of_4 function). Until here both logs are 100% identical, AP sent ANonce, STA sent MIC and then in both logs the STA receives an answer (indicated by "IEEE 802.1X RX"). In the working log this is: Which means that the EAPOL message includes the MIC and the status changed for installed key, secure and encrypted connection. In the re-authentication log the next message we get from the AP is again identical to the start of the handshake and does not include the flags we expect. As this behaviour is 100% consistent for all re-authentication tries (I logged in the hundereds) I am pretty sure the AP actually receives the message but is not happy with the message flow and therefore restarts. Maybe there is some alternative flow for the handshake in case of a re-authentication. For actual sniffing I need more time. |
|
I did not find free access to the full spec yet but I found this:
Just as an idea, are we starting/continue with a new sequence counter for this transaction or are we incrementing the one from the last transaction? Maybe the CISCO AP expects we continue with an increment of the last counter value from the previous transaction or the other way around and other AP's don't check this. |
|
After seeing that @PaulFreund still had issues with the fix for this issue I looked more closely at some sniffer logs from the tests I did to confirm this issue was solved. Then found that I was wrong. The fix (3b606aa) does handle re-authentication but not the way it should be done. The re-authentication shall be done with frame exchange encrypted with currrent keys. esp-idf do all 4-way handshakes unencrypted. Only the initial one can be. Please see the attached logs and sniffer logs. Sniffer logs for both a test with wpa_supplicant under linux and with esp_supplicant attached. For decryption with Wireshark the following PMKs can be used: Compare unencrypted 4-way handshake messages 2/4 and 4/4 from ESP in esp_supplicant.pcapng: 102, 103, 104, 127, 129, 131, 132 With same frames encrypted with wpa_supplicant.pcapng frames: 268, 270 |
|
@mikaelkanstrup Thanks for testing this. I am reopening the issue. The spec is not very explicit about encryption of re-authentication EAP frames. I found the following statement in the spec which is consistent with your expectation. "This standard assumes that IEEE Std 802.1X-2010 does not block the Controlled Port when authentication is triggered through reauthentication. During IEEE 802.1X reauthentication, an existing RSNA can protect all MSDUs exchanged between the STAs. Blocking MSDUs is not required during reauthentication over an RSNA." Theoretically, the security o f re-authentication without encryption should be as good as during first authentication as 802.1x does not distinguish between the two during the handshake. |
|
@sagb2015 Thanks, agree that "the security of re-authentication without encryption should be as good as during first authentication". However as EAPOL Key frames are transferred as data frames I think IEEE 802.11-2016 Chapter 12.9 Per-frame (and sub-chapters) rules should apply here. Check: These I interpret as when dot11RSNAActivated=true:
|
|
@sagb2015 Can you create a test branch with this suggestion? I would like to test it as soon as possible. |
|
Sorry for bumping, I might have the chance for integration in a broader infrastructure soon which will most likely require this to work and it seems that the solution is near. Can we follow up on this please? |
|
We have fixed this internally. Unfortunately, it may take some time to be available on github. Hoping to get it done by early next week. |
|
I'm really, really sorry to bump this again but are there any updates? |
|
And I am sorry that we could not make the fix available sooner. We have quite a CI backlog internally, We will prioritise this and get it merged this week. |
|
Hi @PaulFreund, We still have a lot of CI backlog, so this may take some more time. Can you use this lib for testing at commit 93a8603? This way you can get unblocked. |
|
@sagb2015 Thank you very much! Unfortunately it seems (at least with the old make system) librtc.a is missing git rev-parse HEAD shows only the lib content has been modified |
|
Please keep the librtc.a same. Just replace the files, do not delete existing ones. |
|
Perfect, so far we are a little bit over 30 minutes and I did not have a disconnect, I'm hopeful :) |
|
Okay I have to investigate a little more, after one hour the device is no longer online at all. I will get back to you. |
|
Okay maybe it was really a connection issue, now I'm running three hours without problems, I'll keep you updated :) I will roll out the test firmware on more devices tomorrow |
|
:) Looks good! I had 5 devices running over the weekend and three didn't have any reconnect at all. The other two had a few but without any relation to the half hour seen before. I think they just have worse reception (office complex). Thank you so much! Please keep me updated on merging to master/release |
|
Thanks for the update! We will keep you posted. |
|
This is now available on master 04e024b |
|
Feel free to re-open in case of issues. |
|
With your linked master commit 04e024b I was able to observe both, working and non-working re-authentication. this is shortly before the logs differ: WorkingNot workingI just see a sudden change in state but nothing else with debug level. Once I am able to use the most recent master I will try to get some more data. |
|
Seems like connection station lost AP connection in the failure case. I do not think it is related to EAP (re)authentication. |
Environment
Problem Description
EAP re-authentication is non-functional. EAP/enterprise wifi networks may be configured to re-authenticate at certain intervals to regenerate new crypto keys. This function does not work on ESP32 causing wifi disconnects. Also after being disconnected the chip appears stuck in some invalid state causing it to never perform new connect attempts.
Expected Behavior
Actual Behavior
Steps to repropduce
eap_reauth_periodin hostapd.conf to a low value (like 60) to have the network perform re-authentication often.Debug Logs
Other items if possible
Attached files (logs.tar.gz:
To decrypt traffic in air sniffer packet log (eapol packets for EAP re-auth are encrypted) use the following PMK 25638c633b709113ec73401621f08a8ca6a1d01193e9997cd44ed8dd9c49b27e
If wireshark is used the decrypt key can be configured under:
Edit->Preferences->Protocols->IEEE 802.11->Decyption keys->Edit...
Add a WPA-PSK key with the key above.
The text was updated successfully, but these errors were encountered: