Captive portal (walled garden) network access via Pico W

[sgbaird]

Essentially what's described in https://forum.micropython.org/viewtopic.php?f=15&t=10405 for a Pico W or similar. Basic network access using SSID and password can be done via connecting to the internet with Pico W, but I didn't come across anything obvious to me that would deal with captive portals. I don't know the right terminology (other than captive portal + walled garden) to understand how to navigate this. Even if it's just a simple case where logging into a network requires a username and password in addition to the SSID would be very helpful. Being able to connect to eduroam using user-specified credentials would solve a ton of use-cases.

• https://inrg.soe.ucsc.edu/howto-connect-raspberry-to-eduroam/ (only for SBCs, not microcontrollers)
• https://www.reddit.com/r/raspberry_pi/comments/2frf15/connecting_raspberry_pi_to_campus_internet/
• https://www.reddit.com/r/raspberry_pi/comments/71xumd/is_anyone_having_problems_connecting_raspberry_pi/

In some cases, would it be as simple as using username@ssid and password?

Wondering if micropython-wifi-setup might be useful here.

[george-hawkins]

Hello @sgbaird - you also asked this question by way of an issue against my micropython-wifi-setup project but I'll close that and respond here.

My project will not help you get a Micropython device connected to a network that uses a captive portal - it actually implements a captive portal on a Micropython device in order to use the ability of such portals to push a client to a given web page. The device starts in AP mode, the end user connects to it as they would to any new wireless network, and their phone then pushes them to a captive portal page where they provide the details of e.g. their home or work Wi-Fi network - the Micropython device then shuts down AP mode and connects to that network.

I think you're out of luck if you're hoping to find some library that will help you automatically connect a Micropython device to a network that itself uses a captive portal requiring the user to identify themselves in some way.

Why am I so pessimistic? Because captive portals are completely arbitrary in how they work - think of all the different ones you've interacted with - in coffee shops, airports, colleges etc. There's no standard behind them - they provide a login page that humans can navigate but there's no corresponding standardized API for machines to interact with.

Even getting to the login pages requires quite a lot of client smarts - your phone goes thru some heuristics to work out that it's connected to a captive portal and lets the portal push it to a login page (via redirects that it would not allow in a normally connected scenario). After you've reached the login page, it's completely arbitrary what happens next - some will ask you to enter your phone number so you can receive a code via SMS which you then enter, others require pre-configured usernames and passwords or that you enter credit card details etc.

At this point, there's nothing going that's even vaguely related to any Wi-Fi standard. In fact, even the step that pushes you to the initial login page is not standardized. Captive portals started as a confusing redirect trick - you'd connect to the network, you'd start browsing and instead of getting to the page you wanted you'd confusingly be redirected to a captive portal login page. This was done by spoofing the DNS response for the host of the page you wanted such that you ended up at the captive portal site instead.

This spoofing worked back in the Wild West days of the web but nowadays it's not appreciated, instead, phones and laptops now try and work out if they're in a captive portal setup and if they are they essentially willingly participate in the redirection deception as they understand it's part of the process of getting connected. But it's all done with fairly hacky heuristics.

Your only option really is to use standard mode to connect your Micropython device to the network while also running it in AP mode so that your phone can connect to it and then get the Microphython device to tunnel web traffic between the network it's connected to and your phone such that you can complete the steps, needed to get the device out of the captive portal phase, with your phone.

[sgbaird]

Hi @george-hawkins, I really appreciate your feedback as a lot of this is new to me.

My project will not help you get a Micropython device connected to a network that uses a captive portal - it actually implements a captive portal on a Micropython device in order to use the ability of such portals to push a client to a given web page. The device starts in AP mode, the end user connects to it as they would to any new wireless network, and their phone then pushes them to a captive portal page where they provide the details of e.g. their home or work Wi-Fi network - the Micropython device then shuts down AP mode and connects to that network.

Thanks for clarifying this! This helps a lot, and I think I may use this for something separate.

I think you're out of luck if you're hoping to find some library that will help you automatically connect a Micropython device to a network that itself uses a captive portal requiring the user to identify themselves in some way.

Why am I so pessimistic? Because captive portals are completely arbitrary in how they work - think of all the different ones you've interacted with - in coffee shops, airports, colleges etc. There's no standard behind them - they provide a login page that humans can navigate but there's no corresponding standardized API for machines to interact with.

This makes a lot of sense given the lack of any standard way of interacting with these pages.

Your only option really is to use standard mode to connect your Micropython device to the network while also running it in AP mode so that your phone can connect to it and then get the Microphython device to tunnel web traffic between the network it's connected to and your phone such that you can complete the steps, needed to get the device out of the captive portal phase, with your phone.

What libraries/resources would you suggest to get started here? Using a phone or laptop as a go-between for initial setup seems like an acceptable workaround. After the initial setup, the phone or laptop could be disconnected afterward, correct?

[george-hawkins]

I've been giving this a bit of thought and again I'm going to switch to completely pessimistic.

I think you essentially have to implement a transparent proxy on the Micropython device. This is a non-trivial task - I looked to see if anyone had already done it. I found other people asking about this but no solutions.

Alternative solutions:

• Create your own Wi-Fi network (obviously with no captive portal component) to which you can connect your IoT devices. Then you have to connect the access point of this network to the Wi-Fi network that's got the problematic captive portal. So you've moved the problem from your IoT device to your access point - but it's easier to solve here as access points are full-blown computers (that can run Linux variants like DD-WRT or OpenWrt). You'd have to look into how to connect such a router to another Wi-Fi network and handle navigating the captive portal phase - I can't advise but I'm sure it can be done.
• You could spoof the MAC address of your IoT device with your laptop, complete the captive portal phase using it and then swap in the IoT device.
• Wait for Wi-Fi Easy Connect to become a real thing. I see Espressif do support it (see here) but I don't see it becoming available anytime soon via Micropython and I suspect there's very little support for it out in the wild.

I'm inclined to think MAC address spoofing is the easiest way to go. E.g. I found this page where someone describes the process.

[karfas]

I can't help with parsing HTML responses.

As this eduroam thing is handled completely from wpa_supplicant in the linked Linux examples, I suspect there is some standard authentication (WPA-EAP, MSCHAPv2) in place.

Maybe someone can tell something about the status and planned features of WPA-Enterprise with micropython ?

[sgbaird]

@george-hawkins Thanks for thinking about this some more. Agreed that spoofing seems like the easiest way to deal with arbitrary captive portals, and then there's the question of whether violates the terms of use for the network(s) in question. EDIT: after a quick look, seems like it might not be such a problem, though it's still a concern.

@karfas for eduroam specifically, I'm also wondering if there's a current (or near-future) way to do WPA authentication.

P.S. I posted a question on Raspberry Pi stack exchange to improve visibility and get some additional perspectives. I might give MAC spoofing with the Pico W a try at some point.

[sgbaird]

In addition to possibly violating ToS, MAC spoofing might not be feasible with the Pico W anyway with Eduroam-style logins. For example, University of Toronto's network has the following dialogue (i.e., SSID + Username + Password) as opposed to just SSID + Password. There's no way that I know of to specify a username with a Pico W

I think I'll just need to get a Hologram SIM card with a SIM-enabled router.

[andrewleech]

I agree with everything said already about supporting arbitrary captive portals, but maybe supporting just eduroam would be possible. I can definitely see a library for just eduroam having great utility in the Uni / school / education space.

The login web page that's being shown would just be a form that's submitted, likely via a post request.

Trying to connect to one of these on a laptop should get you the login page on a regular browser. With Dev tools open there, grabbing a copy of the html and recording the network tab while logging in should get enough information to replicate that flow on device.

Each different eduroam install may have a different server address the login is posted to, however as this system is built on fake DNS it may actually be the same on all.

[sgbaird]

This would still hinge on the SDK being updated to support SSID+username+password authentication (assuming that's feasible/possible), right? With Eduroam, there's also the issue of certificates. With microcontrollers, there's also the non-trivial issue of keeping credentials secure.