Linux installer: EAP authentication breaks on recent Linux versions (tested on Ubuntu 26.04, among others) — stale embedded CA bundle fails cert validation

[m4r4d0n4]

The Linux installer pins RADIUS server certificate validation to a CA bundle embedded in the
script at generation time (Config.CA, passed as 802-1x.ca-cert). When the certificate
chain presented during EAP-TLS ends up not covered by that embedded bundle, EAP authentication
fails instantly and 100% reproducibly with no useful error surfaced to the user —
NetworkManager just reports no-secrets, which misleadingly suggests a credentials problem.
We've confirmed this on Ubuntu 26.04 (NetworkManager 1.54.3); given the mechanism is generic
(OpenSSL/wpa_supplicant chain-building from a standalone ca-cert file), we'd expect it to
reproduce on any recent Linux distro shipping current NetworkManager/wpa_supplicant.

This is a regression, not a longstanding issue: the same installer, same embedded CA
bundle, and same credentials work correctly without any changes on older Ubuntu releases and
on many older Linux distro/NetworkManager/wpa_supplicant combinations we tested against. The
failure only shows up on recent versions, which points at a behavior change somewhere in how
newer OpenSSL/wpa_supplicant build the trust chain from a standalone ca-cert PEM file (rather
than at anything wrong with the embedded bundle itself, which validates fine elsewhere).

This is distinct from #354 (NUL-terminated file:// URIs) — we found this with the NUL fix
applied and it still failed, confirming it's a separate root cause. See our writeup in that
PR thread.

Important environment detail: in our deployment, this failure only happens when the client
authenticates through a RADIUS proxy on an AP controller — specifically Extreme Networks
XCC and Identity Engines sitting in front of our RADIUS backend. Access points from other
vendors on the same network (Aerohive and Juniper/Mist) connect successfully with the
same installer, same embedded CA bundle, same credentials — the only difference being
those APs don't route authentication through that proxy layer. This strongly suggests the issue
is in how the certificate chain is presented/reconstructed by the Extreme RADIUS proxy
(possibly re-signing or forwarding a different/longer chain than the backend RADIUS server
would present directly), not in the RADIUS backend itself or in the client's EAP config.

Environment

• Fails on: Ubuntu 26.04 LTS, NetworkManager 1.54.3 (netplan-managed), wpa_supplicant 2.11.
• Works fine on: older Ubuntu releases and other, older Linux/NetworkManager/wpa_supplicant
  combinations, with the exact same installer and CA bundle — narrowing this down to a change
  in recent NM/wpa_supplicant/OpenSSL versions, not the institution's cert setup.
• Institution realm urjc.es, EAP-PEAP/MSCHAPv2.
• Fails: APs behind Extreme Networks XCC / Identity Engines RADIUS proxy.
• Works: APs behind Aerohive and Juniper controllers (no RADIUS proxy hop for auth).

Evidence

CTRL-EVENT-EAP-PEER-CERT depth=1 subject='/C=AT/O=ZeroSSL GmbH/CN=ZeroSSL RSA DV SSL CA 2'
CTRL-EVENT-EAP-PEER-CERT depth=0 subject='/CN=europa.urjc.es'
CTRL-EVENT-EAP-TLS-CERT-ERROR reason=1 depth=1 subject='...ZeroSSL...' err='unable to get local issuer certificate'
CTRL-EVENT-EAP-FAILURE EAP authentication failed

The server's cert chains through a ZeroSSL intermediate to Sectigo Public Server
Authentication Root R46. The bundle embedded in our institution's installer (Comodo AAA +
Sectigo R46 + HARICA) technically already includes that same Sectigo root — yet
The server's cert chains through a ZeroSSL intermediate to Sectigo Public Server
Authentication Root R46. The bundle embedded in our institution's installer (Comodo AAA +
Sectigo R46 + HARICA) technically already includes that same Sectigo root — yet
wpa_supplicant/OpenSSL fails to build the trust chain when it's supplied as a loose ca-cert
file, but only for the APs whose auth goes through the Extreme XCC/Identity Engines proxy, and
only on recent NM/wpa_supplicant versions. We haven't fully pinned down why (possibly the proxy
presents a different/incomplete chain, or newer OpenSSL is stricter about chain building from a
standalone PEM), but the failure is deterministic and 100% reproducible on that path.

What worked

Using the OS's own trust store instead of the embedded bundle:

 s_8021x_data = {
     'eap': [Config.eap_outer.lower()],
     'identity': self.user_data.username,
-    'ca-cert': dbus.ByteArray(f"file://{self.cacert_file}\0".encode()),
+    'system-ca-certs': True,
     match_key: match_value}

Ubuntu already trusts Sectigo R46 by default, so this validates correctly and connects in ~5s,
including through the Extreme proxy path that previously failed. Tested end-to-end (profile
deleted and recreated from scratch).

Discussion

We understand system-ca-certs=True trades "pin exactly the CA the institution intended" for
"trust whatever the OS trusts", which is a real regression in strictness — we've paired it with
domain-suffix-match on our end to limit the blast radius. But this still doesn't explain why
only the Extreme XCC/Identity Engines proxy path fails, only on recent NM/wpa_supplicant, while
direct-to-backend paths (Aerohive, Juniper) and older client versions succeed with the exact
same embedded bundle. Options we can see:

1. Default to system-ca-certs=True and drop the embedded bundle entirely — simplest, works
   around the issue regardless of which proxy/controller or client version is involved, but
   loses the pinning CAT is presumably designed to provide.
2. Keep the embedded bundle as the primary ca-cert, but fall back to system-ca-certs if the
   embedded one fails to validate — closer to current behavior, but adds installer complexity.
3. If this turns out to be specific to how Extreme XCC/Identity Engines proxies present the
   chain (truncated/reordered intermediates) combined with stricter chain-building in recent
   OpenSSL/wpa_supplicant, that may warrant a separate report to Extreme rather than a CAT-side
   fix — but from the client's perspective the installer's rigid pinning to a single embedded
   bundle is what turns a proxy quirk (that only newer clients are sensitive to) into a hard
   authentication failure, so we think it's still worth CAT being more resilient here regardless
   of root cause on the network side.

Happy to open a PR for whichever direction maintainers think fits CAT's design goals, and can
gather more diagnostics on the proxy side if useful (e.g. packet capture of the EAP-TLS
handshake through the Extreme proxy vs. direct, and on older vs. newer clients).

Reproduction

1. Generate an installer for an institution whose deployment includes a RADIUS proxy layer on
   the AP controller (in our case Extreme XCC / Identity Engines) in front of the actual RADIUS
   backend.
2. Connect via an AP that routes authentication through that proxy, from a recent Linux client
   (e.g. Ubuntu 26.04) — observe instant CTRL-EVENT-EAP-TLS-CERT-ERROR /
   unable to get local issuer certificate in wpa_supplicant debug logs despite correct
   credentials.
3. Connect via an AP that does not route through the proxy (in our case Aerohive/Juniper), or
   from an older Linux client — same installer, same credentials, connects successfully.
4. Reconfigure the NetworkManager profile with 802-1x.system-ca-certs=yes and no ca-cert —
   connects successfully through both paths.

[twoln]

Falling back to trusting the whole system CA store would be a serious change and would definitely make the Linux installer operate differently from others. Pinning the exact root CA is an important security feature. Of course we could depend on the system store if we could point to an exact certificate there (like this is done in Windows installers). Dependency on supplied certificates is also crucial for private CAs which are quite common in eduroam.
I believe we should first try to understand what is happening with these particular access points. This is actually very strange. The Access point is supposed to be interested only in the RADIUS protocol messages and it simply passes the whole EAP conversation to the final client. The whole CA related information is contained inside of the EAP conversation which is also encrypted.

[twoln]

Could you the the getduroam app and see how this behaves in your environment? https://github.com/geteduroam/linux-app/releases

[m4r4d0n4]

Thank you for your quick responses.

Let me try it with that installer — I've been using the Python script generated for our
institution at https://cat.eduroam.org/, so I haven't tested the geteduroam app yet.

I'll test it against both paths we've been comparing (behind the Extreme XCC/Identity Engines
proxy, and behind Aerohive/Juniper) to see whether the same certificate validation failure shows
up there too.

Will report back with results.

[m4r4d0n4]

Just tested with the .deb from the geteduroam app you linked
(https://github.com/geteduroam/linux-app/releases) — it connects successfully in our
environment, we will do further testing to see if it works properly with XCC/Identify Engines.

One thing worth flagging: to get it to run at all on Ubuntu 26.04, we had to unlock/disable
AppArmor confinement first — otherwise it wouldn't start/execute. Not sure if that's a known
issue on your end or something worth reporting separately against the geteduroam app itself, but
wanted to mention it here in case it's relevant context.

[jwijenbergh]

geteduroam linux maintainer here, this could be: geteduroam/linux-app@1cb028a

and

geteduroam/linux-app@eed0a01

to expand on the commit message, blobs don't work as they only support one ca cert (missing feature of wpa supplicant that is in distros. That is fixed in https://git.w1.fi/cgit/hostap/commit/?id=9bb25b01fe843924113883a93236d43bb77d4cb1 )

there is no new hostap tagged for this to be fixed.

Only recently NM, started passing these as blobs, hence why you see the issue on 26.04.

see the commit description.

Maybe we should report all of this to upstream somehow.

One thing worth flagging: to get it to run at all on Ubuntu 26.04, we had to unlock/disable
AppArmor confinement first — otherwise it wouldn't start/execute. Not sure if that's a known
issue on your end or something worth reporting separately against the geteduroam app itself, but
wanted to mention it here in case it's relevant context.

This is not a known issue, I will try it when I have the time. Did you try the flatpak too?

Important environment detail: in our deployment, this failure only happens when the client
authenticates through a RADIUS proxy on an AP controller — specifically Extreme Networks
XCC and Identity Engines sitting in front of our RADIUS backend. Access points from other
vendors on the same network (Aerohive and Juniper/Mist) connect successfully with the
same installer, same embedded CA bundle, same credentials — the only difference being
those APs don't route authentication through that proxy layer

I don't have an explanation for this. Maybe you need to check the diff of the NM profile that CAT creates and geteduroam creates. Maybe the aforementioned 'blob' issue is not the case here. But considering geteduroam works, I am suspecting it does. What about ubuntu 24.04?

What if you create a dir of CA files, set the ca-path instead of ca-cert and then run openssl rehash on the ca-path? Does that make CAT connect too?

[restena-sw]

It is a complication here that the issue occurs only when two conditions are met simultaneously: newest Ubuntu && going through specific proxy products.

For the latter, I would not be (overly) surprised if there were such proxy products that actually terminate the EAP conversation and generate a similar-looking but counterfeit CA+server cert. I.e. become a full man-in-the-middle of the RADIUS conversation. Only clients that correctly validate the CA chain would notice, and the failure described here in this bug report would match such a MITM situation.

The mystery then remains why this MITM by the proxy is only attempted for specific clients. It is hard to find arguments why that would be the case.

One suggestion to help dismantle the problem: in your debug log, you only see /strings/ identifying the CA, like /C=AT/O=ZeroSSL GmbH/CN=ZeroSSL RSA DV SSL CA 2.
This is NOT a cryptographically correct way for validating the CA. Any (counterfeit) CA could be crafted to match the string you see, but be different anyway (different keypair). I'm intrigued by seeing no depth=2 ... line that would chain the first intermediate to the Sectigo certificate. If the supplicant does not make the link, maybe there is something wrong with the intermediate.

To get clarity whether the CA/server cert is actually modified by the proxy, one would need to dump the full certificates to a file for later inspection.

If you are able to use eapol_test (part of wpa_supplicant, doing EAP without the Wi-Fi parts, directly from cmdline), then calling wpa_supplicant with its "-o file" option dumps the certificates.

Essentially, a cmdline like

eapol_test -p 1812 -C'eduroam testing' -N89 -t 25 -a 127.0.0.1 -s mysecret -c wpa_supplicant.conf -M22:44:66:00:99:00 -o dumpedcertificates.pem

gives you the full certificate chain, as sent by the server (or the suspected MITM proxy). Of course you'll need to register your device as a RADIUS client with the RADIUS server and supply the server IP and your shared secret in -a and -s.

Alternatively, you might be able to get the certificates via the wpa_supplicant control interface on wpa_cli, see this thread: https://lists.infradead.org/pipermail/hostap/2021-October/039960.html

If you do get a hold of the certificates, compare them to the ones the server actually sends. Any mismatch would be very suspicious.

[restena-sw]

Given that you are an eduroam participant, I tried this with realm @urjc.es

Feeding eapol_test with either the R46 root CA, or with the R46 cross-sign CA towards the old Comodo AAA root:

both variants produce debug output indicating that the R46 certificate is used for chain building.

Root:

`CTRL-EVENT-EAP-PEER-CERT depth=2 subject='/C=GB/O=Sectigo Limited/CN=Sectigo Public Server Authentication Root R46' hash=7bb647a62aeeac88bf257aa522d01ffea395e0ab45c73f93f65654ec38f25a06

CTRL-EVENT-EAP-PEER-CERT depth=1 subject='/C=AT/O=ZeroSSL GmbH/CN=ZeroSSL RSA DV SSL CA 2' hash=ec9d6bcab9818b7dd59ed61da23a658a96a9c4e950edee0d2da93e7cf2174780

CTRL-EVENT-EAP-PEER-CERT depth=0 subject='/CN=europa.urjc.es' hash=d296b36c6328c25af8d75a358c6dd32792d4bf23645be31bfc685c5e946a3c1e

CTRL-EVENT-EAP-PEER-ALT depth=0 DNS:europa.urjc.es`

Cross-signed, ultimately failing because the Comodo Root CA is not also trusted:
`
CTRL-EVENT-EAP-PEER-CERT depth=2 subject='/C=GB/O=Sectigo Limited/CN=Sectigo Public Server Authentication Root R46' hash=92f351bf3d54164dfa8dd8f9e1139d3150349786485d2b9eecd00e2971c1e6c5

CTRL-EVENT-EAP-PEER-CERT depth=0 subject='/CN=europa.urjc.es' hash=d296b36c6328c25af8d75a358c6dd32792d4bf23645be31bfc685c5e946a3c1e

CTRL-EVENT-EAP-PEER-ALT depth=0 DNS:europa.urjc.es

TLS: Certificate verification failed, error 2 (unable to get issuer certificate) depth 2
`

(I don't know why the eapol_test output does not include the original intermediate "depth=1" cert here. It is really not present in the debug output; it was not an omission on my side)

Your debug output does NOT indicate the "depth=2" part of chain building. This really makes it look like the C=AT... intermediate you see is not chained to the issuer certificate you expect. Or that a bug in wpa_supplicant simply doesn't feed the trusted CA list into the validation process in the end. Maybe you are, in the end, still looking at the #354 bug?

[jwijenbergh]

Your debug output does NOT indicate the "depth=2" part of chain building. This really makes it look like the C=AT... intermediate you see is not chained to the issuer certificate you expect. Or that a bug in wpa_supplicant simply doesn't feed the trusted CA list into the validation process in the end. Maybe you are, in the end, still looking at the #354 bug?

did you see: #385 (comment)

it really sounds like that.

[restena-sw]

Ah, I didn't realise the ca-cert blob contains more than one cert (it could contain only the R46 root; as the intermediate is sent in the EAP conversation).

But yes, in that case, it would be explained why only Ubuntu 26.04 has an issue with rooting up to the CA.

But even then, it does NOT explain why this is only an issue when going through these special proxies? For me, that is really the strangest part of it all.

[jwijenbergh]

But even then, it does NOT explain why this is only an issue when going through these special proxies? For me, that is really the strangest part of it all.

Correct. I am really puzzled by this one. So maybe the 'blob issue' is not relevant here after all?

[m4r4d0n4]

Hi, thank you for all your information.

Following up after more testing — I need to correct something in my original report.

Retraction on the AP-vendor correlation: after testing more broadly, the certificate validation failure is not actually specific to APs behind the Extreme XCC/Identity Engines proxy. I conflated it with a separate, unrelated problem (some VR headsets failing to connect to eduroam on those specific platforms, for reasons unrelated to this bug). The EAP-TLS chain-building failure affects our whole eduroam/eduroam2 wifi infrastructure uniformly, regardless of which AP/controller relays the RADIUS traffic. Sorry for the noise — this actually resolves the part @jwijenbergh and @restena-sw were stuck on: there's no proxy-specific mechanism here, so a client-side bug not depending on the AP is exactly what you'd expect.

Root cause, confirmed: our embedded Config.CA bundle was never outdated — it already contains Sectigo Public Server Authentication Root R46 (verified directly with openssl x509 on each cert in the bundle). The failure matches @jwijenbergh's diagnosis: recent NetworkManager (1.54.3 on Ubuntu 26.04) passes a multi-certificate ca-cert PEM to wpa_supplicant as a blob, and wpa_supplicant's blob mechanism only supports a single certificate — so the 3-CA bundle gets effectively truncated and chain-building fails.

Fix confirmed working, without touching system trust: instead of system-ca-certs=True (which @twoln correctly flagged as not appropriate), we switched to 802-1x.ca-path. Concretely, in the installer's __add_connection:

- s_8021x_data['ca-cert'] = dbus.ByteArray(f"file://{self.cacert_file}\0".encode())
+ s_8021x_data['ca-path'] = self.cacert_dir

and in save_ca(), each certificate in the bundle is now written to its own file, with subject-hash symlinks generated the same way openssl rehash/c_rehash would:

  with open(certfile, 'w') as cert:
      cert.write(Config.CA + "\n")

+ # one file per certificate under cat_installer/ca/
+ for index, cert_pem in enumerate(certs):
+     with open(os.path.join(ca_dir, f'ca-{index}.pem'), 'w') as f:
+         f.write(cert_pem + "\n")
+ subprocess.run(['openssl', 'rehash', ca_dir], check=True)

Since NetworkManager always passes ca-path as a directory path — never as a blob — this sidesteps the bug entirely while keeping the exact same 3-CA pinning as the original installer, same approach geteduroam used (geteduroam/linux-app@1cb028a, @eed0a01). Tested end-to-end across our infrastructure: profile deleted and recreated, connects and validates correctly.

[twoln]

This seems to be the reasonable approach. It will require an addition of the full config directory cleanup since we do not want to leave any old certificates there. I will implement this before we put the new CAT version into production (which is planned for the beginning of August).

[m4r4d0n4]

Good point, and it's already handled — I should have mentioned it in the previous comment. save_ca() wipes the ca/ directory before writing anything on every run:

ca_dir = get_config_path() + '/cat_installer/ca'
os.makedirs(ca_dir, 0o700, exist_ok=True)
for old in os.listdir(ca_dir):
    os.remove(os.path.join(ca_dir, old))
# ... then write the current Config.CA certs and re-run `openssl rehash`

So re-running the installer (e.g. after regenerating it with an updated CA bundle) always starts from an empty directory — no stale certificates or leftover hash symlinks from a previous CA bundle can survive a re-run. This matters precisely because ca-path trusts everything it finds in the directory, so a CA that gets dropped from Config.CA needs to actually disappear from disk, not just get outnumbered by new files.

Glad the approach works for the August rollout — let me know if it'd help to have this as a PR against the installer generator rather than just the diff here.

Thanks a lot for the work and the time spent digging into this with us — really appreciate it.

[twoln]

I can work with the diffs. Thank you very much for your input.