Crash on Android 10 with underscore in subdomain #5840

talklittle · 2020-03-03T07:53:20Z

Crash on OkHttp 3.12.9 and 3.12.10 when a URL subdomain contains underscores. Reproducible on API 29 (Android 10) emulator but not API 28 (Android 9).

On OkHttp 3.12.8 it errors but does not crash. Similarly OkHttp 3.12.10 errors but does not crash on Android 9.

Reproduce by making a GET request to an HTTPS URL with underscore in subdomain.

I saw old issue #242 closed saying underscores aren't allowed in hostnames, which may be technically true, but they are sometimes seen in the wild. Examples:

https://open_nsfw.gitlab.io/ (which was highly circulated on the web a few years ago)
https://example_underscore_123.s3.amazonaws.com/
https://instant_regret.reddit.com/

Logcats:

OkHttp 3.12.10 on Android 10:

--------- beginning of crash
E/AndroidRuntime: FATAL EXCEPTION: OkHttp Dispatcher
Process: com.talklittle.testokhttpunderscoresubdomain, PID: 10321
java.lang.IllegalArgumentException: Invalid input to toASCII: example_underscore.s3.amazonaws.com
    at java.net.IDN.toASCII(IDN.java:115)
    at javax.net.ssl.SNIHostName.<init>(SNIHostName.java:99)
    at com.android.org.conscrypt.Platform.getSSLParameters(Platform.java:153)
    at com.android.org.conscrypt.ConscryptFileDescriptorSocket.getSSLParameters(ConscryptFileDescriptorSocket.java:1117)
    at okhttp3.internal.platform.Android10Platform.configureTlsExtensions(Android10Platform.java:40)
    at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:315)
    at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:284)
    at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:169)
    at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258)
    at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
    at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:201)
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:919)
 Caused by: The input does not conform to the STD 3 ASCII rules. line:  0. preContext:  . postContext: e_underscore

    at android.icu.impl.IDNA2003.convertToASCII(IDNA2003.java:219)
    at android.icu.impl.IDNA2003.convertIDNToASCII(IDNA2003.java:277)
    at android.icu.text.IDNA.convertIDNToASCII(IDNA.java:654)
    at java.net.IDN.toASCII(IDN.java:108)
    at javax.net.ssl.SNIHostName.<init>(SNIHostName.java:99) 
    at com.android.org.conscrypt.Platform.getSSLParameters(Platform.java:153) 
    at com.android.org.conscrypt.ConscryptFileDescriptorSocket.getSSLParameters(ConscryptFileDescriptorSocket.java:1117) 
    at okhttp3.internal.platform.Android10Platform.configureTlsExtensions(Android10Platform.java:40) 
    at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:315) 
    at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:284) 
    at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:169) 
    at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258) 
    at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) 
    at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) 
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) 
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) 
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) 
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) 
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) 
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) 
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) 
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) 
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127) 
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) 
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) 
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257) 
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:201) 
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) 
    at java.lang.Thread.run(Thread.java:919)

OkHttp 3.12.8 on Android 10, error but no crash:

javax.net.ssl.SSLHandshakeException: java.lang.IllegalArgumentException: Invalid input to toASCII: example_underscore.s3.amazonaws.com
    at com.android.org.conscrypt.ConscryptFileDescriptorSocket.startHandshake(ConscryptFileDescriptorSocket.java:231)
    at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:320)
    at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:284)
    at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:169)
    at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258)
    at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
    at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:201)
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:919)
Caused by: java.security.cert.CertificateException: java.lang.IllegalArgumentException: Invalid input to toASCII: example_underscore.s3.amazonaws.com
    at com.android.org.conscrypt.ConscryptFileDescriptorSocket.verifyCertificateChain(ConscryptFileDescriptorSocket.java:415)
    at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(Native Method)
    at com.android.org.conscrypt.NativeSsl.doHandshake(NativeSsl.java:387)
    at com.android.org.conscrypt.ConscryptFileDescriptorSocket.startHandshake(ConscryptFileDescriptorSocket.java:226)
        ... 23 more
Caused by: java.lang.IllegalArgumentException: Invalid input to toASCII: example_underscore.s3.amazonaws.com
    at java.net.IDN.toASCII(IDN.java:115)
    at javax.net.ssl.SNIHostName.<init>(SNIHostName.java:99)
    at com.android.org.conscrypt.Platform.getSSLParameters(Platform.java:153)
    at com.android.org.conscrypt.ConscryptFileDescriptorSocket.getSSLParameters(ConscryptFileDescriptorSocket.java:1117)
    at com.android.org.conscrypt.TrustManagerImpl.getTrustedChainForServer(TrustManagerImpl.java:352)
    at android.security.net.config.NetworkSecurityTrustManager.checkServerTrusted(NetworkSecurityTrustManager.java:94)
    at android.security.net.config.RootTrustManager.checkServerTrusted(RootTrustManager.java:89)
    at com.android.org.conscrypt.Platform.checkServerTrusted(Platform.java:224)
    at com.android.org.conscrypt.ConscryptFileDescriptorSocket.verifyCertificateChain(ConscryptFileDescriptorSocket.java:407)
        ... 26 more
Caused by: The input does not conform to the STD 3 ASCII rules. line:  0. preContext:  . postContext: e_underscore
    at android.icu.impl.IDNA2003.convertToASCII(IDNA2003.java:219)
    at android.icu.impl.IDNA2003.convertIDNToASCII(IDNA2003.java:277)
    at android.icu.text.IDNA.convertIDNToASCII(IDNA.java:654)
    at java.net.IDN.toASCII(IDN.java:108)
        ... 34 more

The text was updated successfully, but these errors were encountered:

yschimke · 2020-03-03T08:04:43Z

I've reported upstream. I'm not sure this is something we really want to fix in OkHttp

https://github.com/square/okhttp/blob/okhttp_3.12.x/okhttp/src/main/java/okhttp3/internal/platform/Android10Platform.java#L34-L47

  @SuppressLint("NewApi")
  @IgnoreJRERequirement
  @Override public void configureTlsExtensions(
      SSLSocket sslSocket, String hostname, List<Protocol> protocols) {
    enableSessionTickets(sslSocket);

// Fails here
    SSLParameters sslParameters = sslSocket.getSSLParameters();

    // Enable ALPN.
    String[] protocolsArray = Platform.alpnProtocolNames(protocols).toArray(new String[0]);
    sslParameters.setApplicationProtocols(protocolsArray);

    sslSocket.setSSLParameters(sslParameters);
  }

We would catch a specific error due to a specific bug/strict handling in Conscrypt

yschimke · 2020-03-03T08:07:02Z

Prior to the fix for Android R blacklisting methods, we didn't read the SSL Parameters, we only set the value.

https://github.com/square/okhttp/blob/okhttp_3.12.x/okhttp/src/main/java/okhttp3/internal/platform/AndroidPlatform.java#L123

      setHostname.invokeOptionalWithoutCheckedException(sslSocket, hostname);

yschimke · 2020-03-03T08:11:53Z

If it turns out to be fixed in Android 11 (v30), then we could bump the version check to >= 30, and keep using the Gray listed methods on Android 10 (v29). Since this is a regression in that sense.

Or catch this specific exception on all 3 branches, with 3 different patches IIUC.

https://github.com/square/okhttp/blob/okhttp_3.12.x/okhttp/src/main/java/okhttp3/internal/platform/Android10Platform.java#L68

  public static @Nullable Platform buildIfSupported() {
    try {
      if (getSdkInt() >= 29) {
        Class<?> sslParametersClass =
            Class.forName("com.android.org.conscrypt.SSLParametersImpl");

        return new Android10Platform(sslParametersClass);
      }
    } catch (ClassNotFoundException ignored) {
    }

    return null; // Not an Android 10+ runtime.
  }

swankjesse · 2020-03-03T12:06:54Z

Underscores in domains work in Firefox and Chrome; they should work in Android! We should ask AOSP to fix.

But if Conscrypt won't handshake these URLs anyway, we should catch the IllegalArgumentException and rethrow whichever SSLException subtype we were throwing previously. Throwing unchecked exceptions deep inside of OkHttp is bad news.

yschimke · 2020-03-03T12:09:29Z

I'll put up 3 PRs but it might take a couple of days. Free for someone else to pick it up in the meantime :)

prbprbprb · 2020-03-03T13:35:23Z

I'll raise an AOSP bug for this, but it might be a struggle to get it changed in time for Android 11 (that said, the AOSP bug will probably come back to me anyway, so maybe ;)

Background: on Android IDN.toASCII() is implemented using ICU's IDNA2003 classes which are (a) deprecated[*] and (b) adhere very strictly to the rules so getting changes made upstream will be hard. E.g. in the case of the root domain name . (which different IDN implementations treat differently anyway) it ended up being easier to special-case it in the IDN class.

Special casing underscores would be uglier and probably only work for ASCII domain names, e.g. something like: If IDNA rejects the name and it contains underscores, strip them and try again. If the second attempt succeeds and the returned string is identical, then we know the original string is already ASCII. I'm open to better ideas! It might be easier to push that logic into SNIHostname.

Personally I think accepting underscores is fine, that fight has been lost and we should be compatible with browsers, and at a cursory glance I don't see any CTS tests for IDN saying they should be rejected, so it should be doable.

Meantime, what Jesse said: okhttp should probably catch and rethrow.

[*] I did agitate to start moving towards IDN 2008 in Android 11, but it didn't get any traction.

15characterlimi · 2020-03-03T20:10:28Z

Android platform engineer here :)

Underscores in domains work in Firefox and Chrome; they should work in Android! We should ask AOSP to fix.

My understanding is that underscores aren't RFC compliant. I'm guessing you'd considered this in your statement, and think that AOSP should favor consistency with common browsers over standards compliance? Personally, I'm more hesitant to allow this than you seem to be; I'd at least not want to rush it, since the fact that Android doesn't support them is not new behaviour; is my understanding correct?

yschimke · 2020-03-03T20:28:09Z

@15characterlimi As you know OkHttp strongly favours browser consistency.

But the focus of the 3 fixes is just to swallow an I|llegalArgumentException that crashes apps, and turn into a IOException that users already deal with. There is no additional work to try to make it work on Android 10 via graylisted methods (wouldn't work in Android 11), it's literally to stop crashing the app if someone posts content with _ in a hostname. It's basically a DOS against Apps in the Google Play store unless we fix it.

The Android or Conscrypt fixes are out of scope of this fix, debate with @swankjesse. For now, we've just reported upstream, your call how to deal with those.

yschimke · 2020-03-03T20:53:02Z

I can also swallow the exception completely, so we just drop back to HTTP/1.1. But otherwise probably still work?

swankjesse · 2020-03-19T02:31:07Z

Fixed in 4.5.0-RC1.

electroidru · 2024-03-02T07:35:31Z

@swankjesse @yschimke @15characterlimi @prbprbprb
As I can see the crash was fixed in 4.5.0. However call to the server where hostname uses _ underscores is still illegal, because I'm still getting Invalid input to toASCII error.
So there's no way to use okhttp and android sdk (okhttp shipped with android) to make network call to the hostname with underscores, am I right?

yschimke · 2024-03-02T07:41:40Z

You would need to swap out platform Conscrypt for an alternative SSL stack.

I can try to reproduce with the latest Conscrypt as a dependency.

Or maybe you can test with bouncy castle?

electroidru · 2024-03-02T08:30:32Z

You would need to swap out platform Conscrypt for an alternative SSL stack.

I can reproduce with the latest Conscrypt as a dependency.

Or maybe test with bouncy castle?

Thanks for the reply. I can see now that exception is related to Conscrypt and found the page with available security providers https://square.github.io/okhttp/security/security_providers/#provider-status
Will look into that.

electroidru · 2024-03-02T19:37:36Z

@yschimke With BouncyCastleJsseProvider I was able to solve the _ underscore issue.

Posted answer to the https://stackoverflow.com/a/78095509/1078641 with the solution.

prbprbprb · 2024-03-04T15:53:10Z

@electroidru I'm not sure that "solve" is exactly the right term there... ;)

I'm sure BC's TLS stack works, but (a) it's pure Java so less performant, (b) less well tested as it's unused by default on Android and (c) I wonder how they're handling IDNs if they never call the platform's IDN.toASCII().

This issue kind of fell through the cracks and back in 2020 and at the time I was inclined to agree with Jesse that we should support underscores, but nowadays even the CAs are reluctant to do that for TLS certificates so we should stick with rejecting them: https://groups.google.com/g/mozilla.dev.security.policy/c/7bYUU5uIPCI

Also since 2020 that SSLSocket implementation is all but retired. However by inspection it looks like the new one cal also still throw an unchecked exception on getSSLParameters(), so we should fix that at least.

Edit: "we" in the above being Conscrypt and AOSP.

electroidru · 2024-03-05T07:57:12Z

I'm not sure that "solve" is exactly the right term there... ;)

@prbprbprb Ok, let's call it case instead of issue.

I'm sure BC's TLS stack works, but (a) it's pure Java so less performant, (b) less well tested as it's unused by default on Android and (c) I wonder how they're handling IDNs if they never call the platform's IDN.toASCII().

There are situations when it's difficult to promote changes for the current architecture. It was already in use and _ underscores in subdomains was part of the architecture.
I was already looking for solutions like to build curl from the source and use ndk for the server call which is probably even more less performant and less tested, so for me @yschimke advise was a really helpful.

Also the server uses *. wildcard ssl certificate and quick check on https://www.ssllabs.com/ssltest/ for the url where subdomain contains underscore showing that all is fine for that url. And stackoverflow answer https://stackoverflow.com/a/2183140 quoted that from RFC point of view underscores in the subdomain are also valid.

I suppose that you are probably looking further to protect people from developing not secured apps for the google play, but probably it is possible to leave some space to allow people to change their SSL context for that kind of situation.

talklittle added the bug Bug in existing code label Mar 3, 2020

yschimke mentioned this issue Mar 3, 2020

Conscrypt fails on _ in domain name google/conscrypt#821

Open

yschimke added the android Relates to usage specifically on Android label Mar 3, 2020

This was referenced Mar 3, 2020

Catch IllegalArgumentException from Android internals #5841

Merged

Catch IllegalArgumentException from Android internals [3.14.x] #5842

Merged

Catch IllegalArgumentException from Android internals [3.12.x] #5843

Merged

yschimke added this to the 4.5 milestone Mar 6, 2020

yschimke mentioned this issue Mar 6, 2020

HttpUrl host validation could be more strict #5488

Closed

swankjesse closed this as completed Mar 19, 2020

Flowdalic mentioned this issue Jun 24, 2020

IllegalArgumentException from minidns-hla 0.3.4 MiniDNS/minidns#104

Open

yschimke mentioned this issue Mar 25, 2022

域名下划线问题 #7182

Closed

This was referenced Jul 5, 2024

[SDK 50] Android development builds crash expo/expo#26605

Closed

Can't start my dev client when the server URL contains an underscore on Android expo/expo#30225

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash on Android 10 with underscore in subdomain #5840

Crash on Android 10 with underscore in subdomain #5840

talklittle commented Mar 3, 2020

yschimke commented Mar 3, 2020 •

edited

Loading

yschimke commented Mar 3, 2020

yschimke commented Mar 3, 2020

swankjesse commented Mar 3, 2020

yschimke commented Mar 3, 2020 •

edited

Loading

prbprbprb commented Mar 3, 2020

15characterlimi commented Mar 3, 2020

yschimke commented Mar 3, 2020 •

edited

Loading

yschimke commented Mar 3, 2020

swankjesse commented Mar 19, 2020

electroidru commented Mar 2, 2024

yschimke commented Mar 2, 2024 •

edited

Loading

electroidru commented Mar 2, 2024

electroidru commented Mar 2, 2024 •

edited

Loading

prbprbprb commented Mar 4, 2024 •

edited

Loading

electroidru commented Mar 5, 2024 •

edited

Loading

Crash on Android 10 with underscore in subdomain #5840

Crash on Android 10 with underscore in subdomain #5840

Comments

talklittle commented Mar 3, 2020

yschimke commented Mar 3, 2020 • edited Loading

yschimke commented Mar 3, 2020

yschimke commented Mar 3, 2020

swankjesse commented Mar 3, 2020

yschimke commented Mar 3, 2020 • edited Loading

prbprbprb commented Mar 3, 2020

15characterlimi commented Mar 3, 2020

yschimke commented Mar 3, 2020 • edited Loading

yschimke commented Mar 3, 2020

swankjesse commented Mar 19, 2020

electroidru commented Mar 2, 2024

yschimke commented Mar 2, 2024 • edited Loading

electroidru commented Mar 2, 2024

electroidru commented Mar 2, 2024 • edited Loading

prbprbprb commented Mar 4, 2024 • edited Loading

electroidru commented Mar 5, 2024 • edited Loading

yschimke commented Mar 3, 2020 •

edited

Loading

yschimke commented Mar 3, 2020 •

edited

Loading

yschimke commented Mar 3, 2020 •

edited

Loading

yschimke commented Mar 2, 2024 •

edited

Loading

electroidru commented Mar 2, 2024 •

edited

Loading

prbprbprb commented Mar 4, 2024 •

edited

Loading

electroidru commented Mar 5, 2024 •

edited

Loading