SecureSocket handshake can stall the main thread for long time

[besthyhy]

In my application, the SDK used before was 1.9 and everything worked fine. After upgrading to 1.12, android still works well, but ios will get stuck on the splash screen.

Detailed questions:

1. Sometimes the first time you start it will get stuck on the splash screen.
2. Manually kill the process and then enter the app, there is a high possibility that it will get stuck on the splash screen page.

Code situation

1. pubspec.yaml
   http: 0.12.0+4
2. lib/SplashPage.dart

class SplashPage extends StatefulWidget {
  @override
  State<StatefulWidget> createState() {
    return _StdPageState();
  }
}

class _StdPageState extends State<SplashPage> {
  @override
  void initState() {
    super.initState();
    
    // send Http here
  }
}

3. lib/util/HttpsUtils.dart

import 'dart:convert' show utf8;
import 'package:flutter/material.dart';
import 'package:http/http.dart' as http;
import 'package:inlan_flutter/Config.dart';
import 'package:inlan_flutter/util/DataUtils.dart';
import 'package:inlan_flutter/util/I18nUtils.dart';
import 'package:inlan_flutter/util/JsonUtils.dart';
import 'package:inlan_flutter/util/LogUtils.dart';
import 'package:inlan_flutter/util/ToastUtils.dart';
import 'package:inlan_flutter/util/UpgradeUtils.dart';
import 'package:inlan_flutter/util/UuidUtils.dart';

class HttpsUtils {
  static void apiPostWithUrl(
      String url,
      Map<String, dynamic> body,
      BuildContext context,
      StackTrace callPositionStackTrace,
      State state,
      callback(json)) async {
    body["version"] = Config.apiVersion;
    body["httpId"] = UuidUtils.v4();
    body["lang"] = I18nUtils.languageCode();
    body["path"] = url;

    String fullUrl = Config.apiUrl + url;

    final apiClient = http.Client();

    try {
      await apiClient
          .post(fullUrl,
              headers: {"Content-Type": "application/json"},
              body: JsonUtils.encode(body))
          .then((response) {
        LogUtils.log(response.statusCode, StackTrace.current,
            tag: "statusCode");

        LogUtils.devLog(Config.apiUrl + url, callPositionStackTrace,
            tag: "apiUrl " + url);

        LogUtils.devLog(JsonUtils.prettyJson(body), callPositionStackTrace,
            tag: "req");

        LogUtils.devLog(
            JsonUtils.prettyStr(response.body), callPositionStackTrace,
            tag: "resp ${response.statusCode}");

        switch (response.statusCode) {
          case 200:
            callback(JsonUtils.parse(response.body));
            break;
          case 400:
            Map<String, dynamic> jsonObj = JsonUtils.parse(response.body);
            state.setState(() {
              ToastUtils.short(jsonObj["message"]);
            });
            break;
          case 401:
            Map<String, dynamic> jsonObj = JsonUtils.parse(response.body);
            state.setState(() {
              ToastUtils.withCallback(jsonObj["message"], () {
                DataUtils.setByService('login_data', '-1');
                DataUtils.setByService('isLogined', '-1');
                Navigator.pushNamedAndRemoveUntil(
                    context, '/account', (Route<dynamic> route) => false);
              });
            });
            break;
          case 403:
            break;
          case 406:
            UpgradeUtils().doForceUpgrade(context);
            break;
          case 422:
            Map<String, dynamic> jsonObj = JsonUtils.parse(response.body);
            state.setState(() {
              ToastUtils.short(jsonObj["message"]);
            });
            break;
        }
      });
    } finally {
      apiClient.close();
    }
  }
}

My temporary solution

  void initState() {
    super.initState();
    Timer timer = new Timer(new Duration(milliseconds: 1200), () {
      // send Http here
    });
    
  }

Expected solution

1. What causes the freeze?
2. A better solution?

[tp]

@besthyhy

We ran into the same issue. For us our the issue was caused by our "User-Agent overwrite" not working anymore with Flutter 1.12, which cause the server to respond with status 400.

Maybe providing some user-agent header (headers: {"User-Agent": "foo"}), would also fix your issue and obliviate the need for the timeout?

[ghostgzt]

me too!
It is due to some https url!

[beiger]

url: https://gank.io/api/v2/categories/Article
method: GET
my location: China

ios ui freeze on http request

[lanistor]

Same issue. It's really a very very very terrible problem.
My environment:
http: v0.12.0+2 ~ 4
Dart: 2.8.0
Flutter: 1.15.3

Relative issues: dart-lang/http#400.

[natebosch]

Does this replicate when you use dart:io directly to make the request?

The comments so far don't give much to investigate - is this because there is a Future that does not resolve?

Does anyone have a minimal reproduction?

[lanistor]

@natebosch In my case, i was using dart:io and package:http/http.dart at the same time in my one dart file, dart:io is used to add platform info to http headers.
After readed your words, i removed import dart:io line, it worked fine suddenly, and then i added import dart:io back, it works fine too, and the error cannot occour ever, even i run flutter clean to clean my build folder and run again. It' really wired.
Now i removed import dart:io just in case. But i really don't know why.

[beiger]

If the website uses lets encrypt certification, the ui will be freeze on https request.
iOS platform
url: https://letsencrypt.org/
method: GET
use dart:io directly

[natebosch]

@vifird - the concern isn't having imports to dart:io in the same file - it is whether the suspected bug is caused by this package, or by dart:io. When you use package:http on mobile with flutter it is acting as a thin wrapper around HttpClient. When you use this package on the web it is a thin wrapper around HttpRequest. Usually when something fundamentally doesn't work it can be reproduced without using this package at all, but by using dart:io or dart:html directly. We ask for reproduction cases using these imports to help us route the bug to the right place. We can't solve any problem in this repository if the bug isn't caused by this repository.

@beiger - Can you get more specific? Is there a Future that should complete but doesn't? Does the entire runtime crash?

use dart:io directly

If you can reproduce without using this package we will need to route the issue to the Dart SDK repo. Do you have a complete and minimal reproduction case?

[beiger]

@natebosch
This is my code:

import 'dart:io';

import 'package:flutter/material.dart';
import 'package:package_info/package_info.dart';

class AboutPage extends StatefulWidget {
 @override
 AboutPageState createState() => AboutPageState();
}

class AboutPageState extends State<AboutPage> {
 String _version = "";
 double _top = -20;

 @override
 Widget build(BuildContext context) {
  return Scaffold(
   appBar: AppBar(
    title: "about",
   ),
   body: Stack(
    children: <Widget>[
     AnimatedPositioned(
      duration: const Duration(milliseconds: 2000),
      curve: Curves.bounceOut,
      left: 0,
      top: _top,
      child: Container(
       width: MediaQuery.of(context).size.width,
       child: Center(
        child: Text(
         _version,
         style: TextStyle(
          fontSize: 28,
          color: Colors.blue,
          fontWeight: FontWeight.bold
         ),
        ),
       ),
      ),
     )
    ],
   ),
  );
 }

 @override
 void initState() {
  super.initState();
  _getVersion();
  test();
 }

 Future<void> test() async {
//  var url="https://gank.io/";  // Letsencrypt CA
  var url='https://github.com/dart-lang/http/issues/374'; // other certificate
  var httpClient = HttpClient();
  var request = await httpClient.getUrl(Uri.parse(url));
  var response = await request.close();
  print(response);
 }

 void _getVersion() {
  Future.delayed(Duration(microseconds: 500), () {
   PackageInfo.fromPlatform().then((value) {
    setState(() {
     _version =value.version;
     _top = MediaQuery.of(context).size.height / 2.7;
    });
   });
  });
 }
}

Letsencrypt CA url:

When you open this page, sometimes the page gets stuck in the middle, sometimes the animation gets stuck.

Other CA url:
The animation on the page is very smooth.

[beiger]

Here is my iOS info:

There is no runtime crash.

[natebosch]

If the website uses lets encrypt certification, the ui will be freeze on https request.

Have you tried using an HttpClient from dart:io and setting the badCertificateCallback?

https://api.dart.dev/stable/2.7.2/dart-io/HttpClient/badCertificateCallback.html

[natebosch]

I think I'll move to the SDK repo for now. It's not entirely clear to me what the issue is, but I don't see any evidence it's related to package:http.

[beiger]

I'm using HttpClient from dart:io. I tried to set the badCertificateCallback, the program did not enter callback.

I've only had this kind of stuck situation since last month, and there are some people who have met earlier than me. It used to run well with the same code several month ago. This is only on IOS, Android is good.

[mraleph]

I don't think there is enough information here for us to diagnose problem.

HTTP processing is done asynchronously so it can't just block your UI thread.

I would start by looking for exceptions in the logs.

[beiger]

Many people have encountered this situation.
Maybe this is the problem of iOS, I don't know.

[CyrilHu]

this

I also have this problem, I tried the same request in Safari, It's also slow,
maybe the slow response is the problem of iOS, it's not a big trouble,
but it is awful of UI freeze.
It's necessary to resolve the UI freeze problem, this is really important.

Server ENV:

• Nginx + HTTPS(Let's Encrypt)

Client ENV:

• Device: iPhone XR , iOS 13.4
• Flutter 1.12.13+hotfix.9 • Tools • Dart 2.7.2

TEST:

My Flutter App:

• http: response in 200ms,
• https: about 6 - 10 seconds and the UI frozen, it freeze about 30 seconds when I send 5 request at the same time

iOS Safari:

• http: response in 200ms,
• https: about 5 - 8 seconds, safari is not freeze

MacOS 10.15.4 Safari (without cache):

• http: response in 200ms,
• https: response in 1 second,

This is the stack trace, I pause debug when the UI frozen

[mraleph]

@CyrilHu could you try running the following code on your iOS device (replace url to point to your https server)

import 'dart:async';
import 'dart:developer' as developer;
import 'dart:io';

void log(String s) {
  print(s);
  developer.log(s);
}

Timer stallDetector() {
  final sw = Stopwatch()..start();
  return Timer.periodic(Duration(milliseconds: 5), (_) {
    if (sw.elapsedMilliseconds > 10) {
      log('EVENT LOOP WAS STALLED FOR ${sw.elapsedMilliseconds} ms');
    }
    sw.reset();
  });
}

void main() async {
  final url = 'https://gank.io';
  final timer = stallDetector();
  var sw = Stopwatch()..start();
  var httpClient = HttpClient();
  try {
    var request = await httpClient.getUrl(Uri.parse(url));
    var response = await request.close();
    log('REQUEST COMPLETE IN ${sw.elapsedMilliseconds} ms');
  } finally {
    httpClient.close(force: true);
    timer.cancel();
  }
}

what does it print back?

[CyrilHu]

@CyrilHu could you try running the following code on your iOS device (replace url to point to your https server)

what does it print back?

https test 1:
flutter: EVENT LOOP WAS STALLED FOR 48 ms
[log] EVENT LOOP WAS STALLED FOR 48 ms
flutter: EVENT LOOP WAS STALLED FOR 24 ms
[log] EVENT LOOP WAS STALLED FOR 24 ms
flutter: EVENT LOOP WAS STALLED FOR 30 ms
[log] EVENT LOOP WAS STALLED FOR 30 ms
[log] EVENT LOOP WAS STALLED FOR 3274 ms
flutter: EVENT LOOP WAS STALLED FOR 3274 ms
[log] EVENT LOOP WAS STALLED FOR 62 ms
flutter: EVENT LOOP WAS STALLED FOR 62 ms
flutter: REQUEST COMPLETE IN 3967 ms
[log] REQUEST COMPLETE IN 3967 ms

https test 2:
flutter: EVENT LOOP WAS STALLED FOR 124 ms
[log] EVENT LOOP WAS STALLED FOR 124 ms
flutter: EVENT LOOP WAS STALLED FOR 29 ms
[log] EVENT LOOP WAS STALLED FOR 29 ms
[log] EVENT LOOP WAS STALLED FOR 34 ms
flutter: EVENT LOOP WAS STALLED FOR 34 ms
flutter: EVENT LOOP WAS STALLED FOR 2744 ms
[log] EVENT LOOP WAS STALLED FOR 2744 ms
[log] EVENT LOOP WAS STALLED FOR 39 ms
flutter: EVENT LOOP WAS STALLED FOR 39 ms
flutter: REQUEST COMPLETE IN 3079 ms
[log] REQUEST COMPLETE IN 3079 ms

http test 1:
flutter: EVENT LOOP WAS STALLED FOR 70 ms
[log] EVENT LOOP WAS STALLED FOR 70 ms
[log] EVENT LOOP WAS STALLED FOR 30 ms
flutter: EVENT LOOP WAS STALLED FOR 30 ms
[log] EVENT LOOP WAS STALLED FOR 90 ms
flutter: EVENT LOOP WAS STALLED FOR 90 ms
flutter: REQUEST COMPLETE IN 245 ms
[log] REQUEST COMPLETE IN 245 ms

http test 2:
flutter: EVENT LOOP WAS STALLED FOR 67 ms
[log] EVENT LOOP WAS STALLED FOR 67 ms
[log] EVENT LOOP WAS STALLED FOR 32 ms
flutter: EVENT LOOP WAS STALLED FOR 32 ms
[log] EVENT LOOP WAS STALLED FOR 68 ms
flutter: EVENT LOOP WAS STALLED FOR 68 ms
[log] REQUEST COMPLETE IN 227 ms
flutter: REQUEST COMPLETE IN 227 ms

[ghostgzt]

@mraleph
Is there a temporary solution? This is a very urgent problem!

[mraleph]

Yeah, this seems pretty bad. Based on my cursory analysis of the code most likely cause of the the problem is the fact that we install a certificate verification callback (via SSL_CTX_set_cert_verify_callback) which uses SecTrustEvaluateWithError.

However SecTrustEvaluateWithError comes with the following warning:

Don’t call SecTrustEvaluateWithError from your app’s main run loop because it might require network access to fetch intermediate certificates, or to perform revocation checking. To perform evaluation asynchronously, use SecTrustEvaluateAsyncWithError instead.

Which seems to match the situation we are experiencing here - I think the stall occurs when SecTrustEvaluateWithError is attempting to check revocation status of the certificate. I suspect that you should be able to speed things up by configuring OCSP stapling on the server side.

It is unclear if there are any work-arounds on the client side.

I am extremely unfamiliar with this code e.g. it is unclear to me why we use SSL_CTX_set_cert_verify_callback to verify the whole certificate chain, rather than using SSL_CTX_set_verify to verify individual certificates in the chain - the latter API supports asynchronous verification, while SSL_CTX_set_cert_verify_callback API is completely synchronous. If we must use SSL_CTX_set_cert_verify_callback - then we have to offload handshaking into a thread pool to make it asynchronous, otherwise we will stall UI thread too much.

@bkonyi @zanderso @dnfield you are the ones changing this code most. Can you take a look at this?

/cc @zichangg @sortie

[mraleph]

@ghostgzt I think the only client side workaround I can offer is to use a separate isolate to handle networking and then send results back into the main isolate. This would prevent UI from stalling - but hand shake would still take very long time.

[bkonyi]

It looks like there's a SecTrustEvaluateAsyncWithError that we might be able to use here instead.

We were using SSL_CTX_set_cert_verify_callback as we needed to be able to access the SSLCertContext associated with the connection from the verification callback but SSL_CTX_set_verify doesn't allow for passing an argument to the callback. Can you point me to where in the documentation it says SSL_CTX_set_cert_verify_callback is synchronous whereas SSL_CTX_set_verify is not? Open/BoringSSL documentation is abysmal, so maybe I just missed that somewhere.

[mraleph]

Can you point me to where in the documentation it says SSL_CTX_set_cert_verify_callback is synchronous whereas SSL_CTX_set_verify is not?

It turns out that I actually meant SSL_CTX_set_custom_verify rather than SSL_CTX_set_verify (this library is amazing 🙄).

// SSL_CTX_set_cert_verify_callback sets a custom callback to be called on
// certificate verification rather than |X509_verify_cert|. |store_ctx| contains
// the verification parameters. The callback should return one on success and
// zero on fatal error. It may use |X509_STORE_CTX_set_error| to set a
// verification result.
//
// The callback may use |SSL_get_ex_data_X509_STORE_CTX_idx| to recover the
// |SSL| object from |store_ctx|.
OPENSSL_EXPORT void SSL_CTX_set_cert_verify_callback(
    SSL_CTX *ctx, int (*callback)(X509_STORE_CTX *store_ctx, void *arg),
    void *arg);

enum ssl_verify_result_t BORINGSSL_ENUM_INT {
  ssl_verify_ok,
  ssl_verify_invalid,
  ssl_verify_retry,
};

// SSL_CTX_set_custom_verify configures certificate verification. |mode| is one
// of the |SSL_VERIFY_*| values defined above. |callback| performs the
// certificate verification.
//
// The callback may call |SSL_get0_peer_certificates| for the certificate chain
// to validate. The callback should return |ssl_verify_ok| if the certificate is
// valid. If the certificate is invalid, the callback should return
// |ssl_verify_invalid| and optionally set |*out_alert| to an alert to send to
// the peer. Some useful alerts include |SSL_AD_CERTIFICATE_EXPIRED|,
// |SSL_AD_CERTIFICATE_REVOKED|, |SSL_AD_UNKNOWN_CA|, |SSL_AD_BAD_CERTIFICATE|,
// |SSL_AD_CERTIFICATE_UNKNOWN|, and |SSL_AD_INTERNAL_ERROR|. See RFC 5246
// section 7.2.2 for their precise meanings. If unspecified,
// |SSL_AD_CERTIFICATE_UNKNOWN| will be sent by default.
//
// To verify a certificate asynchronously, the callback may return
// |ssl_verify_retry|. The handshake will then pause with |SSL_get_error|
// returning |SSL_ERROR_WANT_CERTIFICATE_VERIFY|.
OPENSSL_EXPORT void SSL_CTX_set_custom_verify(
    SSL_CTX *ctx, int mode,
    enum ssl_verify_result_t (*callback)(SSL *ssl, uint8_t *out_alert));

In SSL_CTX_set_cert_verify_callback case callback can return either success or failure, but in SSL_CTX_set_custom_verify it can return ssl_verify_retry which means verification is in progress asynchronously.

We were using SSL_CTX_set_cert_verify_callback as we needed to be able to access the SSLCertContext associated with the connection from the verification callback but SSL_CTX_set_verify doesn't allow for passing an argument to the callback.

Apparently there is an ex_data mechanism that allows you to associate arbitrary data with SSL object, see e.g. SSL_set_ex_data. This should allow us to use SSL_CTX_set_custom_verify.

[vsmenon]

@a-siva @franklinyow - is this a candidate for the Sept milestone?

[franklinyow]

Team need more time for the fix, moved to Oct

[franklinyow]

@a-siva Who's working on this now?

[a-siva]

@aam has picked this up.

[aam]

Per discussion with @rmacnak-google the plan is to leave existing synchronous openssl/boringssl API in place, but have SSLFilter::Handshake() run on dart worker thread using IOService infrastructure that is used for similar synchronous os api calls(file, socket, etc)

[aam]

One problem with running SSLFilter::Handshake on separate worker thread is its inability to invoke dart's onBadCertificate synchronously during handshake.

[zanderso]

One problem with running SSLFilter::Handshake on separate worker thread is its inability to invoke dart's onBadCertificate synchronously during handshake.

SSLCertContext::CertificateCallback() running on an IOService thread can send a message to the isolate telling it to run the onBadCertificate callback, then block to wait for the result.

[aam]

Thanks @zanderso . Yeah, that is a possibility, concern would be clean up of blocked IOService thread in case onBadCertificate isolate exits.

As a temporary solution I thought of having async handshake only in case when there is no onBadCertificate provided.

Yet another possibility is to use ssl_verify_retry return code Slava mentioned above with only SecTrustEvaluate part of CertificateVerificationCallback running on a separate thread(spawned via Dart_NewNativePort). This callback will check if it got a response from that thread for that certificate and if so it will synchronously call onBadCertificate (if provided), then return the response to continue ssl's handshake, otherwise it will send a request to that thread and return ssl_verify_retry to suspend ssl's handshake.

[aam]

https://dart-review.googlesource.com/c/sdk/+/165520 with proposed fix.

[a-siva]

Can we close this issue now that the CL has landed ?

[mraleph]

dart-review.googlesource.com/c/sdk/+/165520 with proposed fix.

The fix landed without tests. Is it possible to add some to prevent regressions in the future?

[aam]

#43674 to track adding a test.

[ghostgzt]

which version fixed this problem?

[aam]

So far this landed as 63852d2073c776d6d849e306b2e36274f2143eb3 in flutter master channel( flutter/flutter@63852d2 ).

…

On Sat, Oct 10, 2020 at 4:41 AM Gentle Kwan ***@***.***> wrote: which version fixed this problem? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#41519 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC5BUOAIFGQ3MSB4DANNBDSKBB55ANCNFSM4MIZKDZA> .

[winterdl]

I've got the same issue, with the above update, I will have to test with what dart version, flutter version and what channel, thank you.

[virskor]

Add OCSP configrations is not solve this problem!
For anyone who is using 1.22.6 but not full prepared migrate into Flutter 2.0 still facing about this problem. My workaround below might help you.
Get the current Flutter version path by using this command.

which flutter

open the path printed.
delete path ./cache

cd ./internal
then you need modify three file to change the flutter verison.
engine: 1d12d82d9cb54876f58044aa52198d53ec841c3d
fuchsia-linux: h-DeV4tgEk6jHzUlsoZri9lklKQJDEAk7UVXOEpNKFMC
fuchsia-mac: FFpTJfmj1EGxY5dJDoGNmZ4dwub3GsxHTJHDtKGmQcMC

now you can still using Flutter 1.22.6 but migrate flutter engine to flutter 1.23.0-18.1pre

If you can not compile the IOS app after mofications, try modify ./ios/PodFile

# Uncomment this line to define a global platform for your project
platform :ios, '10.0' 

and add

 target.build_configurations.each do |config|
    config.build_settings['IPHONEOS_DEPLOYMENT_TARGET'] = '10.0'
end

then run

flutter clean \
&& rm ios/Podfile.lock pubspec.lock \
&& rm -rf ios/Pods ios/Runner.xcworkspace

now you can flutter run --release

[mraleph]

I would strongly advise against just manually rolling engine like that because you might end up some bizarre issues due to ABI skew between Flutter Framework and Flutter engine. Instead you should always just switch to a newer version of Flutter (e.g. if you want to use engine from 1.23.0-18.1pre you should better use Flutter at 1.23.0-18.1pre as well).

[virskor]

Using 1.23.0-18.1pre might cause Android GestureDetector not working. we do not have any choice because migrate into Flutter 2.0 is not possible for us at present. But Flutter team might not know this issue is A BIG TROUBLE for Flutter users in China!