Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[iOS] App crash on websocket close if app was in background for long-term #84499

Closed
ycherniavskyi opened this issue Jun 12, 2021 · 14 comments
Closed
Labels
a: release Challenges faced when attempting to productionize an app c: crash Stack traces logged to the console dependency: dart:io Issue in 'dart:io' library dependency: dart Dart team may need to help us platform-ios iOS applications specifically r: fixed Issue is closed as already fixed in a newer version

Comments

@ycherniavskyi
Copy link
Contributor

ycherniavskyi commented Jun 12, 2021

Steps to Reproduce

  1. Run flutter create wscrash
  2. Replace main.dart with next code:
import 'dart:async';
import 'dart:io';

import 'package:flutter/material.dart';

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter WebSocket Crash',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: MyHomePage(),
    );
  }
}

class MyHomePage extends StatefulWidget {
  MyHomePage({Key? key}) : super(key: key);

  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('WebSocket Crash'),
      ),
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            Text(
              'After "Connect to WebSocket" move app to background, then wait 3 min and open. Boom!',
            ),
            TextButton(
              onPressed: () async {
                print('>>> wscrash connecting');
                WebSocket ws =
                    await WebSocket.connect('ws://echo.websocket.org');
                print('>>> wscrash connected');
                Timer(Duration(seconds: 30), () async {
                  print('>>> wscrash closing');
                  await ws.close();
                  print('>>> wscrash closed');
                });
              },
              child: Text('Connet to WebSocket'),
            ),
          ],
        ),
      ),
    );
  }
}
  1. Run the app in release mode flutter run --release
  2. Press Connect to WebSocket button
  3. Move the app to background
  4. Wait for 3-5 min (period depend from some internal iOS logic)
  5. Tap on the app icon to reopen it

Expected results: the app is open then timer handler triggered and close WebSocket without error

Actual results: the app is open then timer handler triggered but on close WebSocket it crashed

Logs
$ flutter run --release
Multiple devices found:
iphone11 (mobile) • 11111111-2222222222222222            • ios            • iOS 14.6
iPhone 12 (mobile)   • 33333333-4444-5555-6666-777777777777 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-5 (simulator)
Chrome (web)         • chrome                               • web-javascript • Google Chrome 91.0.4472.101
[1]: iphone11 (11111111-2222222222222222)
[2]: iPhone 12 (33333333-4444-5555-6666-777777777777)
[3]: Chrome (chrome)
Please choose one (To quit, press "q/Q"): 1
Launching lib/main.dart on iphone11 in release mode...
Found saved certificate choice "Apple Development: ych@gmail.com (XXX)". To clear, use "flutter config".
Signing iOS app for device deployment using developer identity: "Apple Development: ych@gmail.com (XXX)"
Running Xcode build...
 └─Compiling, linking and signing...                        13.6s
Xcode build done.                                           48.7s
Installing and launching...                                        14.0s

Flutter run key commands.
h Repeat this help message.
c Clear the screen
q Quit (terminate the application on the device).

Application finished.
$ flutter analyze
Analyzing wscrash...
No issues found! (ran in 2.0s)
$ flutter doctor -v
[✓] Flutter (Channel stable, 2.2.1, on macOS 11.4 20F71 darwin-x64, locale en-UA)
    • Flutter version 2.2.1 at /usr/local/Caskroom/flutter/1.22.4/flutter
    • Framework revision 02c026b03c (2 weeks ago), 2021-05-27 12:24:44 -0700
    • Engine revision 0fdb562ac8
    • Dart version 2.13.1

[!] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
    • Android SDK at /Users/ych/Library/Android/sdk
    • Platform android-30, build-tools 28.0.3
    • ANDROID_HOME = /Users/ych/Library/Android/sdk
    • Java binary at: /Users/ych/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/202.7351085/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)
    ✗ Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/macos#android-setup for more details.

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.5, Build version 12E262
    • CocoaPods version 1.10.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.2)
    • Android Studio at /Users/ych/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/202.7351085/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)

[✓] IntelliJ IDEA Ultimate Edition (version 2021.1.2)
    • IntelliJ at /Users/ych/Applications/JetBrains Toolbox/IntelliJ IDEA Ultimate.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.56.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.23.0

[✓] Connected device (2 available)
    • iPhone 12 (mobile) • 8DEBEAD6-B758-4696-A656-498B2E5245D3 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-5 (simulator)
    • Chrome (web)       • chrome                               • web-javascript • Google Chrome 91.0.4472.101

! Doctor found issues in 1 category.

Some lines from app Console log

...
default	20:51:12.987155+0300	Runner	flutter: >>> wscrash closing
default	20:51:12.990753+0300	SpringBoard	[application<com.example.wscrash>:2147] Workspace connection invalidated.
default	20:51:12.991018+0300	SpringBoard	[application<com.example.wscrash>:2147] Now flagged as pending exit for reason: workspace client connection invalidated
default	20:51:12.993191+0300	SpringBoard	Application process state changed for com.example.wscrash: <SBApplicationProcessState: 0x28255cca0; pid: 2147; taskState: Suspended; visibility: Foreground>
...
default	20:51:13.012008+0300	mediaserverd	-CMSessionMgr- CMSessionMgrHandleApplicationStateChange: Client com.example.wscrash with pid '2147' is now Terminated. Background entitlement: NO ActiveLongFormVideoSession: NO WhitelistedLongFormVideoApp NO
default	20:51:13.016213+0300	SpringBoard	[com.example.wscrash] com.example.wscrash application state changed to ForegroundRunning
default	20:51:13.016408+0300	SpringBoard	[com.example.wscrash] Ignore becoming foreground for application without push registration
default	20:51:13.016702+0300	SpringBoard	[com.example.wscrash] com.example.wscrash application state changed to Terminated
default	20:51:13.016963+0300	SpringBoard	[com.example.wscrash] Ignore becoming background for application without push registration
default	20:51:13.022229+0300	runningboardd	[application<com.example.wscrash>:2147] termination reported by launchd (2, 13, 13)
default	20:51:13.022264+0300	runningboardd	Removing process: [application<com.example.wscrash>:2147]
default	20:51:13.022476+0300	runningboardd	Removing launch job for: [application<com.example.wscrash>:2147]
default	20:51:13.022644+0300	runningboardd	Removed job for [application<com.example.wscrash>:2147]
default	20:51:13.022700+0300	runningboardd	Removing assertions for terminated process: [application<com.example.wscrash>:2147]
default	20:51:13.022766+0300	runningboardd	Removed last relative-start-date-defining assertion for process application<com.example.wscrash>
default	20:51:13.032129+0300	runningboardd	Calculated state for application<com.example.wscrash>: none (role: None)
default	20:51:13.032272+0300	runningboardd	Calculated state for application<com.example.wscrash>: none (role: None)
default	20:51:13.032545+0300	SpringBoard	[application<com.example.wscrash>:2147] Process exited: <RBSProcessExitContext| specific, status:<RBSProcessExitStatus| domain:signal(2) code:SIGPIPE(13)>>.
default	20:51:13.032710+0300	SpringBoard	[application<com.example.wscrash>:2147] Setting process task state to: Not Running
default	20:51:13.032776+0300	SpringBoard	[application<com.example.wscrash>:2147] Setting process visibility to: Unknown
default	20:51:13.033128+0300	SpringBoard	[application<com.example.wscrash>:2147] Invalidating workspace.
...

As a temporary solution works suggestion from #47203 (comment) (which based on section Handle or disable SIGPIPE from Apple Avoiding Common Networking Mistakes).
Also this #71979 seems related.

@ycherniavskyi
Copy link
Contributor Author

The minimal workaround is adding signal(SIGPIPE, SIG_IGN) to application function call in AppDelegate.swift:

import UIKit
import Flutter

@UIApplicationMain
@objc class AppDelegate: FlutterAppDelegate {
  override func application(
    _ application: UIApplication,
    didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?
  ) -> Bool {
    signal(SIGPIPE, SIG_IGN);
    GeneratedPluginRegistrant.register(with: self)
    return super.application(application, didFinishLaunchingWithOptions: launchOptions)
  }
}

@mraleph
Copy link
Member

mraleph commented Jun 14, 2021

/cc @aam - we should make sure that we suppress SIGPIPE on sockets.

@darshankawar darshankawar added the in triage Presently being triaged by the triage team label Jun 14, 2021
@darshankawar
Copy link
Member

I was unable to replicate after keeping app in background for around ~3 to 5 and ~8 to 10 mins. Labeling it based on above comment.

@darshankawar darshankawar added a: release Challenges faced when attempting to productionize an app dependency: dart Dart team may need to help us platform-ios iOS applications specifically c: crash Stack traces logged to the console passed first triage and removed in triage Presently being triaged by the triage team labels Jun 14, 2021
@ycherniavskyi
Copy link
Contributor Author

@darshankawar did you see the next line in Console app on macOS (when iPhone connected via cable)?

default	13:47:45.511503+0300	runningboardd	[application<com.example.wscrash>:4107] Shutdown sockets (ALL)

Also, I found how to speed up the crash reproduction - lock the screen of the iPhone (sometimes several times).
And I need to mention that this issue reproduces on a real device. I am not sure about the simulator.

Here is how it looks like:
https://user-images.githubusercontent.com/552025/121885042-5a23b500-cd1c-11eb-8acf-b86e4817d79f.mp4

@aam
Copy link
Member

aam commented Jun 14, 2021

/cc @aam - we should make sure that we suppress SIGPIPE on sockets.

We already do that on a server side of a socket https://github.com/dart-lang/sdk/blob/master/runtime/bin/socket_macos.cc#L149, but not on a client side.

@a-siva a-siva added the dependency: dart:io Issue in 'dart:io' library label Jun 14, 2021
@aam
Copy link
Member

aam commented Jun 14, 2021

Two things needs clarification:

  • we already ignore SIGPIPE on macos/ios(as well as linux) https://github.com/dart-lang/sdk/blob/master/runtime/bin/platform_macos.cc#L48. It's not clear why we still see SIGPIPE affecting process execution.
  • if I remove code that ignores SIGPIPE(from platform_macos/platform_linux) I expect to see failures in some test. I can't seem to write a test(pair of client/server apps) that would start crashing unexpectedly if other side closes socket pipe. But I don't see crashes, everything works as before.

Of course, ios might be special(than mac/linux), but that is puzzling.

@mraleph
Copy link
Member

mraleph commented Jun 14, 2021

@aam I'd guess that Platform::Initialize is not actually called from engine code.

@aam
Copy link
Member

aam commented Jun 14, 2021

ah, true @mraleph , it only seems to be invoked from standalone embedder/tester/gen_snapshotter.

Second question remains - it would be nice to understand how to repro this on mac/linux.

@ycherniavskyi
Copy link
Contributor Author

@aam as for removing code that ignores SIGPIPE and absent crashed, maybe it is because of setsockopt with SO_NOSIGPIPE in Socket::CreateBindDatagram and ServerSocket::CreateBindListen.

@aam
Copy link
Member

aam commented Jun 15, 2021

sorry @ycherniavskyi on macos for my tests I removed that code too. On linux we don't do this setsockopt call.
Also, that code only affects server side of the socket, so even if that code is in place client should fail(terminate) if the server drops connection(like you see happening on ios).
I think dart code responsible for writing into socket(socket_patch.dart) normally keeps track of socket state and normally prevents attempts to write into dropped closed socket. The way how paused app on iOS gets resumed though probably throws wrench into that state tracking, so such write-after-close does sneak through before the socket gets notified it was closed.

@ycherniavskyi
Copy link
Contributor Author

@aam to be precise I want to mention that in my case the app crashed on close, but I just check add and the behavior is the same.

() async {
  print('>>> wscrash add');
  ws.add('test');
  print('>>> wscrash delayed');
  await Future<void>.delayed(Duration(seconds: 3));
  print('>>> wscrash closing');
  await ws.close();
  print('>>> wscrash closed');
}

App output default 10:17:56.918102+0300 Runner flutter: >>> wscrash add in Console and then crashed.

@aam
Copy link
Member

aam commented Jun 15, 2021

Yes @ycherniavskyi , similar to ws.add(...) ws.close() results in closing data sequence being sent out, which can't go through if socket pipe was closed https://github.com/dart-lang/sdk/blob/master/sdk/lib/_http/websocket_impl.dart#L743

dart-bot pushed a commit to dart-lang/sdk that referenced this issue Jun 16, 2021
…nnect code.

Fixes flutter/flutter#84499

TEST=socket_sigpipe_test.dart

Change-Id: I220558e74b41c1969efa422254867c11dd17ee91
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/203660
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
@aam
Copy link
Member

aam commented Jun 17, 2021

The fix rolled into flutter master in e57d97a

@aam aam closed this as completed Jun 17, 2021
@darshankawar darshankawar added the r: fixed Issue is closed as already fixed in a newer version label Jun 18, 2021
dart-bot pushed a commit to dart-lang/sdk that referenced this issue Jun 24, 2021
…nnect code.

Fixes flutter/flutter#84499

TEST=socket_sigpipe_test.dart

Change-Id: I220558e74b41c1969efa422254867c11dd17ee91
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/203660
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
@github-actions
Copy link

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 31, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
a: release Challenges faced when attempting to productionize an app c: crash Stack traces logged to the console dependency: dart:io Issue in 'dart:io' library dependency: dart Dart team may need to help us platform-ios iOS applications specifically r: fixed Issue is closed as already fixed in a newer version
Projects
None yet
Development

No branches or pull requests

5 participants