Migrate all users of new Flutter SDKs to the new SDK DAPs

[DanTup]

Currently ~10% of users on Flutter 3.10 should be getting the new DAP (need to check stats). We should increase this as long as it's not generating lots of crashes.

@christopherfujino are you seeing (m)any crashes on stable since 3.10 released?

[christopherfujino]

There are some. Let me collate the stack traces and numbers for you tomorrow.

[christopherfujino]

Number 1, with 36 affected clients:

RPCError: setLibraryDebuggable: (112) Service has disappeared
at new _OutstandingRequest(vm_service.dart:1822)
at VmService._call(vm_service.dart:2366)
at VmService.setLibraryDebuggable(vm_service.dart:2280)
at IsolateManager._sendLibraryDebuggables.<anonymous closure>(isolate_manager.dart:830)
at <asynchronous gap>(async)
at Future.wait.<anonymous closure>(future.dart:525)
at <asynchronous gap>(async)
at IsolateManager._sendLibraryDebuggables(isolate_manager.dart:818)
at <asynchronous gap>(async)
at Future.wait.<anonymous closure>(future.dart:525)
at <asynchronous gap>(async)
at IsolateManager._configureIsolate(isolate_manager.dart:393)
at <asynchronous gap>(async)
at IsolateManager.registerIsolate(isolate_manager.dart:222)
at <asynchronous gap>(async)
at IsolateManager.handleEvent(isolate_manager.dart:165)
at <asynchronous gap>(async)
at DartDebugAdapter.handleIsolateEvent(dart.dart:2109)
at <asynchronous gap>(async)
at DartDebugAdapter._withErrorHandling(dart.dart:2417)
at <asynchronous gap>(async)

Number 2, with only 5 affected clients:

RPCError: resume: (-32000) Bad state: The client closed with pending request "resume".
at new _OutstandingRequest(vm_service.dart:1822)
at VmService._call(vm_service.dart:2366)
at VmService.resume(vm_service.dart:2241)
at IsolateManager.resumeThread(isolate_manager.dart:288)
at IsolateManager._handlePause(isolate_manager.dart:502)
at IsolateManager.handleEvent(isolate_manager.dart:178)
at <asynchronous gap>(async)
at DartDebugAdapter.handleDebugEvent(dart.dart:2076)
at <asynchronous gap>(async)
at DartDebugAdapter._withErrorHandling(dart.dart:2417)
at <asynchronous gap>(async)

Number 3, with 8 affected clients (and unfortunately no stacktrace):

RPCError: lookupResolvedPackageUris: (-32000) Bad state: The client closed with pending request "lookupResolvedPackageUris".

Number 4, with 3 affected clients, almost identical to number 1:

RPCError: getIsolate: (112) Service has disappeared
at new _OutstandingRequest(vm_service.dart:1822)
at VmService._call(vm_service.dart:2366)
at VmService.getIsolate(vm_service.dart:2090)
at IsolateManager._sendLibraryDebuggables(isolate_manager.dart:803)
at IsolateManager._configureIsolate(isolate_manager.dart:394)
at IsolateManager.registerIsolate(isolate_manager.dart:222)
at IsolateManager.handleEvent(isolate_manager.dart:165)
at DartDebugAdapter.handleIsolateEvent(dart.dart:2109)
at <asynchronous gap>(async)
at DartDebugAdapter._withErrorHandling(dart.dart:2417)
at <asynchronous gap>(async)

[christopherfujino]

Overall we are under our crash rate threshold, but I worry increasing usage of the debug-adapter 10x would take us over.

[christopherfujino]

This release is using package:dds version 2.7.10: https://github.com/flutter/flutter/blob/3.10.0/packages/flutter_tools/pubspec.yaml#L13

[DanTup]

@christopherfujino this is useful, thanks!

The "Service has disappeared" issues look like flutter/flutter#116235 which is fixed for the next DDS. The others look like variations of the same thing (which is, requests failing during shutdown that we're not handling).

I suspect most (if not all) of these crashes aren't actually visible to the user, but I don't want to flood the error reports even if that's the case. I'll leave everything as-is for now (10%) and we can see how it goes.

I'll try and get fixes in DDS for the remaining issues for future releases. Do you have access to these stats only for stable/beta, or master too?

[christopherfujino]

We do have crash reports for master, however, it's difficult to get useful signal, because:

1. there isn't a whole lot of usage
2. there's a lot of maintainers messing with their checkouts on it
3. the valid reports we DO get are spread out across many different versions (since they will be on whatever commit happened to be tip of tree last time they pulled)

[DanTup]

Sure, mostly I'm just interested in "are there any crashes from flutter debug_adapter" though. Having none doesn't indicate no issues, but having some may give me a head-start on fixing things without waiting another release cycle.

I'll look at the other issues note above first though, then we can figure out next steps.

I'm not going to change the current settings from 10%, but do keep me posted if you see any new crashes (or are concerned about numbers). Like say, I suspect these are mostly invisible so users might not actually be impacted, although if you see any reports that suggest otherwise, do let me know too (because the fixes I'm making in some cases are just assuming this is during shutdown and we can ignore some kinds of errors from DDS/VM Service).

[christopherfujino]

Sure, mostly I'm just interested in "are there any crashes from flutter debug_adapter" though. Having none doesn't indicate no issues, but having some may give me a head-start on fixing things without waiting another release cycle.

I guess I'm saying clicking through every report on master is a lot more work for me and I'd rather wait for a beta release.

[DanTup]

Ah, gotcha :)

Np, I'll work on what we know anyway and we can continue to monitor stable.

[DanTup]

So, I reviewed all the call stacks above and actually believe all of them are resolved by the most recent change. All of the errors are either "Service has disappeared" or "The client closed with pending request" and the code changes will discard either of those types of errors now.

@christopherfujino if you've seen any others on stable, please let me know. If not, I might ask @bkonyi for a DDS release soon and then we can monitor next beta.

[christopherfujino]

So crashers I'm seeing on the latest beta/3.11.0-0.1.pre (with dds: 2.7.10:

Number 1 (this looks almost the same as number 1 in #4543 (comment))

RPCError: getIsolate: (112) Service has disappeared
at new _OutstandingRequest(vm_service.dart:1865)
at VmService._call(vm_service.dart:2417)
at VmService.getIsolate(vm_service.dart:2133)
at IsolateManager._sendLibraryDebuggables(isolate_manager.dart:803)
at IsolateManager._configureIsolate(isolate_manager.dart:394)
at IsolateManager._handlePause(isolate_manager.dart:490)
at IsolateManager.handleEvent(isolate_manager.dart:178)
at <asynchronous gap>(async)
at DartDebugAdapter.handleDebugEvent(dart.dart:2076)
at <asynchronous gap>(async)
at DartDebugAdapter._withErrorHandling(dart.dart:2417)
at <asynchronous gap>(async)

Number 2 (this exactly is number 1 above):

RPCError: setLibraryDebuggable: (112) Service has disappeared
at new _OutstandingRequest(vm_service.dart:1865)
at VmService._call(vm_service.dart:2417)
at VmService.setLibraryDebuggable(vm_service.dart:2331)
at IsolateManager._sendLibraryDebuggables.<anonymous closure>(isolate_manager.dart:830)
at <asynchronous gap>(async)
at Future.wait.<anonymous closure>(future.dart:525)
at <asynchronous gap>(async)
at IsolateManager._sendLibraryDebuggables(isolate_manager.dart:818)
at <asynchronous gap>(async)
at Future.wait.<anonymous closure>(future.dart:525)
at <asynchronous gap>(async)
at IsolateManager._configureIsolate(isolate_manager.dart:393)
at <asynchronous gap>(async)
at IsolateManager.registerIsolate(isolate_manager.dart:222)
at <asynchronous gap>(async)
at IsolateManager.handleEvent(isolate_manager.dart:165)
at <asynchronous gap>(async)
at DartDebugAdapter.handleIsolateEvent(dart.dart:2109)
at <asynchronous gap>(async)
at DartDebugAdapter._withErrorHandling(dart.dart:2417)
at <asynchronous gap>(async)

Number 3 (this one might be new?):

RPCError: resume: (-32603) resume: WipError -32000 Can only perform operation while paused.
at new _OutstandingRequest(vm_service.dart:1865)
at VmService._call(vm_service.dart:2417)
at VmService.resume(vm_service.dart:2292)
at IsolateManager.resumeThread(isolate_manager.dart:288)
at IsolateManager._handlePause(isolate_manager.dart:502)
at IsolateManager.handleEvent(isolate_manager.dart:178)
at <asynchronous gap>(async)
at DartDebugAdapter._configureExistingIsolates.<anonymous closure>(dart.dart:759)
at <asynchronous gap>(async)
at Future.wait.<anonymous closure>(future.dart:525)
at <asynchronous gap>(async)
at DartDebugAdapter._configureExistingIsolates(dart.dart:744)
at <asynchronous gap>(async)
at DartDebugAdapter._withErrorHandling(dart.dart:2417)
at <asynchronous gap>(async)
at DartDebugAdapter.connectDebugger(dart.dart:692)
at <asynchronous gap>(async)
at FlutterDebugAdapter._connectDebugger(flutter_adapter.dart:366)
at <asynchronous gap>(async)
at FlutterDebugAdapter._handleDebugPort(flutter_adapter.dart:447)
at <asynchronous gap>(async)

Number 4 (this is number 3 from #4543 (comment))

RPCError: lookupResolvedPackageUris: (-32000) Bad state: The client closed with pending request "lookupResolvedPackageUris".

Number 5 (again similar):

RPCError: getVM: (112) Service has disappeared
at new _OutstandingRequest(vm_service.dart:1865)
at VmService._call(vm_service.dart:2417)
at VmService.getVM(vm_service.dart:2223)
at DartDebugAdapter.connectDebugger(dart.dart:686)
at <asynchronous gap>(async)
at FlutterDebugAdapter._connectDebugger(flutter_adapter.dart:366)
at <asynchronous gap>(async)
at FlutterDebugAdapter._handleDebugPort(flutter_adapter.dart:447)
at <asynchronous gap>(async)

Number 6 (again similar):

RPCError: getIsolate: (112) Service has disappeared
at new _OutstandingRequest(vm_service.dart:1865)
at VmService._call(vm_service.dart:2417)
at VmService.getIsolate(vm_service.dart:2133)
at MappedIterator.moveNext(iterable.dart:393)
at Future.wait(future.dart:523)
at DartDebugAdapter._configureExistingIsolates(dart.dart:739)
at DartDebugAdapter.connectDebugger.<anonymous closure>(dart.dart:693)
at DartDebugAdapter._withErrorHandling(dart.dart:2417)
at DartDebugAdapter.connectDebugger(dart.dart:692)
at <asynchronous gap>(async)
at FlutterDebugAdapter._connectDebugger(flutter_adapter.dart:366)
at <asynchronous gap>(async)
at FlutterDebugAdapter._handleDebugPort(flutter_adapter.dart:447)
at <asynchronous gap>(async)

Not sure how many of these should be fixed in in the latest DDS release, but thought I would list them all.