Add one liner documentation how to actually use CI mode #240

janknips · 2022-07-29T12:28:46Z

I can't figure out how to use it, because there used to be a flag for it, which is now gone, and the entire project contains no mention of the word CI anymore :D

welcome · 2022-07-29T12:28:48Z

Hi! Thanks for opening your first issue here! 😄

fzyzcjy · 2022-07-29T12:29:33Z

You are right, I forgot to update the doc...

fzyzcjy · 2022-07-29T12:32:02Z

This is how I personally use it:

dart run convenient_test_manager_dart # in one shell
flutter run path/to/your/integration_test/main_test.dart --host-vmservice-port 9753 --disable-service-auth-codes --dart-define CONVENIENT_TEST_APP_CODE_DIR=sth # in another shell, run your worker app

janknips · 2022-07-29T12:36:21Z

But will it terminate like that? Seems like this is what I'm already doing locally 😁 So for CI I would need a mode that terminates after all tests ran

fzyzcjy · 2022-07-29T23:07:31Z

Yes it terminates automatically after all tests run.

janknips · 2022-08-01T07:50:54Z

This is how I personally use it:

dart run convenient_test_manager_dart # in one shell
flutter run path/to/your/integration_test/main_test.dart --host-vmservice-port 9753 --disable-service-auth-codes --dart-define CONVENIENT_TEST_APP_CODE_DIR=sth # in another shell, run your worker app

Could not find file convenient_test_manager 😛 Which directory are you running in / which build command did you run before this?

fzyzcjy · 2022-08-01T07:55:12Z

Could not find file convenient_test_manager

https://github.com/fzyzcjy/flutter_convenient_test/tree/master/packages/convenient_test_manager_dart

janknips · 2022-08-01T13:41:56Z

Thanks, got it to run. Unfortunately, it seems quite unreliable, e.g. worker disconnects, etc, at least I never got my tests to pass, which usually easily pass.

Maybe it would be good to add another barebone CI mode that will just detect wether flutter test was used, and in that case, just work as if only integration_test was used, without using the manager. What do you think? At least in my use case I can imagine just calling this in CI, since it's fast, and failed tests could still be re-run locally afterwards.

Logs:

2022-08-01T13:40:56.402721Z|info|VmServiceWrapperService|Connecting to vm service at ws://127.0.0.1:9753/ws. Please ensure your Flutter app has port=9753
2022-08-01T13:40:56.404170Z|warn|VmServiceWrapperService|init failed e=SocketException: Connection refused (OS Error: Connection refused, errno = 61), address = 127.0.0.1, port = 63525 s=#0      _NativeSocket.startConnect (dart:io-patch/socket_patch.dart:682:35)[NL]#1      _RawSocket.startConnect (dart:io-patch/socket_patch.dart:1817:26)[NL]#2      RawSocket.startConnect (dart:io-patch/socket_patch.dart:27:23)[NL]#3      Socket._startConnect (dart:io-patch/socket_patch.dart:2038:22)[NL]#4      Socket.startConnect (dart:io/socket.dart:759:21)[NL]#5      _ConnectionTarget.connect (dart:_http/http_impl.dart:2453:20)[NL]#6      _HttpClient._getConnection.connect (dart:_http/http_impl.dart:2867:12)[NL]#7      _HttpClient._getConnection (dart:_http/http_impl.dart:2872:12)[NL]#8      _HttpClient.
_openUrl (dart:_http/http_impl.dart:2727:12)[NL]#9      _HttpClient.openUrl (dart:_http/http_impl.dart:2591:7)[NL]#10     _WebSocketImpl.connect (dart:_http/websocket_impl.dart:1021:42)[NL]#11     WebSocket.connect (dart:_http/websocket.dart:360:22)[NL]#12     vmServiceConnectUri (package:vm_service/vm_service_io.dart:32:44)[NL]#13     VmServiceWrapperService.connect (package:convenient_test_manager_dart/services/vm_service_wrapper_service.dart:30:31)[NL]#14     _awaitWorkerAvailable (file:///Users/jan/StudioProjects/flutter_convenient_test/packages/convenient_test_manager_dart/bin/convenient_test_manager_dart.dart:63:37)[NL]<asynchronous suspension>[NL]#15     main (file:///Users/jan/StudioProjects/flutter_convenient_test/packages/convenient_test_manager_dart/bin/convenient_test_manager_d
art.dart:25:3)[NL]<asynchronous suspension>[NL]
2022-08-01T13:40:57.387018Z|debug|ConvenientTestManagerService|getWorkerCurrentRunConfig ans=integrationTest: {[NL]  reportSuiteInfo: true[NL]  executionFilter: {[NL]    filterNameRegex: match-nothing^$[NL]    strategy: {[NL]      allMatch: {[NL]      }[NL]    }[NL]  }[NL]  defaultRetryCount: 1[NL]  autoUpdateGoldenFiles: false[NL]}[NL] currSuperRunController=seenTearDownAll: false[NL]    
2022-08-01T13:40:57.650498Z|debug|ReportHandlerService|SetUpAll 
2022-08-01T13:40:57.675875Z|debug|VideoRecorderStore|startRecord call ScreenVideoRecorderService begin
2022-08-01T13:40:57.678795Z|info|ScreenVideoRecorderServiceIosSimulator|startRecord begin /tmp/ConvenientTest/RUN-20220801-033838-263/Video/20220801_034057.mov
2022-08-01T13:40:57.689800Z|debug|ReportHandlerService|handleReportSuiteInfo called groupId: 1[NL]groups: {[NL]  id: 570286508[NL]  name: Sign in tests:[NL]  parentId: 1[NL]  entryIds: 34833574[NL]  entryIds: 210372701[NL]}[NL]groups: {[NL]  id: 1[NL]  name: [NL]  parentId: -1[NL]  entryIds: 570286508[NL]}[NL]tests: {[NL]  id: 34833574[NL]  name: Sign in tests: can sign in as feedFollower[NL]  parentId: 570286508[NL]}[NL]tests: {[NL]  id: 210372701[NL]  name: Sign in tests: can sign in as contributor[NL]  parentId: 570286508[NL]}[NL]
2022-08-01T13:40:57.690148Z|debug|ReportHandlerService|handleReportSuiteInfo thus clearAll
2022-08-01T13:40:57.690500Z|debug|MiscDartService|clearAll
2022-08-01T13:40:57.693762Z|debug|ReportHandlerService|handleReportSuiteInfo thus clear
2022-08-01T13:40:57.694380Z|debug|ReportSaverService|clear
2022-08-01T13:40:57.719333Z|debug|ReportHandlerService|StateChange: testName=(tearDownAll) state=status: running[NL]result: success[NL]
2022-08-01T13:40:57.732418Z|debug|ReportHandlerService|Message: Warning: integration_test plugin was not detected.
2022-08-01T13:40:57.732796Z|debug|ReportHandlerService|Message: 
2022-08-01T13:40:57.733438Z|debug|ReportHandlerService|Message: If you're running the tests with `flutter drive`, please make sure your tests
2022-08-01T13:40:57.734213Z|debug|ReportHandlerService|Message: are in the `integration_test/` directory of your package and use
2022-08-01T13:40:57.734957Z|debug|ReportHandlerService|Message: `flutter test $path_to_test` to run it instead.
2022-08-01T13:40:57.736262Z|debug|ReportHandlerService|Message: 
2022-08-01T13:40:57.738320Z|debug|ReportHandlerService|Message: If you're running the tests with Android instrumentation or XCTest, this means
2022-08-01T13:40:57.741175Z|debug|ReportHandlerService|Message: that you are not capturing test results properly! See the following link for
2022-08-01T13:40:57.742487Z|debug|ReportHandlerService|Message: how to set up the integration_test plugin:
2022-08-01T13:40:57.743804Z|debug|ReportHandlerService|Message: 
2022-08-01T13:40:57.745787Z|debug|ReportHandlerService|Message: https://flutter.dev/docs/testing/integration-tests#testing-on-firebase-test-lab
2022-08-01T13:40:57.747829Z|debug|ReportHandlerService|Message: 
2022-08-01T13:40:57.752263Z|debug|ReportHandlerService|TearDownAll resolvedExecutionFilter: {[NL]}[NL]
2022-08-01T13:40:57.753437Z|info|VideoRecorderStore|stopRecord skip since recordingVideoInfo==null
2022-08-01T13:40:57.795430Z|debug|VideoRecorderStore|startRecord call ScreenVideoRecorderService end
2022-08-01T13:40:57.802530Z|debug|ReportHandlerService|StateChange: testName=(tearDownAll) state=status: complete[NL]result: success[NL]
2022-08-01T13:40:58.223250Z|debug|ScreenVideoRecorderServiceIosSimulator|[STDERR] Recording started[NL]
2022-08-01T13:40:59.406865Z|info|main|waitWorkerAvailable check
2022-08-01T13:40:59.407203Z|info|VmServiceWrapperService|Connecting to vm service at ws://127.0.0.1:9753/ws. Please ensure your Flutter app has port=9753
2022-08-01T13:40:59.464867Z|info|_ServiceConnectionManager|vmServiceOpened
2022-08-01T13:40:59.501701Z|info|_ServiceConnectionManager|handleServiceEvent kind=ServiceRegistered service=reloadSources method=s0.reloadSources
2022-08-01T13:40:59.503067Z|info|_ServiceConnectionManager|handleServiceEvent kind=ServiceRegistered service=hotRestart method=s0.hotRestart
2022-08-01T13:40:59.503176Z|info|_ServiceConnectionManager|handleServiceEvent kind=ServiceRegistered service=flutterVersion method=s0.flutterVersion
2022-08-01T13:40:59.503272Z|info|_ServiceConnectionManager|handleServiceEvent kind=ServiceRegistered service=compileExpression method=s0.compileExpression
2022-08-01T13:40:59.503369Z|info|_ServiceConnectionManager|handleServiceEvent kind=ServiceRegistered service=flutterMemoryInfo method=s0.flutterMemoryInfo
2022-08-01T13:40:59.503465Z|info|_ServiceConnectionManager|handleServiceEvent kind=ServiceRegistered service=flutterGetSkSL method=s0.flutterGetSkSL
2022-08-01T13:41:02.576583Z|info|main|waitWorkerAvailable check
2022-08-01T13:41:02.578734Z|info|main|monitorWorkerAvailable check
2022-08-01T13:41:02.578871Z|info|main|step extra sleep to avoid too quickly hot-restart worker
2022-08-01T13:41:07.582168Z|info|main|monitorWorkerAvailable check
2022-08-01T13:41:08.581594Z|info|main|step reloadInfo
2022-08-01T13:41:08.583016Z|info|VmServiceWrapperService|hotRestartThrottled triggered
2022-08-01T13:41:08.584786Z|debug|SingleRunningExecutor|call runner() reason= triggerTime=2022-08-01 15:41:08.583506 arg=null
2022-08-01T13:41:08.587864Z|info|_ServiceConnectionManager|hotRestart start
2022-08-01T13:41:08.589244Z|info|main|awaitSuiteInfoNonEmpty check numGroupEntries=4
2022-08-01T13:41:08.590753Z|info|main|step hotRestartAndRunTests
2022-08-01T13:41:08.591026Z|debug|MiscDartService|hotRestartAndRunTests filterNameRegex=.*
2022-08-01T13:41:08.591200Z|info|VmServiceWrapperService|hotRestartThrottled triggered
2022-08-01T13:41:08.591934Z|info|main|step awaitSuperRunStatusTestAllDone
2022-08-01T13:41:09.798020Z|info|_ServiceConnectionManager|hotRestart end resp={type: Success}
2022-08-01T13:41:09.798168Z|debug|VmServiceWrapperService|hotRestartThrottledExecutor deliberately extra wait
2022-08-01T13:41:10.014816Z|debug|ConvenientTestManagerService|getWorkerCurrentRunConfig ans=integrationTest: {[NL]  reportSuiteInfo: true[NL]  executionFilter: {[NL]    filterNameRegex: .*[NL]    strategy: {[NL]      allMatch: {[NL]      }[NL]    }[NL]  }[NL]  defaultRetryCount: 1[NL]  autoUpdateGoldenFiles: false[NL]}[NL] currSuperRunController=seenTearDownAll: false[NL]    
2022-08-01T13:41:10.233986Z|debug|ReportHandlerService|SetUpAll 
2022-08-01T13:41:10.235581Z|debug|VideoRecorderStore|startRecord call ScreenVideoRecorderService begin
2022-08-01T13:41:10.235676Z|info|ScreenVideoRecorderServiceIosSimulator|startRecord begin /tmp/ConvenientTest/RUN-20220801-034108-587/Video/20220801_034110.mov
2022-08-01T13:41:10.236413Z|info|ScreenVideoRecorderServiceIosSimulator|stopRecord begin
2022-08-01T13:41:10.236476Z|info|ScreenVideoRecorderServiceIosSimulator|stopRecord send signals
2022-08-01T13:41:10.237278Z|info|ScreenVideoRecorderServiceIosSimulator|stopRecord await exitCode
2022-08-01T13:41:10.238181Z|debug|ReportHandlerService|handleReportSuiteInfo called groupId: 1[NL]groups: {[NL]  id: 570286508[NL]  name: Sign in tests:[NL]  parentId: 1[NL]  entryIds: 34833574[NL]  entryIds: 210372701[NL]}[NL]groups: {[NL]  id: 1[NL]  name: [NL]  parentId: -1[NL]  entryIds: 570286508[NL]}[NL]tests: {[NL]  id: 34833574[NL]  name: Sign in tests: can sign in as feedFollower[NL]  parentId: 570286508[NL]}[NL]tests: {[NL]  id: 210372701[NL]  name: Sign in tests: can sign in as contributor[NL]  parentId: 570286508[NL]}[NL]
2022-08-01T13:41:10.238306Z|debug|ReportHandlerService|handleReportSuiteInfo thus clearAll
2022-08-01T13:41:10.238353Z|debug|MiscDartService|clearAll
2022-08-01T13:41:10.238468Z|debug|ReportHandlerService|handleReportSuiteInfo thus clear
2022-08-01T13:41:10.238517Z|debug|ReportSaverService|clear
2022-08-01T13:41:10.240339Z|debug|ReportHandlerService|StateChange: testName=Sign in tests: can sign in as feedFollower state=status: running[NL]result: success[NL]
2022-08-01T13:41:10.380292Z|debug|ReportHandlerService|Message: 2022-08-01T13:41:10.257496Z|info|LogHandle|🟤 START  null null
2022-08-01T13:41:10.383371Z|debug|ReportHandlerService|handleReportLogEntry called
🚀 [STATUS] [0:02] [1x pending, 1x running] [Test=Sign in tests: can sign in as feedFollower]
2022-08-01T13:41:10.657689Z|debug|ReportHandlerService|Message: 2022-08-01T13:41:10.599764Z|info|LogHandle|🔵 START APP  null null
2022-08-01T13:41:10.663036Z|debug|ReportHandlerService|handleReportLogEntry called
2022-08-01T13:41:11.729195Z|info|_ServiceConnectionManager|VMService.onDone called
2022-08-01T13:41:11.869360Z|debug|ScreenVideoRecorderServiceIosSimulator|[STDERR] Wrote video to: /tmp/ConvenientTest/RUN-20220801-033838-263/Video/20220801_034057.mov[NL]
2022-08-01T13:41:11.877720Z|info|ScreenVideoRecorderServiceIosSimulator|stopRecord exitCode=0
2022-08-01T13:41:11.886680Z|debug|VideoRecorderStore|startRecord call ScreenVideoRecorderService end
2022-08-01T13:41:12.239680Z|debug|ScreenVideoRecorderServiceIosSimulator|[STDERR] Recording started[NL]
2022-08-01T13:41:12.583738Z|info|main|monitorWorkerAvailable check
2022-08-01T13:41:12.584165Z|error|main|monitorWorkerAvailable see hot restart not available, thus exit with code=2 (vmServiceWrapperService.hotRestartAvailable=false, vmServiceWrapperService.connected=false)

and the runner:

flutter: 00:00 +0: Sign in tests: can sign in as feedFollower
flutter: 2022-08-01T13:41:10.257496Z|info|LogHandle|🟤 START  null null
flutter: 2022-08-01T13:41:10.599764Z|info|LogHandle|🔵 START APP  null null
Lost connection to device.

fzyzcjy · 2022-08-01T13:44:12Z

Lost connection to device.

Strongly suspect it is because: #188. See that issue for a lot of details.

It is discussed in: flutter/flutter#37818

And it is fixed: flutter/engine#34496, but not landed in stable yet.

janknips · 2022-08-01T13:46:11Z

Lost connection to device.

Strongly suspect it is because: #188. See that issue for a lot of details.

It is discussed in: flutter/flutter#37818

And it is fixed: flutter/engine#34496, but not landed in stable.

I see, thanks 👍 I will try again with master branch and let you know. Let's hope for a either hotfix or fast next update 😅

fzyzcjy · 2022-08-01T13:46:17Z

Unfortunately, it seems quite unreliable, e.g. worker disconnects, etc, at least I never got my tests to pass, which usually easily pass.

If the only problem is "lost connection to device" maybe it is #188 problem; if many other problems feel free to create separate issues

fzyzcjy · 2022-08-01T13:46:42Z

I see, thanks 👍 I will try again with master branch and let you know. Let's hope for a either hotfix or fast next update 😅

Yes personally I also suffer from this problem :( So eagerly looking forward to next stable!

janknips · 2022-08-01T13:47:26Z

Unfortunately, it seems quite unreliable, e.g. worker disconnects, etc, at least I never got my tests to pass, which usually easily pass.

If the only problem is "lost connection to device" maybe it is #188 problem; if many other problems feel free to create separate issues

Thanks, yeah for now I can't really tell if anything else is broken 😄 will let you know if I find any other issues!

fzyzcjy · 2022-08-01T13:47:28Z

Maybe it would be good to add another barebone CI mode that will just detect wether flutter test was used, and in that case, just work as if only integration_test was used, without using the manager. What do you think? At least in my use case I can imagine just calling this in CI, since it's fast, and failed tests could still be re-run locally afterwards.

Some features will be lost (e.g. "isolation mode", "all screenshots and videos", etc). But I guess it is implement-able. Feel free to PR!

fzyzcjy · 2022-08-01T13:48:28Z

Thanks, yeah for now I can't really tell if anything else is broken 😄 will let you know if I find any other issues!

Sure, take your time! I personally feel it stable except for that problem; for my own case it is sometimes also not stable because I did e2e tests with a backend and docker etc which is source of unstableness - but that is surely not your case and not related to this package

stale · 2022-09-30T15:42:16Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions · 2022-10-22T12:04:55Z

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new issue.

fzyzcjy mentioned this issue Aug 4, 2022

Add option to disable splash screen fade on FlutterViewController flutter/flutter#37818

Closed

vHanda mentioned this issue Aug 26, 2022

CI: Build conventient_test_manager_dart on Linux #265

Merged

stale bot added the wontfix This will not be worked on label Sep 30, 2022

stale bot closed this as completed Oct 8, 2022

github-actions bot locked as resolved and limited conversation to collaborators Oct 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add one liner documentation how to actually use CI mode #240

Add one liner documentation how to actually use CI mode #240

janknips commented Jul 29, 2022

welcome bot commented Jul 29, 2022

fzyzcjy commented Jul 29, 2022

fzyzcjy commented Jul 29, 2022

janknips commented Jul 29, 2022

fzyzcjy commented Jul 29, 2022

janknips commented Aug 1, 2022

fzyzcjy commented Aug 1, 2022

janknips commented Aug 1, 2022 •

edited

Loading

fzyzcjy commented Aug 1, 2022 •

edited

Loading

janknips commented Aug 1, 2022

fzyzcjy commented Aug 1, 2022

fzyzcjy commented Aug 1, 2022

janknips commented Aug 1, 2022

fzyzcjy commented Aug 1, 2022

fzyzcjy commented Aug 1, 2022

stale bot commented Sep 30, 2022

github-actions bot commented Oct 22, 2022

Add one liner documentation how to actually use CI mode #240

Add one liner documentation how to actually use CI mode #240

Comments

janknips commented Jul 29, 2022

welcome bot commented Jul 29, 2022

fzyzcjy commented Jul 29, 2022

fzyzcjy commented Jul 29, 2022

janknips commented Jul 29, 2022

fzyzcjy commented Jul 29, 2022

janknips commented Aug 1, 2022

fzyzcjy commented Aug 1, 2022

janknips commented Aug 1, 2022 • edited Loading

fzyzcjy commented Aug 1, 2022 • edited Loading

janknips commented Aug 1, 2022

fzyzcjy commented Aug 1, 2022

fzyzcjy commented Aug 1, 2022

janknips commented Aug 1, 2022

fzyzcjy commented Aug 1, 2022

fzyzcjy commented Aug 1, 2022

stale bot commented Sep 30, 2022

github-actions bot commented Oct 22, 2022

janknips commented Aug 1, 2022 •

edited

Loading

fzyzcjy commented Aug 1, 2022 •

edited

Loading