Stryker fails to connect to vstest on M1 mac

[dmitryvorobey]

Describe the bug
Hi guys! I've tried to install the Stryker and run it from the test's directory but it was unsuccessful.
Error message is "Stryker failed to connect to vstest.console"
Is it a bug or I doing smth wrong?

Logs
Probably it the most useful part of logs

[16:35:39 DBG] Initial build successful
[16:35:39 DBG] VsTest zip was copied to: /var/folders/0d/kw0r8pcx1rn06j4d2g_bmjj00000gn/T/o2gew0gf.rrm/.vstest/vstest.zip
[16:35:39 DBG] VsTest zip was unzipped to: /var/folders/0d/kw0r8pcx1rn06j4d2g_bmjj00000gn/T/o2gew0gf.rrm/.vstest
[16:35:39 DBG] Using vstest from deployed vstest package
[16:35:39 DBG] Using vstest.console: OSX for OS /var/folders/0d/kw0r8pcx1rn06j4d2g_bmjj00000gn/T/o2gew0gf.rrm/.vstest/tools/netcoreapp2.1/vstest.console.dll
[16:35:39 INF] Time Elapsed 00:00:03.4676349
Unhandled exception.

Stryker failed to connect to vstest.console

log-20220727.txt

Expected behavior
Stryker should successful mutate code and run tests

Desktop (please complete the following information):

• OS: [MacOS 12.4 (21F79]
• Type of project [framework, core, xamarin]
• Framework Version [.NET 6]
• Stryker Version [version '2.1.0']

Additional context
Macbook M1

[rouke-broersma]

Could you see if you have any vstest logs in the log directory?

[dmitryvorobey]

I have only 1 log file under the test_folder/Stryker Output/date/logs and you can find it in the description section.
Let me know if I should provide more info.

[rouke-broersma]

Did you run stryker with --log-to-file?

There should be some log files like

log-20220727.txt <- you provided this one                  'Runner 1_VsTest-log.host.22-07-27_18-01-13_11461_4.txt'   'Runner 1_VsTest-log.host.22-07-27_18-01-21_62350_6.txt'
'Runner 0_VsTest-log.host.22-07-27_18-01-17_62973_4.txt'   'Runner 1_VsTest-log.host.22-07-27_18-01-16_40575_10.txt'  'Runner 1_VsTest-log.host.22-07-27_18-01-29_07182_6.txt'
'Runner 0_VsTest-log.host.22-07-27_18-01-18_81922_6.txt'   'Runner 1_VsTest-log.host.22-07-27_18-01-17_51375_4.txt'   'Runner 1_VsTest-log.txt'
'Runner 0_VsTest-log.host.22-07-27_18-01-26_27576_10.txt'  'Runner 1_VsTest-log.host.22-07-27_18-01-18_55957_13.txt'   TestDiscoverer_VsTest-log.host.22-07-27_18-01-11_68905_11.txt
'Runner 0_VsTest-log.host.22-07-27_18-01-27_13977_10.txt'  'Runner 1_VsTest-log.host.22-07-27_18-01-19_61452_11.txt'   TestDiscoverer_VsTest-log.txt
'Runner 0_VsTest-log.txt'                                  'Runner 1_VsTest-log.host.22-07-27_18-01-20_56882_11.txt'

[dmitryvorobey]

Yeap, I ran dotnet stryker --verbosity trace --log-to-file command
It's not a big deal to run it one more time.
There is a new log file in attach
log-20220727.txt
You also mention some vstest logs but as I said I probably have an issue with it. The error says Stryker failed to connect to vstest.console.
It's a screenshot log folder's content

[rouke-broersma]

The library we use to connect to vstest.console should have it's own logfile, I expected that log file to be there containing with some hint as to why connection to vstest.console failed. Seems that it has not managed to even do that.

Unfortunately I do not have a Mac. I think @dupdob has a mac but he has not reported such issues so I doubt they have them. I think it could be that theirs is not an M1 mac. I have a feeling this might be related to M1 macs. In that case we cannot troubleshoot this for you unfortunately.

I think at this point the best course of action is to report this issue over on https://github.com/microsoft/vstest. I could report the issue for you so it has some details on how we integrate with vstest but they will likely ask us to provide some diagnostics or do some tests for us which we cannot do since we cannot currently reproduce this.

Do you think this is something you could assist with after we report the issue with them?

[dmitryvorobey]

Thanks a lot!
Let me know how I can assist and what kind of info I should provide.
Hope this issue will be resolved in the future.

[dupdob]

hi
for info, I use Parallel Desktop for OSS C# dev. Alternatively, I use Rider on Mac for some hobby projects. So I do not use Stryker on Mac OS.
If I have some free time, I will give it a spin to see if everything works as expected.

[dupdob]

I just made a check, ran smoothly on my intel Mac (MBP2020) on Ventura (MacOS 13).
Latest Stryker version (V2.11), tested on Net 5 and Net 6.
So this may be related to M1 Macs

[dupdob]

for info: I added some logs (in my current PR) as an attempt to get some details on why VsTest fails to start.
This will not fix anything, just give us some more info whenever this happens.

[libravado]

I am also getting VsTest console timeout errors, as described. I'm on a HP Z-Book running Windows 10.
Stryker <= 1.5.3 seems ok
Stryker >= 2 all give me the issue

[dupdob]

hi @libravado
you short description hints toward a different problem. It looks like #2069 or #2002. Refactoring lead to a redesign of the timeout calculation, with some regressions as a sad by product.
Can you share the project you are testing ?
Alternatively, can you open a specific issue with some details (incl. verbose log files)?

[rouke-broersma]

@dmitryvorobey I am rolling a new version (2.1.2) where we should get a little more information from the exception that is thrown that could perhaps help us move forward. It should be released within the next hour. If you have time I would appreciate it if you could give it a try and report back with the actual exception.

[rouke-broersma]

@dmitryvorobey any chance you could help us out here?

[ysbakker]

I was unable to reproduce this issue on my M1 macbook pro (2020, 16GB) running macOS 12.4. I ran Stryker succesfully on a few projects, and did not encounter any issues.

Let me know if I can test projects more specific to this issue.

[rouke-broersma]

In that case and since we have had no further reports I will close the issue as not reproducible. Thank you!