IDE executed for UI tests crashing on macOS

[PawelLipski]

MacBook Pro, 2019
macOS Ventura 13.0.1
Plugin version: v3.2.2
Target IDE version: 2022.3

$ java -version
openjdk version "17.0.5" 2022-10-18 LTS
OpenJDK Runtime Environment Corretto-17.0.5.8.1 (build 17.0.5+8-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.5.8.1 (build 17.0.5+8-LTS, mixed mode, sharing)

$ j -Ptests=squash uiTest

> Task :verifyPluginConfiguration
[gradle-intellij-plugin :git-machete-intellij-plugin:verifyPluginConfiguration] The following plugin configuration issues were found:
- The 'since-build' property is lower than the target IntelliJ Platform major version: 221 < 223.
- The Java configuration specifies sourceCompatibility=11 but IntelliJ Platform 2022.3 requires sourceCompatibility=17.
See: https://jb.gg/intellij-platform-versions

> Task :uiTest_2022.3
OpenJDK 64-Bit Server VM warning: Archived non-system classes are disabled because the java.system.class.loader property is specified (value = "com.intellij.util.lang.PathClassLoader"). To use archived non-system classes, this property must not be set
CompileCommand: exclude com/intellij/openapi/vfs/impl/FilePartNodeRoot.trieDescend bool exclude = true

com.virtuslab.gitmachete.uitest.UITestSuite STANDARD_OUT
    Installing IntelliJ(2022.3)
    Fetching https://cache-redirector.jetbrains.com/intellij-jbr/jbr-17.0.5-osx-x64-b653.14.tar.gz into /var/folders/mn/7g5_8tfj05bctk7ykhk1xxzr0000gn/T/ide-probe/cache/4dcbb965b2c98168cc64de1d6334675e.gz
    Extracting jar:file:/Users/pawel_pawellipski/.gradle/caches/modules-2/files-2.1/org.virtuslab.ideprobe/driver_2.13/0.50.0/5c6d80674684bd641a2c008d96230b2db564f890/driver_2.13-0.50.0.jar!/ideprobe_2.12-0.50.0.zip from jar into /var/folders/mn/7g5_8tfj05bctk7ykhk1xxzr0000gn/T/ide-probe/cache/b9a21b46f8cdfd73f8b0f088ce36235d.zip
    Fetching https://packages.jetbrains.team/maven/p/ij/intellij-dependencies/com/intellij/remoterobot/robot-server-plugin/0.11.15/robot-server-plugin-0.11.15.zip into /var/folders/mn/7g5_8tfj05bctk7ykhk1xxzr0000gn/T/ide-probe/cache/85cd505faee53bded778bb86be87ed21.zip
    Installed Plugin(/Users/pawel_pawellipski/git-machete-intellij-plugin/build/distributions/git-machete-intellij-plugin-3.2.2-14-SNAPSHOT+git.0e5d4cc.zip)
    Installed Plugin(ideprobe_2.12-0.50.0)
    Installed Plugin(https://packages.jetbrains.team/maven/p/ij/intellij-dependencies/com/intellij/remoterobot/robot-server-plugin/0.11.15/robot-server-plugin-0.11.15.zip)
    Display mode: Native
    Starting process /var/folders/mn/7g5_8tfj05bctk7ykhk1xxzr0000gn/T/ide-probe/instances/intellij-instance-2022.3--1TRovOG7Qhy5q7Zf2sjhlQ/Contents/bin/idea in /private/var/folders/mn/7g5_8tfj05bctk7ykhk1xxzr0000gn/T/ide-probe/workspaces/ideprobe-workspace-ofDD51lSTUuwnGN7LzfHJg/ws
    [intellij-stderr] 2022-12-05 15:06:12.525 idea[36986:3034433] allVms required (null)
    [intellij-stderr] 2022-12-05 15:06:12.532 idea[36986:3034433] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '*** -[NSTaggedPointerString rangeOfString:options:range:locale:]: nil argument'
    [intellij-stderr] *** First throw call stack:
    [intellij-stderr] (
    [intellij-stderr]   0   CoreFoundation                      0x00007ff800ff743b __exceptionPreprocess + 242
    [intellij-stderr]   1   libobjc.A.dylib                     0x00007ff800b46e25 objc_exception_throw + 48
    [intellij-stderr]   2   Foundation                          0x00007ff801d7cf45 -[NSString rangeOfString:options:range:locale:] + 580
    [intellij-stderr]   3   Foundation                          0x00007ff801d7ccf2 -[NSString rangeOfString:options:range:] + 29
    [intellij-stderr]   4   Foundation                          0x00007ff801da494f -[NSString hasPrefix:] + 60
    [intellij-stderr]   5   idea                                0x000000010acf085a validationJavaVersion + 3418
    [intellij-stderr]   6   idea                                0x000000010acede23 main + 99
    [intellij-stderr]   7   dyld                                0x00007ff800b73310 start + 2432
    [intellij-stderr] )
    [intellij-stderr] libc++abi: terminating with uncaught exception of type NSException
    [intellij-stderr] /var/folders/mn/7g5_8tfj05bctk7ykhk1xxzr0000gn/T/ide-probe/instances/intellij-instance-2022.3--1TRovOG7Qhy5q7Zf2sjhlQ/Contents/bin/idea: line 2: 36986 Abort trap: 6           /var/folders/mn/7g5_8tfj05bctk7ykhk1xxzr0000gn/T/ide-probe/instances/intellij-instance-2022.3--1TRovOG7Qhy5q7Zf2sjhlQ/Contents/MacOS/idea "$@"

com.virtuslab.gitmachete.uitest.UITestSuite > classMethod FAILED
    java.net.SocketTimeoutException: Accept timed out
        at java.base/sun.nio.ch.NioSocketImpl.timedAccept(NioSocketImpl.java:708)
        at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:752)
        at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:675)
        at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:641)
        at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:617)
        at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:574)
        at java.base/java.net.ServerSocket.accept(ServerSocket.java:532)
        at org.virtuslab.ideprobe.ide.intellij.InstalledIntelliJ.$anonfun$startIn$1(InstalledIntelliJ.scala:61)
        at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:62)
        at scala.concurrent.package$.blocking(package.scala:124)
        at org.virtuslab.ideprobe.ide.intellij.InstalledIntelliJ.startIn(InstalledIntelliJ.scala:61)
        at org.virtuslab.ideprobe.IntelliJFixture.startIntelliJ(IntelliJFixture.scala:96)
        at org.virtuslab.ideprobe.RunningIntelliJPerSuiteBase.setup(ClassFixtures.scala:31)
        at org.virtuslab.ideprobe.RunningIntelliJPerSuiteBase.setup$(ClassFixtures.scala:26)
        at com.virtuslab.gitmachete.uitest.UITestSuite$.org$virtuslab$ideprobe$junit4$RunningIntelliJPerSuite$$super$setup(UITestSuite.scala:18)
        at org.virtuslab.ideprobe.junit4.RunningIntelliJPerSuite.setup(ClassFixtures.scala:10)
        at org.virtuslab.ideprobe.junit4.RunningIntelliJPerSuite.setup$(ClassFixtures.scala:10)
        at com.virtuslab.gitmachete.uitest.UITestSuite$.setup(UITestSuite.scala:18)
        at com.virtuslab.gitmachete.uitest.UITestSuite.setup(UITestSuite.scala)

1 test completed, 1 failed

> Task :uiTest_2022.3 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':uiTest_2022.3'.
> There were failing tests. See the report at: file:///Users/pawel_pawellipski/git-machete-intellij-plugin/build/reports/tests/uiTest_2022.3/index.html

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 4m 29s
177 actionable tasks: 24 executed, 153 up-to-date

[PawelLipski]

Doesn't happen with 2022.2.4 though.

[PawelLipski]

This is low-level enough that I won't be surprised if it only happens e.g. on older Intel-powered Macs (including my own) but not on M1... @mkondratek could you check on your M1 and/or Intel? I'll check on my M1 later in the day

[PawelLipski]

Same on M1 Pro :/ so probably an issue with IDEA 2022.3 on Mac in UI tests... might be an issue with ide-probe 🤔

[mkondratek]

I can check intel today and m1 on Friday

[mkondratek]

I can confirm the same for my macOS with intel

[PawelLipski]

Okay, the workaround is as follows:

So force using the official resolver, and .dmg extension.

Proper solution TBD, but according to @tpasternak, instead of using the launcher, ide-probe would need to run java using options from product-info.json: both a list of jars, and command line options

[PawelLipski]

Since #1814, running UI tests against EAPs (but not EAP-SNAPSHOTs) is supported on Mac OS 🎉