Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VM crash or System.exit called on modulepath-tests [HZ-4652] #26304

Closed
AleksPeychev opened this issue Apr 6, 2024 · 4 comments
Closed

VM crash or System.exit called on modulepath-tests [HZ-4652] #26304

AleksPeychev opened this issue Apr 6, 2024 · 4 comments
Labels
Source: Internal PR or issue was opened by an employee Team: Integration to-jira Type: Test-Failure
Milestone
Backlog

Comments

@AleksPeychev
Copy link
Contributor

AleksPeychev commented Apr 6, 2024 •
edited
Loading

master (commit 009146c751e0ae6e848bd171a9b8688d6db77c5d)
Failed on :

Stacktrace: 
08:11:09 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.2.5:test (default-test) on project modulepath-tests: 
08:11:09 [ERROR] 
08:11:09 [ERROR] Please refer to /home/jenkins/jenkins_slave/workspace/Hazelcast-master-nightly-little-endian/hazelcast/modulepath-tests/target/surefire-reports for the individual test results.
08:11:09 [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
08:11:09 [ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
08:11:09 [ERROR] Command was /bin/sh -c cd '/home/jenkins/jenkins_slave/workspace/Hazelcast-master-nightly-little-endian/hazelcast/modulepath-tests' && '/usr/share/jdks/oracle-17/bin/java' '-Xms8g' '-Xmx8g' '--add-exports' 'java.base/jdk.internal.ref=ALL-UNNAMED' '--add-opens' 'jdk.management/com.sun.management.internal=ALL-UNNAMED' '--add-opens' 'java.base/sun.nio.ch=ALL-UNNAMED' '--add-opens' 'java.base/java.lang=ALL-UNNAMED' '--add-opens' 'java.management/sun.management=ALL-UNNAMED' '-Dhazelcast.phone.home.enabled=false' '-Dhazelcast.logging.type=none' '-Dhazelcast.test.use.network=false' '-Dlog4j.skipJansi=true' '@/home/jenkins/jenkins_slave/workspace/Hazelcast-master-nightly-little-endian/hazelcast/modulepath-tests/target/surefire/surefireargs-20240406051109443_15800' '/home/jenkins/jenkins_slave/workspace/Hazelcast-master-nightly-little-endian/hazelcast/modulepath-tests/target/surefire' '2024-04-05T22-08-50_810-jvmRun1' 'surefire-20240406051109443_15798tmp' 'surefire_11-20240406051109443_15799tmp'
08:11:09 [ERROR] Error occurred in starting fork, check output in log
08:11:09 [ERROR] Process Exit Code: 1
08:11:09 [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
08:11:09 [ERROR] Command was /bin/sh -c cd '/home/jenkins/jenkins_slave/workspace/Hazelcast-master-nightly-little-endian/hazelcast/modulepath-tests' && '/usr/share/jdks/oracle-17/bin/java' '-Xms8g' '-Xmx8g' '--add-exports' 'java.base/jdk.internal.ref=ALL-UNNAMED' '--add-opens' 'jdk.management/com.sun.management.internal=ALL-UNNAMED' '--add-opens' 'java.base/sun.nio.ch=ALL-UNNAMED' '--add-opens' 'java.base/java.lang=ALL-UNNAMED' '--add-opens' 'java.management/sun.management=ALL-UNNAMED' '-Dhazelcast.phone.home.enabled=false' '-Dhazelcast.logging.type=none' '-Dhazelcast.test.use.network=false' '-Dlog4j.skipJansi=true' '@/home/jenkins/jenkins_slave/workspace/Hazelcast-master-nightly-little-endian/hazelcast/modulepath-tests/target/surefire/surefireargs-20240406051109443_15800' '/home/jenkins/jenkins_slave/workspace/Hazelcast-master-nightly-little-endian/hazelcast/modulepath-tests/target/surefire' '2024-04-05T22-08-50_810-jvmRun1' 'surefire-20240406051109443_15798tmp' 'surefire_11-20240406051109443_15799tmp'
08:11:09 [ERROR] Error occurred in starting fork, check output in log
08:11:09 [ERROR] Process Exit Code: 1
08:11:09 [ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:643)
08:11:09 [ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:285)
08:11:09 [ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250)
08:11:09 [ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1241)
08:11:09 [ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1090)
08:11:09 [ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:910)
08:11:09 [ERROR] 	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:126)
08:11:09 [ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:342)
08:11:09 [ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:330)
08:11:09 [ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
08:11:09 [ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:175)
08:11:09 [ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.access$000(MojoExecutor.java:76)
08:11:09 [ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor$1.run(MojoExecutor.java:163)
08:11:09 [ERROR] 	at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute(DefaultMojosExecutionStrategy.java:39)
08:11:09 [ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:160)
08:11:09 [ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:105)
08:11:09 [ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:73)
08:11:09 [ERROR] 	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:53)
08:11:09 [ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:118)
08:11:09 [ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:261)
08:11:09 [ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:173)
08:11:09 [ERROR] 	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:101)
08:11:09 [ERROR] 	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:910)
08:11:09 [ERROR] 	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:283)
08:11:09 [ERROR] 	at org.apache.maven.cli.MavenCli.main(MavenCli.java:206)
08:11:09 [ERROR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
08:11:09 [ERROR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
08:11:09 [ERROR] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
08:11:09 [ERROR] 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
08:11:09 [ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:283)
08:11:09 [ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:226)
08:11:09 [ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:407)
08:11:09 [ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:348)
08:11:09 [ERROR] -> [Help 1]
08:11:09 [ERROR] 
08:11:09 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
08:11:09 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
08:11:09 [ERROR] 
08:11:09 [ERROR] For more information about the errors and possible solutions, please read the following articles:
08:11:09 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
08:11:09 [ERROR] 
08:11:09 [ERROR] After correcting the problems, you can resume the build with the command
08:11:09 [ERROR]   mvn <args> -rf :modulepath-tests

Standard output can be found here - https://s3.console.aws.amazon.com/s3/buckets/j-artifacts/Hazelcast-master-nightly-little-endian/73/
@AleksPeychev AleksPeychev added this to the 5.4 Backlog milestone Apr 6, 2024
@github-actions github-actions bot changed the title VM crash or System.exit called on modulepath-tests VM crash or System.exit called on modulepath-tests [HZ-4652] Apr 6, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 6, 2024

Internal Jira issue: HZ-4652

@olukas
Copy link
Contributor

olukas commented Apr 8, 2024 •
edited
Loading

Crashed again on:

Console output: 
19:20:44 [INFO]  T E S T S
19:20:44 [INFO] -------------------------------------------------------
19:20:45 [ERROR] Error occurred during initialization of boot layer
19:20:45 [ERROR] java.lang.module.FindException: Unable to derive module descriptor for /home/jenkins/jenkins_slave/workspace/Hazelcast-master-nightly-force-offload/hazelcast/hazelcast/target/hazelcast-5.5.0-SNAPSHOT.jar
19:20:45 [INFO] 
19:20:45 [INFO] Results:
19:20:45 [INFO] 
19:20:45 [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
19:20:45 [INFO] 
19:20:45 [WARNING] Corrupted channel by directly writing to native stream in forked JVM 1. See FAQ web page and the dump file

@AyberkSorgun AyberkSorgun modified the milestones: 5.4 Backlog, Backlog Apr 8, 2024
@JackPGreen
Copy link
Contributor

@AleksPeychev you've put this on the 5.4 backlog, but I think this is only in master (post 5.4 branch creation). Just wanted to check the fix wouldn't need backporting.

@olukas
Copy link
Contributor

olukas commented Apr 12, 2024

@JackPGreen backport is not needed since CP -> EE is not part of 5.4.z branch.

devOpsHazelcast pushed a commit that referenced this issue Apr 12, 2024
@JackPGreen @actions-user
Fix hanging modulepath-tests after CP -> EE migration [HZ-4652][HZ-…
8622fd7 
…4672] (#1352)

When [CP functionality was migrated from OS to
EE](https://github.com/hazelcast/hazelcast-mono/pull/1178), references
to the migrated classes were left in OS `META-INF`.

Because of this, when the `modulepath-tests` were executed, the
following error was logged:
```
Error occurred during initialization of boot layer
java.lang.module.FindException: Unable to derive module descriptor for /home/jenkins/jenkins_slave/workspace/Hazelcast-master-RedHatJDK21-Jack-HZ-4672/hazelcast/hazelcast/target/hazelcast-5.5.0-SNAPSHOT.jar
Caused by: java.lang.module.InvalidModuleDescriptorException: Provider class com.hazelcast.cp.event.impl.CpEventDataSerializerHook not in JAR file hazelcast-5.5.0-SNAPSHOT.jar
```

Can reproduce locally by running the `modulepath-tests`:
```
./mvnw clean test --file hazelcast/modulepath-tests/pom.xml
```

In addition, following this error, Java hangs, leading to:
```
Build timed out (after 300 minutes). Marking the build as failed.
```
This appears to be a behaviour across multiple JDKs that I don't
understand, but is not the root cause of our issue so have not
addressed.

Changes:
- Move references to EE classes from OS to EE `META-INF`
- [Remove Maven `--debug` argument, added to investigate this
issue](https://github.com/hazelcast/hazelcast-mono/pull/1325)

Fixes:
- #26304
- #26306
- [HZ-4652](https://hazelcast.atlassian.net/browse/HZ-4652)
- [HZ-4672](https://hazelcast.atlassian.net/browse/HZ-4672)

[HZ-4652]:
https://hazelcast.atlassian.net/browse/HZ-4652?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

GitOrigin-RevId: 078dd919faf529cb0e9aa9c30f44c0a59ecff930
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Source: Internal PR or issue was opened by an employee Team: Integration to-jira Type: Test-Failure
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
4 participants
@olukas @JackPGreen @AyberkSorgun @AleksPeychev