Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PAYARA-4116: Properly shutdown server on startup failure #4301

Merged
merged 1 commit into from Nov 7, 2019

Conversation

@pdudits
Copy link
Contributor

pdudits commented Nov 6, 2019

Description

This is a bug fix

Since cca summer 2017 server would hang when startup failure occurred, thread dump would show one or two threads hanging in

  java.lang.Thread.State: TIMED_WAITING
	  at java.lang.Thread.sleep(Thread.java:-1)
	  at com.sun.enterprise.v3.admin.StopServer.doExecute(StopServer.java:74)
	  at com.sun.enterprise.v3.admin.StopDomainCommand.execute(StopDomainCommand.java:96)
	  at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:558)
	  at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:554)
	  at java.security.AccessController.doPrivileged(AccessController.java:-1)
	  at javax.security.auth.Subject.doAs(Subject.java:360)
	  at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:553)
	  at org.glassfish.api.AsyncImpl$1$1.run(AsyncImpl.java:76)

This posed few interesting questions:

  • Why there is no instance of GlassFish available and this command is stuck in endless loop?
  • Why is there two of them?
  • Why AppServerStartup delegates to command, which then delegates to GlassFish which then delegates to AppServerStartup?

The answers to those are:

GlassFish impplementations (that differ by runtime, e. g. server, embedded, micro) register themselves after kernel starts. It used to be an invariant, that kernel never throws an exception, so the registration always happened. But since 1d7999e kernel start may throw, and StopServer is out of luck.

There are multiple paths to AppServerStartup#shutdown. Failure during startup happens to hit more of them.

The mechanism of delegating shutdown to admin command, which happens to delegate back, comes from early design of Glassfish V3, where shutdown sequence was placed directly in the command. But this code very quickly moved next to startup code, and this delegation is superflous.

Therefore these changes:

  • If kernel throws exception, immediately dispose the instance
  • Stop kernel directly rather than delegating to stop command
  • Guarantee thread safety of shutdown

Shutdown may be invoked synchronously as reaction to failed proceed, or asynchronously from RunLevelListener - it's hard to tell what the timing guarantees are, especially after startup completes.
Therefore a state machine in addition to synchronicity of stop now guards, that shutdown sequence in invoked only once.

PayaraExecutorService now may shutdown before it initialized.

Testing

New tests

Testing Performed

Injecting failures via ByteMan to Init and Startup Runlevel services, emulating startup errors common with 5.193; Injecting failures to grizzly server binds to emulate blocked ports. Server now consistenly shuts down.

Test suites executed

  • Payara samples in remote mode

Testing Environment

Zulu JDK 1.8.0_192, Windows 10, Maven 3.6.2

* If kernel throws exception, guarantee disposal of instance
* Stop kernel directly rather than delegating to stop command
* Guarantee thread safety of shutdown

Delegating to stop-domain was relict of early Glassfish 3 where shutdown
sequence was separate from startup.

Shutdown may be invoked synchronously or asynchronously from RunLevelListener,
therefore state machine guards that shutdown sequence in invoked only once.

PayaraExecutorService now may shutdown before it initialized.
@pdudits

This comment has been minimized.

Copy link
Contributor Author

pdudits commented Nov 6, 2019

jenkins test please

@Cousjava Cousjava added this to the 5.194 milestone Nov 7, 2019
@pdudits pdudits merged commit 1babb00 into payara:master Nov 7, 2019
58 checks passed
58 checks passed
Payara Quick Build and Test Quick build and test passed!
Details
security/snyk - api/payara-api/pom.xml (payara-ci) No new issues
Details
security/snyk - api/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/admin/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/admingui/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/ant-tasks/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/appclient/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/batch/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/common/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/concurrent/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/connectors/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/core/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/deployment/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/distributions/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/ejb/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/extras/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/featuresets/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/flashlight/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/grizzly/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/ha/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/installer/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/jdbc/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/jms/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/load-balancer/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/orb/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/osgi-platforms/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/packager/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/payara-appserver-modules/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/persistence/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/registration/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/resources/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/security/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/tests/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/transaction/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/web/pom.xml (payara-ci) No new issues
Details
security/snyk - appserver/webservices/pom.xml (payara-ci) No new issues
Details
security/snyk - copyright/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/admin/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/cluster/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/common/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/core/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/deployment/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/diagnostics/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/distributions/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/flashlight/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/grizzly/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/hk2/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/osgi-platforms/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/packager/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/payara-modules/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/resources-l10n/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/resources/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/security/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/test-utils/pom.xml (payara-ci) No new issues
Details
security/snyk - nucleus/tests/pom.xml (payara-ci) No new issues
Details
security/snyk - pom.xml (payara-ci) No new issues
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.