Server sometimes can stuck on "Reloading ResourceManager"

[molor]

Expected behavior

The server is starting.

Observed/Actual behavior

The server is not starting.

Steps/models to reproduce

This is not the first time I have encountered this. I do not know how to reproduce this. This happens randomly, possibly depending on the phase of the moon :/

It looks like the server thread is waiting for something or another important thread is stuck on something, but I do not know how and where to look at the stack trace. Right now the server is in this state and still running, so if you can tell me which program can be used to see what its process is doing, I will send you the necessary information to fix this problem.

This is always looks like this:

And that's all. Time in now 13:20, CPU load in the task manager is ~1%, the server took up 2 MiB of RAM and just does nothing until it is restarted..

Listening for transport dt_socket at address: 1023 on the last line: I tried to connect to server via Eclipse Debug, but later disconnected because it did not give me any info :c

Plugin list

null

Paper version

I encountered this problem back on versions 1.15 and 1.14, but it happened rarely and I did not think that it would happen again in new versions. Maybe this has already been fixed in 1.17, but I think not.

{"currentVersion":"git-Paper-777 (MC: 1.16.5)"}

Agreements

• I am running the latest version of Paper available from https://papermc.io/downloads.
• I have searched for and ensured there isn't already an open issue regarding this.
• My version of Minecraft is supported by Paper.

Other

No response

[molor]

Startup command:

java -Xms2G -Xmx2G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=200 -XX:+UnlockExperimentalVMOptions -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=40 -XX:G1HeapRegionSize=8M -XX:G1ReservePercent=20 -XX:G1HeapWastePercent=5 -XX:G1MixedGCCountTarget=4 -XX:InitiatingHeapOccupancyPercent=15 -XX:G1MixedGCLiveThresholdPercent=90 -XX:G1RSetUpdatingPauseTimePercent=5 -XX:SurvivorRatio=32 -XX:+PerfDisableSharedMem -XX:MaxTenuringThreshold=1 -Dfile.encoding="utf-8" -Xdebug -Xrunjdwp:transport=dt_socket,address=1023,server=y,suspend=n -jar "server.jar" nogui

[Proximyst]

Run jstack $pid where $pid is the appropriate number from jps and paste the contents into a code block or paste.gg, please.

[molor]

I think that this is bad

[molor]

I caught it

https://paste.gg/p/anonymous/191efa7e74234538a3177f65fe761223

[molor]

Same on 1.17.1-133 (eclipse's remote debug connection stack trace, because jps & jstack again reports that jvm.dll not loaded by target process)

Java HotSpot(TM) 64-Bit Server VM [localhost:1023] (Suspended)	
	Thread [main] (Suspended)	
		Unsafe.park(boolean, long) line: not available [native method]	
		LockSupport.park(Object) line: 211	
		CompletableFuture$Signaller.block() line: 1860	
		ForkJoinPool.managedBlock(ForkJoinPool$ManagedBlocker) line: 3137	
		CompletableFuture<T>.waitingGet(boolean) line: 1894	
		CompletableFuture<T>.get() line: 2068	
		Main.main(OptionSet) line: 192	
		Main.main(String[]) line: 293	
	Daemon Thread [Log4j2-TF-1-AsyncLogger[AsyncContext@2a139a55]-1] (Suspended)	
		Unsafe.park(boolean, long) line: not available [native method]	
		LockSupport.parkNanos(Object, long) line: 252	
		AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) line: 1661	
		TimeoutBlockingWaitStrategy.waitFor(long, Sequence, Sequence, SequenceBarrier) line: 38	
		ProcessingSequenceBarrier.waitFor(long) line: 56	
		BatchEventProcessor<T>.processEvents() line: 159	
		BatchEventProcessor<T>.run() line: 125	
		Log4jThread(Thread).run() line: 831	
	Daemon Thread [WindowsStreamPump] (Suspended)	
		Kernel32.WaitForSingleObject(long, int) line: not available [native method]	
		WindowsSupport.readConsoleInput(int, int) line: 97	
		JansiWinSysTerminal.processConsoleInput() line: 112	
		JansiWinSysTerminal(AbstractWindowsTerminal).pump() line: 458	
		0x0000000800d0b730.run() line: not available	
		Thread.run() line: 831	
	Daemon Thread [Timer hack thread] (Suspended)	
		Thread.sleep(long) line: not available [native method]	
		SystemUtils$4.run() line: 581	
	Daemon Thread [Worker-Main-1] (Suspended)	
		Unsafe.park(boolean, long) line: not available [native method]	
		LockSupport.park() line: 341	
		AbstractQueuedSynchronizer$ConditionNode.block() line: 505	
		ForkJoinPool.managedBlock(ForkJoinPool$ManagedBlocker) line: 3137	
		AbstractQueuedSynchronizer$ConditionObject.await() line: 1614	
		LinkedBlockingQueue<E>.take() line: 435	
		ThreadPoolExecutor.getTask() line: 1056	
		ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1116	
		ThreadPoolExecutor$Worker.run() line: 630	
		ServerWorkerThread(Thread).run() line: 831	
	Daemon Thread [Worker-Main-2] (Suspended)	
		Unsafe.park(boolean, long) line: not available [native method]	
		LockSupport.park() line: 341	
		AbstractQueuedSynchronizer$ConditionNode.block() line: 505	
		ForkJoinPool.managedBlock(ForkJoinPool$ManagedBlocker) line: 3137	
		AbstractQueuedSynchronizer$ConditionObject.await() line: 1614	
		LinkedBlockingQueue<E>.take() line: 435	
		ThreadPoolExecutor.getTask() line: 1056	
		ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1116	
		ThreadPoolExecutor$Worker.run() line: 630	
		ServerWorkerThread(Thread).run() line: 831	
	Daemon System Thread [Reference Handler] (Suspended)	
		Reference<T>.waitForReferencePendingList() line: not available [native method]	
		Reference<T>.processPendingReferences() line: 243	
		Reference$ReferenceHandler.run() line: 215	
	Daemon System Thread [Finalizer] (Suspended)	
		waiting for: ReferenceQueue$Lock  (id=27)	
		Object.wait(long) line: not available [native method]	
		ReferenceQueue<T>.remove(long) line: 155	
		ReferenceQueue<T>.remove() line: 176	
		Finalizer$FinalizerThread.run() line: 171	
	Daemon System Thread [Signal Dispatcher] (Suspended)	
	Daemon System Thread [Attach Listener] (Suspended)	
	Daemon System Thread [Notification Thread] (Suspended)	
	Daemon System Thread [Common-Cleaner] (Suspended)	
		waiting for: ReferenceQueue$Lock  (id=28)	
		Object.wait(long) line: not available [native method]	
		ReferenceQueue<T>.remove(long) line: 155	
		CleanerImpl.run() line: 140	
		InnocuousThread(Thread).run() line: 831	
		InnocuousThread.run() line: 134	

[stefvanschie]

Occurs on Paper 1.17.1 build 189 as well. I didn't have any kind of remote debug running. The output from jstack and my start script can be found here: https://paste.gg/p/anonymous/f5d4cd939b144d25bfa6239dde3a5e36.

[stefvanschie]

Also occurs on Paper 1.17.1 build 196: https://paste.gg/p/anonymous/7cc2e07776a043c699ede3487857ccae.

[molor]

Why we can't fix this?

[electronicboy]

Well, do you know what causes it?

…

On Mon, 23 Aug 2021, 08:43 Zhenya, ***@***.***> wrote: Why we can't fix this? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6020 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJMAZCRPOBTBF5WE4GH2YDT6H3ZTANCNFSM47NUZMWQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[Puremin0rez]

I've had this occur today during a normal nightly restart, I did a thread dump as well as a heap dump in case anything useful can be figured out from doing a more extensive dive into the state of the server during the deadlock.

Thread dump: https://pastebin.com/raw/8MGREYk3

Heap dump: https://app.treasure.cloud/share/public/f17a6053ab071f20c85a058f229e4e02003f1ea4cd1b957b454897e785a70a61
(190 MB - Link Password: zOUAkmYyhp65xqwmHDQtuC3Efon4gh1/xTXBvZeN3Fbwaqg5)