1.20.1 Crash when ConcurrentModificationException

[xuancat0208]

Stack trace

[07:29:52] [Region Scheduler Thread #1/ERROR]: [io.papermc.paper.threadedregions.ThreadedRegionizer] Failed to acquire region ThreadedRegion{state=2,sectionCount=60,sections=[RegionSection{sectionCoordinate=[0, 0],chunkCount=187,chunksBitset=7fff7fff7fff7fff7fff7fff7fff7fff1fff1fff1fff7fff0000000000001fff,nonEmptyNeighbours=4,hash=89858455},RegionSection{sectionCoordinate=[-5, -3],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=1618705301},RegionSection{sectionCoordinate=[5, 2],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=701515795},RegionSection{sectionCoordinate=[1, -3],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=435427958},RegionSection{sectionCoordinate=[-5, -2],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=594893820},RegionSection{sectionCoordinate=[5, 1],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=1,hash=2053066268},RegionSection{sectionCoordinate=[3, -2],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=1882338545},RegionSection{sectionCoordinate=[-1, 1],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=478510324},RegionSection{sectionCoordinate=[1, -2],chunkCount=176,chunksBitset=0000000000000000ffffffffffff0000ffffffffffffffffffffffffffffffff,nonEmptyNeighbours=4,hash=1882510179},RegionSection{sectionCoordinate=[3, 5],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=1963998760},RegionSection{sectionCoordinate=[-3, -1],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=1783280927},RegionSection{sectionCoordinate=[3, 0],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=1,hash=434286807},RegionSection{sectionCoordinate=[4, 3],chunkCount=190,chunksBitset=7fff7fff7fff7fff7f807f807fff7fff7fc07fc47fc07f807fff7fff7fc07fc0,nonEmptyNeighbours=5,hash=1106376853},RegionSection{sectionCoordinate=[-4, -4],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=169426532},RegionSection{sectionCoordinate=[4, 5],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=1096708377},RegionSection{sectionCoordinate=[0, -3],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=1,hash=765936951},RegionSection{sectionCoordinate=[2, 4],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=902559332},RegionSection{sectionCoordinate=[0, 1],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=611416030},RegionSection{sectionCoordinate=[0, -1],chunkCount=240,chunksBitset=7fff7fff7fff7fff7fff7fff7fff7fff7fff7fff7fff7fff7fff7fff7fff7fff,nonEmptyNeighbours=5,hash=169234319},RegionSection{sectionCoordinate=[-3, -2],chunkCount=195,chunksBitset=7fff7fff7fff7fff7fff7fff7fff7fff7fff7fff7fff7fff0000000000007fff,nonEmptyNeighbours=3,hash=1156397832},RegionSection{sectionCoordinate=[5, 4],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=1980324917},RegionSection{sectionCoordinate=[1, 0],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=4,hash=479984710},RegionSection{sectionCoordinate=[3, 1],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=617198189},RegionSection{sectionCoordinate=[-1, -1],chunkCount=184,chunksBitset=ffc0ffc0ffc0ffc0fff0fff0fff0fff0fff0fff0fff0fff0fff0fff0fff0fff0,nonEmptyNeighbours=3,hash=164751495},RegionSection{sectionCoordinate=[3, -3],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=1,hash=1254925878},RegionSection{sectionCoordinate=[2, -1],chunkCount=126,chunksBitset=01ff01ff01ff01ff01ff01ff01ff01ff01ff01ff01ff01ff0000000001ff01ff,nonEmptyNeighbours=3,hash=889578876},RegionSection{sectionCoordinate=[-2, -4],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=1,hash=2138094570},RegionSection{sectionCoordinate=[-2, 0],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=967482325},RegionSection{sectionCoordinate=[4, 4],chunkCount=105,chunksBitset=7fff7fff7fff7fff00007fff7fff7fff00000000000000000000000000000000,nonEmptyNeighbours=3,hash=1983858262},RegionSection{sectionCoordinate=[-4, -1],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=529324070},RegionSection{sectionCoordinate=[2, 3],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=3,hash=421502592},RegionSection{sectionCoordinate=[0, -2],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=4,hash=1872711575},RegionSection{sectionCoordinate=[5, 3],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=3,hash=61585114},RegionSection{sectionCoordinate=[-5, -4],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=1,hash=330618805},RegionSection{sectionCoordinate=[3, 4],chunkCount=70,chunksBitset=ffc0ffc0ffc0ffc00000ffc0ffc0ffc000000000000000000000000000000000,nonEmptyNeighbours=3,hash=262823183},RegionSection{sectionCoordinate=[2, -2],chunkCount=99,chunksBitset=000000000000000001ff01ff01ff000001ff01ff01ff01ff01ff01ff01ff01ff,nonEmptyNeighbours=3,hash=1990125876},RegionSection{sectionCoordinate=[4, 1],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=328237817},RegionSection{sectionCoordinate=[-2, 1],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=1,hash=1297349431},RegionSection{sectionCoordinate=[3, -1],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=705195540},RegionSection{sectionCoordinate=[-4, -2],chunkCount=130,chunksBitset=ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0000000000000ffc0,nonEmptyNeighbours=3,hash=281839031},RegionSection{sectionCoordinate=[2, 2],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=286923219},RegionSection{sectionCoordinate=[-2, -3],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=1405147687},RegionSection{sectionCoordinate=[2, 0],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=101373507},RegionSection{sectionCoordinate=[-2, -1],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=3,hash=33001563},RegionSection{sectionCoordinate=[5, 5],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=1,hash=1423052790},RegionSection{sectionCoordinate=[-5, -1],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=1,hash=338347264},RegionSection{sectionCoordinate=[1, -1],chunkCount=224,chunksBitset=ffffffffffffffffffffffffffffffffffffffffffffffff00000000ffffffff,nonEmptyNeighbours=5,hash=541575654},RegionSection{sectionCoordinate=[-1, 0],chunkCount=46,chunksBitset=01c001c000c000c0ffc0ffc001c001c0000000000000ffc00000000000000000,nonEmptyNeighbours=3,hash=1632798908},RegionSection{sectionCoordinate=[-1, -2],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=195588807},RegionSection{sectionCoordinate=[1, 1],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=1,hash=701066804},RegionSection{sectionCoordinate=[-3, -4],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=349424089},RegionSection{sectionCoordinate=[3, 3],chunkCount=160,chunksBitset=ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0,nonEmptyNeighbours=5,hash=825946651},RegionSection{sectionCoordinate=[3, 2],chunkCount=20,chunksBitset=000000000000000000000000000000000000000000000000ffc0ffc000000000,nonEmptyNeighbours=3,hash=1976825461},RegionSection{sectionCoordinate=[-3, -3],chunkCount=180,chunksBitset=00000000000000007fff7fff7fff7fff7fff7fff7fff7fff7fff7fff7fff7fff,nonEmptyNeighbours=3,hash=1799690704},RegionSection{sectionCoordinate=[-4, -3],chunkCount=120,chunksBitset=0000000000000000ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0ffc0,nonEmptyNeighbours=3,hash=514288492},RegionSection{sectionCoordinate=[2, 5],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=1,hash=2020341897},RegionSection{sectionCoordinate=[-2, -2],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=3,hash=771769961},RegionSection{sectionCoordinate=[2, -3],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=2,hash=1298559157},RegionSection{sectionCoordinate=[2, 1],chunkCount=0,chunksBitset=0000000000000000000000000000000000000000000000000000000000000000,nonEmptyNeighbours=1,hash=308402607},RegionSection{sectionCoordinate=[4, 2],chunkCount=30,chunksBitset=0000000000000000000000000000000000000000000000007fff7fff00000000,nonEmptyNeighbours=3,hash=1370085802}]}
java.util.ConcurrentModificationException: null
	at java.util.ArrayDeque.nonNullElementAt(ArrayDeque.java:270) ~[?:?]
	at java.util.ArrayDeque$DescendingIterator.next(ArrayDeque.java:748) ~[?:?]
	at io.papermc.paper.threadedregions.RegionizedTaskQueue$PrioritisedQueue.mergeInto(RegionizedTaskQueue.java:334) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at io.papermc.paper.threadedregions.RegionizedTaskQueue$PrioritisedQueue.mergeInto(RegionizedTaskQueue.java:316) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at io.papermc.paper.threadedregions.RegionizedTaskQueue$RegionTaskQueueData.mergeInto(RegionizedTaskQueue.java:229) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at io.papermc.paper.threadedregions.TickRegions$TickRegionData.mergeInto(TickRegions.java:295) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at io.papermc.paper.threadedregions.ThreadedRegionizer$ThreadedRegion.mergeInto(ThreadedRegionizer.java:844) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at io.papermc.paper.threadedregions.ThreadedRegionizer.onRegionRelease(ThreadedRegionizer.java:487) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at io.papermc.paper.threadedregions.ThreadedRegionizer$ThreadedRegion.markNotTicking(ThreadedRegionizer.java:937) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at io.papermc.paper.threadedregions.TickRegions$ConcreteRegionTickHandle.markNotTicking(TickRegions.java:355) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at io.papermc.paper.threadedregions.TickRegionScheduler$RegionScheduleHandle.runTick(TickRegionScheduler.java:415) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at ca.spottedleaf.concurrentutil.scheduler.SchedulerThreadPool$TickThreadRunner.run(SchedulerThreadPool.java:525) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
[07:29:52] [Region Scheduler Thread #1/ERROR]: [io.papermc.paper.threadedregions.TickRegionScheduler] Uncaught exception in tick thread "Region Scheduler Thread #1"
java.util.ConcurrentModificationException: null
	at java.util.ArrayDeque.nonNullElementAt(ArrayDeque.java:270) ~[?:?]
	at java.util.ArrayDeque$DescendingIterator.next(ArrayDeque.java:748) ~[?:?]
	at io.papermc.paper.threadedregions.RegionizedTaskQueue$PrioritisedQueue.mergeInto(RegionizedTaskQueue.java:334) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at io.papermc.paper.threadedregions.RegionizedTaskQueue$PrioritisedQueue.mergeInto(RegionizedTaskQueue.java:316) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at io.papermc.paper.threadedregions.RegionizedTaskQueue$RegionTaskQueueData.mergeInto(RegionizedTaskQueue.java:229) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at io.papermc.paper.threadedregions.TickRegions$TickRegionData.mergeInto(TickRegions.java:295) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at io.papermc.paper.threadedregions.ThreadedRegionizer$ThreadedRegion.mergeInto(ThreadedRegionizer.java:844) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at io.papermc.paper.threadedregions.ThreadedRegionizer.onRegionRelease(ThreadedRegionizer.java:487) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at io.papermc.paper.threadedregions.ThreadedRegionizer$ThreadedRegion.markNotTicking(ThreadedRegionizer.java:937) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at io.papermc.paper.threadedregions.TickRegions$ConcreteRegionTickHandle.markNotTicking(TickRegions.java:355) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at io.papermc.paper.threadedregions.TickRegionScheduler$RegionScheduleHandle.runTick(TickRegionScheduler.java:415) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at ca.spottedleaf.concurrentutil.scheduler.SchedulerThreadPool$TickThreadRunner.run(SchedulerThreadPool.java:525) ~[folia-1.20.1.jar:git-Folia-"34039e3"]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
[07:29:52] [Region shutdown thread/INFO]: [RegionShutdownThread] Awaiting scheduler termination for 60s
[07:29:52] [Region shutdown thread/INFO]: [RegionShutdownThread] Scheduler halted
[07:29:52] [Region shutdown thread/INFO]: Stopping server

Plugin and Datapack List

• Spartan

Actions to reproduce (if known)

The very small probability of occurrence in a long run seems unpredictable.

Folia version

This server is running Folia version git-Folia-"34039e3" (MC: 1.20.1) (Implementing API version 1.20.1-R0.1-SNAPSHOT) (Git: 34039e3)
You are running the latest version
Previous version: git-Folia-"7c6e251" (MC: 1.20.1)

Other

No response

[ocelotpotpie]

Can you post more of the log? Do you have more info on what was happening to reproduce this?

How many cores do you have?

[xuancat0208]

It is observed that this runs for a long time before it appears once, and the latest one is running for more than 1 day continuously.

JVM parameter:

/usr/lib/jvm/java-17-openjdk-amd64/bin/java -Xms32768M -Xmx32768M -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=200 -XX:+UnlockExperimentalVMOptions -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=40 -XX:G1HeapRegionSize=8M -XX:G1ReservePercent=20 -XX:G1HeapWastePercent=5 -XX:G1MixedGCCountTarget=4 -XX:InitiatingHeapOccupancyPercent=15 -XX:G1MixedGCLiveThresholdPercent=90 -XX:G1RSetUpdatingPauseTimePercent=5 -XX:SurvivorRatio=32 -XX:+PerfDisableSharedMem -XX:MaxTenuringThreshold=1 --enable-preview -jar ./folia-1.20.1-3.jar --nogui

CPU AMD Ryzen 7 5800X - 8c/16t - 3.8 GHz/4.7 GHz
RAM 64 GB ECC 2400 MHz
Data disks 2×960 GB SSD NVMe Soft RAID

The rest of the journal is nothing special, only Spartan's mistakes.

[Spottedleaf]

I don't understand how this crash could occur. I've made a fix in this area (81fe50f), but I don't think it is likely to have resolved the original issue. Can you update at least and see if it still occurs?

[xuancat0208]

OK, I will run 3 day ~ 1 week to test.

[xuancat0208]

It has been running continuously for 4 days without this error.
The error has been fixed!