World Generation is slow and errorneous in multiplayer

[skaldarnar]

World generation in multiplayer is slow and chunky (no pun intended), especially with more players on the server.

• chunks loading slowly - it is easy to reach the "edge of the (generated) world", even with normal walking speed.
• chunks are missing - some chunks are missing for clients, and even waiting for a long time does not bring them up. The missing chunks are not the same for all clients.
• LOD chunks or not visible - not sure whether they are not generated at all, or just slow to generate and queuing up behind regular chunk generation

This happens on https://github.com/MovingBlocks/Terasology/releases/tag/v5.2.0-rc.1 and previous releases.

---

Related to #4707

[skaldarnar]

Potentially related: server running out of memory (OOM) during the play-test:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
./run_linux_load_last.sh: line 3: 11769 Aborted                 (core dumped) java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1089 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Xms128m -Xmx6144m -jar libs/Terasology.jar --headless --homedir=server --override-default-config=override.cfg --load-last-game --oom-score=500 --max-data-size=7500m
root@testserver:/opt/terasology# which java
/usr/bin/java
root@testserver:/opt/terasology# java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed mode)

[DarkWeird]

Slow loading: How many players was and how faster they move in different direction?
(I suspect full cpu utilization)

Missing chunks: looks familiar. I think fixed it (don't remember pr)
Seems it is problem at network system(not same chunks) or in remote chunk provider.

Lod chunks : are we provide chunks for multiplayer? (Or every client generate lod chunks for self use?)

Oom:
Chunks takes many memory. Especially players in non overlap regions ... And some one can request extreme distance..
Need to profile a bit.

[naalit]

An overview of and some problems with the parts of the chunk generation system I've looked into, which accounts for part of these problems:

Whenever a player moves, the game calculates all chunks in range of that player that aren't loaded and submits them to the ChunkProcessingPipeline. The ChunkProcessingPipeline adds them to a map by position and also a PriorityBlockingQueue, so it doesn't add duplicates to the queue. This queue is supposed to be sorted by the distance to the closest player, but because that changes and the queue implementation assumes it doesn't change, that doesn't really work - so one way chunks can take forever to load is if it's first queued when it's far away from the closest player, so it goes to the back of the queue, and then when players get closer the queue doesn't update so it stays on the back. As long as other chunks are still being added to the queue, that chunk will never get to the front. That's the primary problem that #4822 is meant to fix.

That queue doesn't contain Chunks, it contains RunnableFuture<Chunk>s, so the chunks aren't generated until they're unqueued, but it probably takes up a lot of memory. There are four worker threads which continuously take work from the queue, run it, and move on to the next task; there's also one "reactor" thread (unrelated to the Project Reactor library) which checks for finished tasks and queues the next stage of processing for each chunk. When a chunk is no longer in range, the RunnableFuture is canceled, so it's not removed from the queue (and so the queue may grow in size) but if it's unqueued to be run it won't do any work.

When a chunk is finished, it's added to a BlockingQueue in the LocalChunkProvider. The next time update is called, that queue is drained on the main thread and each chunk is added to the world and events are sent.

That last paragraph is still true with #4822; the first two paragraphs aren't, though. Instead, that PR replaces the queue with a list of Vector3ic chunk positions which is deduplicated, kept sorted by distance to closest player (chunks too far away are removed), and updated every time a player moves. There are two (a configurable constant) threads which are Reactor Subscribers, each of which continuously polls the top eight chunk positions in the list (this is synchronized so the threads don't get the same chunks) and generates those chunks. Then the chunks are added to the processing map and a queue, like the old version. After the batch of eight chunks is generated, all chunks in the queue are processed as much as they can be (some tasks like light merging must wait until other chunks are generated, in which case the chunk will stay in the queue). When all tasks are complete for a chunk, it's added to the ready queue in the LocalChunkProvider, just like before.