Server overloaded, but hardware barely used.

[joryirving]

Hi there, running a vanilla server in docker.
Windows 10, WSL2 with an ubuntu backend.
HP Z820, Dual-Xeon E5-2680
64GB DDR3 RAM
No other containers runnning.

I'm running into an issue where Windows shows maybe 11% CPU usage, with spikes to 70%, however docker/Vmmem never exceeds ~10% total, but when I go the container stats, I spike from 50-200% CPU usage, and get constant console errors stating:

[15:29:47] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 4943ms or 98 ticks behind
[15:30:09] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7761ms or 155 ticks behind
[15:30:30] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5486ms or 109 ticks behind
[15:31:02] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 2029ms or 40 ticks behind
[15:31:26] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 8537ms or 170 ticks behind
[15:31:43] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 2649ms or 52 ticks behind
[15:32:04] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5831ms or 116 ticks behind
[15:32:23] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 3914ms or 78 ticks behind
[15:32:45] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7204ms or 144 ticks behind

This only happens when I have 4-5 players connected. I've increased the MEMORY to 4G, and it alleviated the issue when only 3 players were connected, but obviously, I don't want to feed it something like 16GB just so like 6 people can play.

Docker run command was:
docker run -d --init --name=minecraft-vanilla -e EULA=true -v ~/docker/minecraft:/data -p 25565:25565 --restart=always -e TZ=America/Edmonton -e ENABLE_AUTOPAUSE=TRUE -e MEMORY=4G itzg/minecraft-server:latest

[joryirving]

Should also mention, using the progrium/stress docker container, I can successfully engage all 32threads.

[itzg]

I'm sorry, I'm not able to help with performance issues. Please report those issues to Mojang since the container ultimately just exec's the java process against the minecraft server jar.

My guess is that the minecraft server, like most game engines, is primarily single threaded. As such, it's doesn't matter if you throw 32 cores at it, it's going to max out one of those cores. As such, your findings with progrium/stress indeed confirm the WSL2 VM is making proper use of your hardware; however, those findings don't translate to Mojang's implementation of the minecraft server.

You may want to try some of the other image tags, such as itzg/minecraft-server:adopt15 to see if using a newer version of Java happens to perform differently. You may also want to try some different TYPEs like PAPER.

[joryirving]

I found this link: #466 (comment) and tried the activeprocessor count, which initially I thought may be working, but didn't actually solve the issue.
I rebooted the host, and recreated the container with a docker-compose, and on re-creation, I saw the CPU Usage spike to 800%, which means it should be using 8 cores. I'm hopeful it was Java using that, and not just docker itself. I'm hoping to get some friends on there to stress test, as it seems to happen when people are in multiple different chunks.

For reference, I'm also trying the adopt13 tag, and here is my current docker-compose.

version: '3'
services:
  vanilla:
    image: itzg/minecraft-server:adopt13
    ports:
      - "25565:25565"
    volumes:
      - ~/docker/minecraft:/data
    environment:
      EULA: "TRUE"
      TZ: "America/Edmonton"
      ENABLE_AUTOPAUSE: "TRUE"
      INIT_MEMORY: "1G"
      MAX_MEMORY: "4G"
      MAX_PLAYERS: 15
    deploy:
      resources:
        limits:
          cpus: 8
          memory: 4G
        reservations:
          cpus: 2
          memory: 1G
    restart: always

I'm curious what kind of hardware people are running for larger servers? I wouldn't think that a 5 person vanilla server with no mods could max out a core that easily.

[flololan]

any news on this? did setting cpus help?

[joryirving]

It didn't appear to use any more than 3 cores. One for the main java process, and 1-2 for cleanup and such.

I found that my issue was disk I/O. My Ubuntu vhdx was sitting on a platter disc (HDD) to save space on the primary SSD. I ended up tossing in an old 120GB SATA SSD, migrating the WSL installation to that SSD, and then starting up the server (which was running off the /home/user directory within the Ubuntu WSL).

That appears to have resolved everything.

[flololan]

Hmm ok. We just had a horrible server launch with Forge-Server. VPS with 10 cores and 60Gb ram. Ram was barely used (around 15gb) and except one core it the rest was more or less idle... the vps uses an ssd... dedicated hardware is not an option sadly

[joryirving]

What OS is docker running on?

[flololan]

ubuntu 20.04.2

[flololan]

If you want to see my docker-compose/stack (cleaned up):

version: '3'

services:
  minecraft-server:
    ports:
     - "25565:25565"
    volumes:
      - "mcserver:/data"    

    environment:
      MEMORY: "50G"
      MAX_PLAYERS: "20"
      EULA: "TRUE"
      SERVER_NAME: "xxxxx"
      ICON: "xxxxxx"
      TYPE: "FORGE"
      PVP: "false"
      ALLOW_FLIGHT: "true"
      ALLOW_NETHER: "true"
      GENERATE_STRUCTURES: "true"
      SPAWN_ANIMALS: "true"
      SPAWN_MONSTERS: "true"
      SPAWN_NPCS: "true"
      MODE: "survival"
      FORCE_GAMEMODE: "true"
      OVERRIDE_SERVER_PROPERTIES: "true"
      OVERRIDE_OPS: "true"
      OPS: "xxxxxx"
      MOTD: "xxxxxx"

    image: itzg/minecraft-server

    tty: true
    stdin_open: true
    networks:
     - traefik-public
    deploy:
     labels:
  [Trafik Stuff]

 [other service for web redirection]

volumes:
  mcserver:
    driver: local
    driver_opts:
       o: bind
       type: none
       device: /root/minecraft

networks:
  traefik-public:
    external: true

[joryirving]

That looks fine. I assume using htop shows that only a few cored are being used, and tracking disk i/o doesn't show any issues?

[flololan]

I/O looks fine. Just seems to be a problem with single core usage sadly...

[joryirving]

Is it possible that the initial generation is causing a huge CPU spike, and if you let it idle for a while, it'll calm down? Not sure how powerful the cores on your VPS are.

[flololan]

Yeah it calmed down but mostly because we lowered the player count (we were aiming for at least 40 player). One core is still more or less non stop at 100% though.
hardinfo says it is a Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz

I think I will try your example from 23 days ago not sure if it works with forge though but as far as I've seen the latest forge is not required to have java 8

[joryirving]

I still around 100% once I have a single player on the server. Adding players doens't appear to impact performance, even though CPU usage stays high. I've never had more than 5 players connecting simultaneously, but they were all spread out in different chunks across the overworld/nether, and performance didn't appear impacted.

This is an updated version of my docker compose:

services:
  vanilla:
    container_name: minecraft_vanilla
    image: itzg/minecraft-server:latest
    ports:
    - 25565:25565
    volumes:
      - /home/xxx/docker/minecraft/vanilla:/data
    environment:
      EULA: "TRUE"
      TZ: "America/Edmonton"
      ENABLE_AUTOPAUSE: "TRUE"
      MAX_TICK_TIME: -1
      INIT_MEMORY: "1G"
      MAX_MEMORY: "4G"
      MAX_PLAYERS: 12
    restart: unless-stopped
  forge:
    container_name: minecraft_forge
    image: itzg/minecraft-server:latest
    ports:
    - 25566:25565
    volumes:
    - /home/xxx/docker/minecraft/forge:/data
    - /home/xxx/docker/minecraft/mods:/mods:ro
    environment:
      EULA: "TRUE"
      TZ: "America/Edmonton"
      TYPE: FORGE
      ENABLE_AUTOPAUSE: "TRUE"
      MAX_TICK_TIME: -1
      INIT_MEMORY: "1G"
      MAX_MEMORY: "4G"
      MAX_PLAYERS: 12
    restart: unless-stopped```

[flololan]

Ah I see you have no more CPU options. Hmmm. Ok. Thanks anyway. Will have to see what to do

[itzg]

@benab and @finndo77 please read the comments from #555 (comment) and down.

[yllekz]

FWIW there's been a multitude of open issues on Mojang's official Jira regarding Linux server performance being unacceptable for quite some time now. I personally also ran into this and got quite frustrated at how terribly the server performed. I got around it by moving to the Paper mod which fixed it via their own code in spite of Mojang's lack of movement on the issue. I'm not sure if it's been fixed to date, but just wanted to share my experience if you are seemingly at the end of your rope with this issue.

[itzg]

Thanks @yllekz . Yes, the latest combo I have heard working best for people, even on RaspberryPi, is TYPE=PAPER with the itzg/minecraft-server:multiarch-latest or itzg/minecraft-server:java-15 (both of which are multi-architecture).

[anutting]

I'm running an instance in AWS of size m5.xlarge (4 vCPU, 16GB Memory) with the JVM configured to use 8GB of Memory. My small server of 5 people is still seeing some "Can't keep up!" messages resulting in really stuttery mob movement. We haven't truly tested the performance with everyone online a flying around multiple dimensions, but I have a feeling the perfomance won't be great. Judging by some other posts in this thread, we should have no issue supporting 5 people on a server. Does anybody have tips for improving performance?

[itzg]

@anutting have you tried TYPE=PAPER or is that what you're already using ?

[anutting]

No, I'm running a vanilla 1.17 server. Would that help?

…

On Mon, Jun 28, 2021, 10:38 PM Geoff Bourne ***@***.***> wrote: @anutting <https://github.com/anutting> have you tried TYPE=PAPER or is that what you're already using ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#744 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFDCIFUF5PVLTFVV2CYF2PDTVEW4HANCNFSM4W7X7CVA> .

[jsravn]

Using PAPER fixed all my performance problems as well.

[Adrian-at-CrimsonAzure]

Unfortunately Paper doesn't seem to have a 1.17.1-rc1 image, Caves and Cliffs with the latest patches and datapack is what I want to checkout.

[anutting]

I've tried PAPER and it seems to be working with my 1.17 server, however the performance was not any better. Memory and CPU were way too high which resulted in server crashes. Still looking for suggestions on how to increase server stability. I'm using https://github.com/vatertime/minecraft-spot-pricing for reference.

[yllekz]

Unless something changed incredibly recently, Paper doesn't have a 1.17 release yet. They have a GitHub page tracking its development.

[jsravn]

@anutting seems like you need a more powerful instance perhaps? If paper doesn't solve it for you, there isn't much else you can do I think. Paper does a lot to improve performance - which is atrocious on vanilla, unfortunately.

[itzg]

FYI, I'm going to convert this issue to a discussion because a) it's a great discussion going and b) there's nothing I can do in the image itself to improve performance.

[jbeck22]

Hi there, running a vanilla server in docker. Windows 10, WSL2 with an ubuntu backend. HP Z820, Dual-Xeon E5-2680 64GB DDR3 RAM No other containers runnning.

I'm running into an issue where Windows shows maybe 11% CPU usage, with spikes to 70%, however docker/Vmmem never exceeds ~10% total, but when I go the container stats, I spike from 50-200% CPU usage, and get constant console errors stating:

[15:29:47] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 4943ms or 98 ticks behind
[15:30:09] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7761ms or 155 ticks behind
[15:30:30] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5486ms or 109 ticks behind
[15:31:02] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 2029ms or 40 ticks behind
[15:31:26] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 8537ms or 170 ticks behind
[15:31:43] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 2649ms or 52 ticks behind
[15:32:04] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 5831ms or 116 ticks behind
[15:32:23] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 3914ms or 78 ticks behind
[15:32:45] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7204ms or 144 ticks behind

This only happens when I have 4-5 players connected. I've increased the MEMORY to 4G, and it alleviated the issue when only 3 players were connected, but obviously, I don't want to feed it something like 16GB just so like 6 people can play.

Docker run command was: docker run -d --init --name=minecraft-vanilla -e EULA=true -v ~/docker/minecraft:/data -p 25565:25565 --restart=always -e TZ=America/Edmonton -e ENABLE_AUTOPAUSE=TRUE -e MEMORY=4G itzg/minecraft-server:latest

stupid question, but what did you use to gather your graphs and data like that? Looks very interesting.

[joryirving]

The top image is just htop inside the WSL2 terminal.
The second image was Docker desktop
Third image is obviously windows task manager, and the final image was a portainer container.

[Dratnet]

Found this post while looking to optimize two separate Minecraft servers running within K3s cluster.

Added these arguments to the JVM_XX_OPTS key as space-delimited values per @itzg JVM Options

-XX:+UseG1GC -XX:+ParallelRefProcEnabled
-XX:MaxGCPauseMillis=200 -XX:+UnlockExperimentalVMOptions
-XX:+DisableExplicitGC -XX:+AlwaysPreTouch
-XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=40
-XX:G1HeapRegionSize=8M -XX:G1ReservePercent=20
-XX:G1HeapWastePercent=5 -XX:G1MixedGCCountTarget=4
-XX:InitiatingHeapOccupancyPercent=15
-XX:G1MixedGCLiveThresholdPercent=90
-XX:G1RSetUpdatingPauseTimePercent=5 -XX:SurvivorRatio=32
-XX:+PerfDisableSharedMem -XX:MaxTenuringThreshold=1

Reference: https://aikar.co/2018/07/02/tuning-the-jvm-g1gc-garbage-collector-flags-for-minecraft/

• Credits go entirely to Aikar.
• Technical explanations of the flags can be seen near the end of the post.

Used this on Paper Vanilla 1.20.2 and BMC 2 [Fabric] 1.20.2 both hosted with the latest flag and 8GB of RAM. Haven't done a full measurement of performance improvement; but it is visibly better on BMC 2 server, less tick error messages in the logs.

[itzg]

Thanks for the info @Dratnet

There's also a convenience option for Aikar's flags

https://docker-minecraft-server.readthedocs.io/en/latest/configuration/jvm-options/#enable-aikars-flags

[Dratnet]

This is what I get for reading the doc and stopping at "JVM options" in the readme. Oof.

[finndo77]

wow, thought this one was closed out. I was able to resolve all of my performance issues by

• STOP using Aikar's flags (unless they have been updated in the last 20 months or so) unless you are running 1.17.x or older servers, they are no longer relevant to current Minecraft releases. (I did find this on reddit about 18-20 months ago that people were no longer seeing performance improvements with this flag)
• Minecraft now uses more cores, it is still a "single core/thread" java app, but it can now offload chunk loading to a separate core and some other features. Giving it 4 cores seems to help a lot, I can't say more than that would be beneficial or not.
• give it lots more RAM
  • after 1.19 came out minecraft became a RAM hog, not because it needs, but it over uses it. it might run at 4GB just fine, but sometimes it wants 12GB for a few moments, then will slowly drop back down to 4GB again, it did not do this on older versions, I used to run 1.16 with 1GB and I could get 8-10 players on there easy with no issues, around 12-14 players would cause lag spikes.
  • I have a server right now running on Oracle Cloud free tier (4 core AMPERE server, basically a mid/low end cell phone) with 22GB of ram assigned to MC and it handles 14 players concurrently, spread out over 30,000 blocks x 30,000 blocks, and I never get server is falling behind errors, except after somewhere around the 5th-7th person to log on, there is a 100-2500 ms lag spike during their initial connection that goes away immediately. I have dropped it to a max of 12 though in case future features cause issues.
• I have not used the autopause flag since my last post on this issue.
• performance will improve on an NVME, not necessary, but you can grab 1TB for $50-$80 right now and is worth getting rid of IO performance as the headache
  • if the storage drive is running behind due to high IO operations, the CPU will churn at a high % using "wait" cycles constantly hitting up the storage system for the data it needs, the more backed up processes that need info from the storage system, the more processes that will be hitting up the storage system for data, the more CPU usage you get...
  • 1.20 has added a lot of bloat to the amount of background processing minecraft has to perform and it is only getting worse each release, expect higher requirements as 1.21 comes out

I hope some of that helps, at least try some of it for troubleshooting where available (not telling you to spend money if you can resolve it another way)

My server on Oracle cloud, has been online since 1.19 snapshots started. I make a clean server for each new major release version, and then clean the map and copy it and player data and achievements over to the new server, so my regular players don't lose their builds or gear, but gain the new version's snapshot functionality and features.

Once a month or so I run a storage cleanup and every year I do a map reset of chunks not used much with MCASelector, I do a broad sweep of any chunk not used for more than 5 minutes, and then a second run of any chunk not accessed for 180 days. Here is my docker launch script for the server I am running on Oracle (linux server), followed by my clean up commands I run and my rebuild script. (I know some of the commands are the default settings, but they are there to override any changes that may have occurred)

#! /bin/bash

echo ""
echo "starting up mc-snapshot server"
docker run -d\
  --name mc-snapshot\
  -e OVERRIDE_SERVER_PROPERTIES=true\
  -e UID=1001\
  -e GID=1001\
  -e INIT_MEMORY=22G\
  -e MAX_MEMORY=22G\
  -e VERSION=snapshot\
  -e INITIAL_ENABLED_PACKS="bundle,update_1_21"\
  -e EULA=TRUE\
  -e ENABLE_RCON=true\
  -e RCON_PORT=25575\
  -e OPS=Me\
  -e MOTD='1.21 experimental features enabled!'\
  -e PVP=true\
  -e SPAWN_PROTECTION=5\
  -e SIMULATION_DISTANCE=8\
  -e VIEW_DISTANCE=18\
  -e PLAYER_IDLE_TIMEOUT=6000\
  -e MAX_WORLD_SIZE=75000\
  -e MODE=survival\
  -e FORCE_GAMEMODE=true\
  -e DIFFICULTY=hard\
  -e SERVER_NAME=mc-snapshot\
  -e LEVEL=mc-snapshot\
  -e SNOWACCUMULATIONHEIGHT=6\
  -e MAX_PLAYERS=12\
  -e SPAWN_ANIMALS=true\
  -e SPAWN_MONSTERS=true\
  -e SPAWN_NPCS=true\
  -e ALLOW_NETHER=true\
  -p 25565:25565\
  -v /srv/minecraft/mc-snapshot:/data\
  -v /etc/timezone:/etc/timezone:ro\
  itzg/minecraft-server:java17

echo "startup process for mc-snapshot completed"
echo ""

exit

and this I run every couple of months, sometimes it gives me a good 4GB back in drive space

# removing old log files greater than 15 days"
echo "removing old logs greater than 15 days"
find /srv/minecraft/mc121/logs/ -maxdepth 1 -mtime +15 -type f -name *.gz -exec rm -f {} \;


echo "pruning old docker images"
docker image prune -a

also, here is my rebuild script (mostly just calls various other little scripts, but has some things that might be useful for anyone), I added some extra comments that are not in the script to explain why I am doing some of the things. I run this on a scheduled task DAILY.

#!/bin/bash

# making sure docker can run
# there are settings on Oracle that by default block docker from having internet access,
# unfortunately these reset EVERY reboot.   I run it every time, in case something forced the server to restart
# (Oracle does that if you don't reboot often enough or there is an outage)
# I use echo commands as I have the output of the script entered into a system log file so later
# I can look at the output if it failed
echo "running docker post-reboot script"
/srv/minecraft/scripts/docker-reboot-fix.sh

# announcing to server that restart is incoming
# "wall" prints the message to all logged in terminal users on the Oracle server, so if I am on the server
# I know when it starts and ends this process
wall "mc-snapshot is about to be rebuilt"
docker container exec mc-snapshot rconcli "say server is restarting to check for a new snapshot"
docker container exec mc-snapshot rconcli "say if everything goes well, should take about 20 seconds"

# downloading updated docker image files
# I know it will download these during the run command if missing, but this way
# I have them ready to go while the server is still running, making the downtime as short as 10 seconds
echo "downloading updated docker image files"
docker image pull itzg/minecraft-server:java17

# removing old log files greater than 15 days"
echo "removing old logs greater than 15 days"
find /srv/minecraft/mc-snapshot/logs/ -maxdepth 1 -mtime +15 -type f -name *.gz -exec rm -f {} \;

echo ""

# stopping docker container mc-snapshot
echo "stopping docker container mc-snapshot"
docker container stop mc-snapshot

# cleaning up old jars
echo "cleaning up old jars"
rm -rf /srv/minecraft/mc-snapshot/*.jar
rm -rf /srv/minecraft/mc-snapshot/versions/*

# removing docker container mc-snapshot
echo "removing docker container mc-snapshot"
docker container rm mc-snapshot

# starting new container mc-snapshot
echo "starting new container mc-snapshot"
/srv/minecraft/scripts/launch-mc-snapshot.sh

echo ""
echo "done"
wall "rebuild complete"
date
echo ""

exit