feature RQ -- Bedrock Worlds backup

[MattrCoUk]

My server's been been working great on my MacMini, thanks to this repo! With few friends we've started building a village, with people connecting at various times. It would be a shame if all wonderful builds they create got destroyed.

Bedrock server has the built-in backup functionality which works as I tested. But it is far from automated.

It involves tying in the server console save hold, save query, copy world files and save resume.

I was wondering if it would be somehow possible to automate it. Run with an interval. Perhaps a shell script in the container sending mentioned commands to the Bedrock console and then copying files to an external volume.

I could work on this but don’t have clue where to start...

Maybe I could use something like Expect?

[itzg]

I agree that it would be really nice to automate that process, but yes it would need to be something that could inspect the save query and react accordingly.

Right now the server process is wrapped by this helper tool I developed
https://github.com/itzg/entrypoint-demoter

That tool is currently generic to any type of containerized process; however, I'm thinking I need to spin off a bedrock server specific version of that. With that it could expose bedrock-specific operations such as doing a backup.

Enhancing that Go-based tool might have been more than what you were offering, but that's where I would see the enhancement happening. I have been wanting to expose console access via a web API anyway, so I might prioritize working on both of these changes soon myself.

[MattrCoUk]

Great stuff! thanks!

[MattrCoUk]

btw.. If you play MC yourself it would be a pleasure to invite u to our lil server, it’s great fun. Let me know!

[itzg]

I'm kind of a solitary LAN-server player, but yeah I should really get out there in the "world" ;) Shoot me your server address in email (itzgeoff@gmail.com) or DM me on Discord and I'll jump on sometime.

[MattrCoUk]

I’ve created a little script that seems to be working fine on 1.16.1, thought I share.
It requires expect and can be run with cron or lunchd every half hour or so.
(You might need to verify your docker path by running which docker)

#!/usr/bin/expect --

set world_dir <dirs>/worlds
set docker_container <container name>
set world_name <world name>
set backup_dest “<dirs>"
set timestamp [timestamp -format {%Y-%m-%d--%H-%M}]
set world_compress_timeout 240
set world_arch_mv_timeout 30
# log_file <path>backup-world.log  # the log file might get large quickly!

spawn /usr/local/bin/docker attach --detach-keys=Q $docker_container
expect "DEBU*”


while true {
  
  send "save hold\r"
  expect "Saving...*" { break }
  expect "The command is already running*" { break }
}
sleep 1

while true {
  
  send "save query\r"
  expect "Data saved.*" { break }
}

while true {
  
  send "Q"
  expect eof { break }
}

spawn bash
send "cd \"$world_dir\"\r"
expect "*$ "

set timeout $world_compress_timeout
send "tar -zcf $world_name.$timestamp.tgz $world_name\r"
expect "*$ "

set timeout $world_arch_mv_timeout
send "mv $world_name.$timestamp.tgz \"$backup_dest\"\r"
expect "*$ "
send "exit\r"
expect eof

set timeout 10
spawn /usr/local/bin/docker attach --detach-keys=Q $docker_container
expect "DEBU*"
while true {
  
  send "save resume\r"
  expect "Changes to the level are resumed.*" { break }
}

while true {
  
  send "Q"
  expect eof { break }
}
exit

[DocBrown101]

Here is my simple backup-script. Works since one year!

#!/bin/bash
docker-compose -f "/home/daniel/dropbox/data/docker/3-minecraft/docker-compose.yml" down

_day="$(date +'%A')"
_file="minecraft-data-${_day}.tar"
_path="/home/daniel/dropbox/data/docker-backup/${_file}"

tar -P -cf ${_path} --exclude='bedrock_server*' /home/daniel/docker-data/minecraft --overwrite

docker-compose -f "/home/daniel/dropbox/data/docker/3-minecraft/docker-compose.yml" up -d --build

[wedgef5]

I'm sort of new to Minecraft, and I'm investigating setting up a Bedrock server to run on our LAN for our kids and a few of their friends. I don't have a clear understanding of the process of backing up worlds. When you speak of commands like "save hold" and "save query", is that to facilitate backing up a running server? Would it be an acceptable workaround to periodically stop the server, make a backup of the worlds folder and then start the server, assuming this is done when the server is definitely not in use?

[itzg]

@wedgef5 yes, doing the backup while the server is stopped would be a great alternative.

[Kaiede]

Just wanted to comment that I've been playing around with a tool that can do these sort of backups. I started by creating a CLI tool I could run via systemd on the docker host, and I recently containerized it with a bit of help from reading through the docker-mc-backup scripts. It is able to backup a running server just fine, but it does require access to the host's docker.sock file in order to accomplish this.

It's a bit rough around the edges at the moment because it's not what I'd consider fully containerized and so the config file contains values it doesn't actually need, but it is what I've been using for about 3 weeks now (via systemd) and the last few days (via docker) on my own two-server host.

Docker Hub: https://hub.docker.com/r/kaiede/minecraft-bedrock-backup
GitHub: https://github.com/Kaiede/docker-minecraft-bedrock-backup
CLI Tool: https://github.com/Kaiede/BedrockifierCLI

[itzg]

Awesome. @Kaiede when you're ready for it, I would be happy to add a "Backups solutions" section to the README that links to your image.

[Kaiede]

I've done the cleanup I want to do around the configuration. I think I've managed to get it to the "minimum viable product" state that I'm happy with, and the changes have shown to be stable on my home server. So feel free to add the reference.

Mostly worried that the documentation needs some revision, but at this point, will have to see where folks get hung up on configuring things.

[itzg]

There it is:

https://github.com/itzg/docker-minecraft-bedrock-server#solutions-for-backing-up-data

[MattrCoUk]

This is awesome. I’m just wondering how that compares performance-wise vs just making dumps of a world via MC server’s CLI and just tar-ing them? My server runs on a quite low-RAM machine stretched up to limit so wondering if another docker container would take-up much resources.

[itzg]

Containers don't add any memory overhead, so @Kaiede 's backup container would only occupy the memory for the bash script to sleep most of the time. Since the tool is compiled from Swift, I'm guessing it has very low memory footprint for those moments it runs.

[Kaiede]

The processes running in the container on my server are eating about 4KB of RAM total while sleeping. When backing up my two containers and trimming the backup list (one 80MB world, and a 300MB world), the peak RAM stayed under 60MB (RSS) and took less than one second (NVMe SSD). I don’t see this growing a lot with larger worlds, since the buffers are fixed in size.

Since the tool is compiled from Swift, I'm guessing it has very low memory footprint for those moments it runs.

Depends on what you compare it to. Something like the POSIX suite of tools are hard to beat. But compared to say, the node process for manymine, or dockerd, it uses less. The Swift runtime isn’t quite as good as the C/C++ runtime in memory usage.

[MattrCoUk]

I've done the cleanup I want to do around the configuration. I think I've managed to get it to the "minimum viable product" state that I'm happy with, and the changes have shown to be stable on my home server. So feel free to add the reference.

Mostly worried that the documentation needs some revision, but at this point, will have to see where folks get hung up on configuring things.

Just setting a new world for my server so decided to finally give it a go :-)
Hunged-up on paths config... :-|

docker-compose for my server for reference:

version: '3.4'

volumes:
  emer2_vol: 
    external: true

services:

  bds2:
    container_name: emer2

    image: itzg/minecraft-bedrock-server
    environment:
      # blah...

    ports:
      - 19133:19132/udp
      - 39134:39134
      - 39134:39134/udp
      - 39135:39135
      - 39135:39135/udp
    sysctls:
      net.ipv4.ip_local_port_range: 39134 39135

    volumes:
      - emer2_vol:/data

    stdin_open: true
    tty: true
    restart: unless-stopped




  
  backup:
      image: kaiede/minecraft-bedrock-backup
      name: emer2_backup
      restart: always
      depends_on:
        - "bds2"
      environment:
        BACKUP_INTERVAL: "3h"

      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
        - /opt/bedrock/backups:/backups
        - /opt/bedrock/server:/server

I would like to save the backup in e. g. /Volumes/Storage/mine/emer2/backup/world. Do I need to create an external volume and define it in docker-compose, similarly as I do for the data volume? or do I just replace the /opt/bedrock/backups bit. Also is the external data volume going to work with the above setup?

(also not sure if should just post this in https://github.com/Kaiede/docker-minecraft-bedrock-backup)

[itzg]

@MattrCoUk since you designated emer2_vol external, then yes you'll need to create that yourself. You also need to attached emer2_vol to the backup container at /data in order to point it at the content to backup.

[wedgef5]

Thanks for working on this @Kaiede

Will this solution also work on a Win10 host with appropriate path changes?

[Kaiede]

I would like to save the backup in e. g. /Volumes/Storage/mine/emer2/backup/world. Do I need to create an external volume and define it in docker-compose, similarly as I do for the data volume? or do I just replace the /opt/bedrock/backups bit. Also is the external data volume going to work with the above setup?

@MattrCoUk : My example for the backup is mostly there to give an idea what's possible. Whatever you map to /backups is where backup files get written to, and where config.json is expected to be, by default. So replace /opt/bedrock/backups with whatever you want so you can access things externally, such as /Volumes/Storage/mine/emer2/backup/world. I will say one thing though, is that I have never quite figured out how to deal with named volumes for restoring backups. It's honestly easier to restore while the server and backup containers are shut down, which makes it harder to access named volumes (something I never figured out how to do). So I'd generally recommend using external volumes for both containers so it's possible to shut down the containers, unzip the *.mcworld file into the world folder, and then start the containers back up.

Will this solution also work on a Win10 host with appropriate path changes?

@wedgef5 : In principle, it should, the same way running the container on macOS works. They both have to virtualize Linux to run either the backup container or the bedrock server container. So just keep in mind that paths inside the container are still going to be *nix paths.

[MattrCoUk]

So I’ve set it up on both of my worlds and it seems to be working splendidly!

here’s my docker-compose for reference
uses 2 external volumes

version: '3.4'
volumes:
  # create volumes before running docker-compose:
  # docker volume create --opt device=path/to/server/data --opt o=bind bs_data_volume --opt type=none
  # docker volume create --opt device=path/to/backup --opt o=bind bs_backup_volume --opt type=none
  bs_data_volume: 
    external: true
  bs_backup_volume: 
    external: true

services:
  bedrock_server:
    container_name: bedrock_server
    image: itzg/minecraft-bedrock-server
    volumes:
      - bs_data_volume:/data
    # ... etc.

  bs_backup:
    container_name: bs_backup
    image: kaiede/minecraft-bedrock-backup
    restart: always
    depends_on:
      - "bedrock_server"
    environment:
      BACKUP_INTERVAL: "3h"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - bs_backup_volume:/backups
      - bs_data_volume:/server

notes:

server's container_name and the service name has to be the same for backups to work correctly
would be nice to have an option to run backups on schedule (such as 6:00AM, 12:30PM etc.)

[Kaiede]

server's container_name and the service name has to be the same for backups to work correctly

docker-compose will name a container with a prefix if you don't specify the container_name manually. So if I have my compose file at: /opt/mygreatserver/docker-compose.yml and name the service bedrock_server the container winds up being named mygreatserver_bedrock_server, and other containers get the same prefix, effectively grouping them together. As long as the name docker itself sees is in the config.json, it will work.

So yeah, naming it explicitly is a good way to keep things easier to remember and work with for sure.

would be nice to have an option to run backups on schedule (such as 6:00AM, 12:30PM etc.)

Feel free to open an issue on my github repo for this. I intentionally left it out of the first iteration because:

• Making users enable cron on the docker host is effectively a support time sink.
• Integrating something cron-like in the container itself is pretty heavyweight.
• I have ideas that would require turning the backup tool itself into an always-running service monitoring the bedrock server (triggering backups on logout/login). If I go that route, then cron behaviors really should be handled by this service directly.

I figured being able to trim the backups, and run them a bit more frequently would help address some of the need for cron scheduling in the short term at least.

[itzg]

I agree with @Kaiede about the hesitations for directly supporting cron schedules. At the risk of being a heavy answer to an easy question, but kubernetes CronJob workloads are a robust way to support scheduled container activities

https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/

[MattrCoUk]

I’ve noticed that on one of my servers the backup produces some additional files. Other server with a similar backup settings doesn’t have them. Wonder if those are important.

[Kaiede]

I’ve noticed that on one of my servers the backup produces some additional files.

Right now, when a backup is fired, all worlds detected in the worlds folder for a server are backed up. A “valid” world is one where there is a folder with “levelname.txt” inside, and the tool can extract a useful string from that file.

The main issue here is that since the tool is unaware of what world the server is using, or if the user might have swapped out the worlds between backups firing, it just backs it all up. This is something I can probably document better.

So what’s likely happening here is that you’ve got a second folder in your server’s worlds folder, and it’s levelname.txt file contains the string “PMC87f…”. I can’t really say anything about what created that folder in your case, but if you were to dig into your server directory and find that folder, and clean it up, these extra backups would stop being made. Being only 6KB in size, it almost seems like a world that was “created” but never properly named or logged into.

[MattrCoUk]

I’ve noticed that on one of my servers the backup produces some additional files.

Right now, when a backup is fired, all worlds detected in the worlds folder for a server are backed up. A “valid” world is one where there is a folder with “levelname.txt” inside, and the tool can extract a useful string from that file.

The main issue here is that since the tool is unaware of what world the server is using, or if the user might have swapped out the worlds between backups firing, it just backs it all up. This is something I can probably document better.

So what’s likely happening here is that you’ve got a second folder in your server’s worlds folder, and it’s levelname.txt file contains the string “PMC87f…”. I can’t really say anything about what created that folder in your case, but if you were to dig into your server directory and find that folder, and clean it up, these extra backups would stop being made. Being only 6KB in size, it almost seems like a world that was “created” but never properly named or logged into.

Ahhhh, I forgot that I had to change the server's name temporarily in order to verify it on planet.minecraft. It generated a new world..... Silly me!
Great feature with backing up all worlds automatically btw.

[claflico]

Would anybody be able to help me understand the backup process a bit better? I've been reading about "save hold", "save query", screen, expect, etc. for a couple of hours now and still not quite comprehending everything.

Taking the docker commands (docker stop, docker start, docker attach) out of the picture; if we wanted to take a backup from INSIDE the running container is there a straightforward process for doing so?

@Kaiede I've looked at your BedrockifierCLI tool and that looks like it is used outside of the docker container, am I interpreting that correctly?

Thank you for any guidance or information.

[Kaiede]

Taking the docker commands (docker stop, docker start, docker attach) out of the picture; if we wanted to take a backup from INSIDE the running container is there a straightforward process for doing so?

Not really. You need to have access to the input/output of the server's console to issue commands to do this safely. So while you could write a script that runs inside the container, it would look a lot like a script that runs outside the container. The crux of taking a backup while the server runs relies on issuing the "save hold" and "save resume" commands. Those commands are there to tell the server to stop writing to disk for a moment so the backup can be made ("save hold"). Once the backup is made, you can issue "save resume" again to tell the server it's safe to write to disk again. Needing access to the server's console is what makes it a bit of a pain to get backup scripts working right.

The alternative is to shut down the server, make the backup and start it again instead, which achieves the same result, but brings down the server for anyone logged in while it does it. (This is the docker stop/start approach)

Docker itself isn't necessarily why it is complicated, and makes both approaches above a bit easier, to be honest.

I've looked at your BedrockifierCLI tool and that looks like it is used outside of the docker container, am I interpreting that correctly?

It currently assumes that the server to be backed up is in a docker container, but the tool itself can be used outside a container. I used it what way to backup my server container for a bit while I was working on the containerized version of the backup tool. But it doesn't support being used either in the same container as the server, or if the server isn't in a docker container at all.

The catch is that BedrockifierCLI can only really be built for Mac/Linux (Swift on Windows is still pretty experimental). So the containerized backup tool is honestly more compatible, and more "set and forget".

[wedgef5]

I've gotten this solution working on Win10 Pro. Thanks to @Kaiede for putting it together! Below is my docker-compose.yml which is using a mount to the host NTFS filesystem. Docker will issue a warning about this due to relatively poor performance of using Windows host FS mounts like this. I figure with the relatively low amount of IO that it shouldn't be an issue. I've also been investigating using a WSL2 Ubuntu instance to host the backup data, but I haven't actually tried it yet. It seems that it should be possible, though. My primary goal is to have the backed up worlds out of the container where they can be picked up by my cloud backup solution (iDrive). The easiest way to achieve that is for the backup to be on the host FS.

version: '3.4'

services:
  bedrock:
    image: itzg/minecraft-bedrock-server
    container_name: bedrock_server
    restart: always
    environment:
      EULA: "TRUE"
      SERVER_NAME: "Home Server"
      WHITE_LIST: "TRUE"
      WHITE_LIST_USERS: "wedgef5"
      GAMEMODE: creative
      DIFFICULTY: normal
      ONLINE_MODE: true
    ports:
      - 19132:19132/udp
    volumes:
      - bds:/data
    stdin_open: true
    tty: true
  backup:
    image: kaiede/minecraft-bedrock-backup
    container_name: bedrock_backup
    restart: always
    depends_on:
      - "bedrock"
    environment:
      BACKUP_INTERVAL: "3h"
      TZ: "America/Chicago"
    volumes:
      - //var/run/docker.sock:/var/run/docker.sock
      - G:/Minecraft-Backup:/backups
      - bds:/server
volumes:
  bds: {}

[Kaiede]

@wedgef5 Plumbing data around is always a pain, for sure.

Since I use B2 for my NAS backups already, I added a duplicati container to handle uploading my backups directory to a B2 container that houses my VM backup data. It looks like duplicati can be configured to upload to iDrive as well. That would let you wrap the whole thing up in either a Ubuntu VM or WSL2 setup.

I generally found a Hyper-V or VMWare VM was a bit more efficient (power, performance) than using Docker Desktop.

[tuxpeople]

@itzg No worries. I could ask my son if it's a question concerning in-game knowledge, but not for server stuff 😂

@Kaiede I think I understand: There's currently no way to do the necessary commands over the network, therefore you have to rely on attaching to the container to directly issue commands to it.

There's something for Kubernetes similar to docker attach: kubectl attach. I can use it to isse commands interactively, however, I don't see a way to script it atm. Am I assuming right that there's also no way to issue the neccessary commands from inside the bedrock container itself? And I also assuming correctly that the "normal" client protocol is unsuitable for this?

I have a way to issue commands to the server from within the running bedrock container. And also to read the response without asking @itzg to add additional tools to the container. Let's see how far I can go with this approach. Can you (@Kaiede) tell me how you create a .mcworld file from this:

[2023-10-24 07:08:35:392 INFO] Data saved. Files are now ready to be copied.
world/db/CURRENT:16, world/db/000016.ldb:208360, world/db/000015.log:0, world/db/MANIFEST-000013:354, world/level.dat:2733, world/level.dat_old:2733, world/levelname.txt:5

Thanks!

[Kaiede]

I think I understand: There's currently no way to do the necessary commands over the network, therefore you have to rely on attaching to the container to directly issue commands to it.

Correct. And this process is specific to the container engine being used. Kubernetes, being built around orchestration across a cluster of nodes, introduces wrinkles. I haven’t worked with it, so I’m not sure how straight-forward it would be to support. Not only does the backup container need access to the host’s docker instance, it needs to be able to share the data volume with the minecraft container. This gets messy in cases where the two containers might wind up on different nodes when using Kubernetes. I’m not sure how orchestration works when dealing with shared volumes.

I only run two nodes (one x86 and one arm64), so haven’t really needed something with cluster capability.

Am I assuming right that there's also no way to issue the neccessary commands from inside the bedrock container itself? And I also assuming correctly that the "normal" client protocol is unsuitable for this?

The normal client protocol is unsuitable because it means implementing a full client login with a Microsoft account, which is built for interactive logins. Note that logging into a server assumes that the client has already performed authentication and provides the token (the JWT listed in the protocol spec you shared) to the server. Also, I don’t remember if save commands are available to operators via the Bedrock client. So this would also be messy for an end user to configure and operate.

Keep in mind, the requirement of host docker access is needed for scripting of any kind from another container. For the backup container it’s just easier to attach and interact with the console directly, as the requirements are the same as asking the container to run scripts for you.

Can you (@Kaiede) tell me how you create a .mcworld file from this:

An .mcworld file is just a zip file of the world. It’s pretty simple once you know what the “root” of the zip file needs to be. If you really want to know more about how it works. you can export a single-player world from Bedrock on Windows, and rename the .mcworld to .zip and take a look inside. Or examine an existing mcworld file on linux using the zip tool to list the contents.

I wasn’t aware of the truncation requirements (Mojang doesn’t document this stuff), so my tool is rather crude here. It just bulk grabs everything on both Java and Bedrock. My backups have restored properly when I needed to do so, and I didn’t encounter corruption, so it just slipped under the radar. I’ll have to take a look to see if I can make the backups more correct going forward.

But thinking about this longer term, depending on how reliable the Bedrock console behavior is, I kinda think wrapping the server in a tool that translates to Rcon (or at least provides some sort of semi-secure console access over a docker/kubernetes private network) is the right approach. It would require integrating the work into the server container, but it is the cleanest way, and supporting Rcon would make it easier for tools to support both Bedrock and Java like mine does.

[tuxpeople]

Correct. And this process is specific to the container engine being used. Kubernetes, being built around orchestration across a cluster of nodes, introduces wrinkles. I haven’t worked with it, so I’m not sure how straight-forward it would be to support. Not only does the backup container need access to the host’s docker instance, it needs to be able to share the data volume with the minecraft container. This gets messy in cases where the two containers might wind up on different nodes when using Kubernetes. I’m not sure how orchestration works when dealing with shared volumes.

I do Kubernetes for living, therefore I know a thing or two. But you're right, it would be a bit messy. Underneath Kubernetes, you can have different container runtimes and also different storage plugins. Therefore, I can see only three approaches to Kubernetes backup which do make sense:

• Rely on the Kubernetes API to interact with the pod (container)
• Run the backup from inside the minecraft-bedrock-server container (would also work on Docker)
• Interact with the Minecraft server directly

The last thing is not possible on Bedrock due to the lack of rcon.

Can you (@Kaiede) tell me how you create a .mcworld file from this:

An .mcworld file is just a zip file of the world. It’s pretty simple once you know what the “root” of the zip file needs to be. If you really want to know more about how it works. you can export a single-player world from Bedrock on Windows, and rename the .mcworld to .zip and take a look inside. Or examine an existing mcworld file on linux using the zip tool to list the contents.

Yep, I see it now. Thanks!

I wasn’t aware of the truncation requirements (Mojang doesn’t document this stuff), so my tool is rather crude here. It just bulk grabs everything on both Java and Bedrock. My backups have restored properly when I needed to do so, and I didn’t encounter corruption, so it just slipped under the radar. I’ll have to take a look to see if I can make the backups more correct going forward.

But thinking about this longer term, depending on how reliable the Bedrock console behavior is, I kinda think wrapping the server in a tool that translates to Rcon (or at least provides some sort of semi-secure console access over a docker/kubernetes private network) is the right approach. It would require integrating the work into the server container, but it is the cleanest way, and supporting Rcon would make it easier for tools to support both Bedrock and Java like mine does.

My current status on backup is the following: I do have a working backup script to be run inside the container to issue the commands needed and copy/truncate the files. The script will compress the backup and write it into a folder. I was able to successfully import a backup of my server into my iPads Minecraft. The script gets automatically added to the container by Kubernetes and I can set an optional env-var to specify a name for the backup.

As @itzg makes good containers (they do not contain unnecessary tools), I had some issues :-) The current version of script has the following caveats:

• I need to do some nasty stuff to get the PID of the Minecraft server process (as I can't be sure that it's always PID 11). This could be made more mature if @itzg would write the PID of the server process into a file.
• I can't create .mcworld files as there's no zip utility in the container. I'm currently doing .tar.gz. That's good enough for backup, but not if one would like to use the backup for importing it into a Minecraft client like the iPad Minecraft.
• It can't be run regularly based on a schedule (but can be run manually if a backup should be made)
• Script needs some cleanup and a few more variables to be configured using env vars.
• Script has currently clue of retention periods and deletion of old backups.

[tuxpeople]

@itzg would you consider merging a PR for adding zip to the container? It would allow the creation of .mcworld files, which are ZIP files.

With ZIP I can conduct some testing and I'd be happy to open another PR later to discuss the possibility of adding a backup script to the container.

[itzg]

@tuxpeople I would prefer backup operations to be separate from this image since the final step exec's off to the bedrock server executable. With that said, I don't mind zip being added to the packages, but be aware that I may choose to reject a PR that overall extends the container's functional scope.

[tuxpeople]

@itzg I completely see your point. What I propose is far from what's considered best practices. It's not in alignment with general container best practices and also not with Kubernetes best practices. Therefore, I'm not happy with as well. But as far as I can see, it's the best of all the bad options. But maybe I'm missing something.

Access to the files is not a problem. Regardless of the storage solution used for the Kubernetes deployment. The big issue is to speak with Bedrock Server. What I'm doing now in the script is to echo the commands into the server process's stdin and read the answers from its stdout. This can only be done from inside the Bedrock container.

I can do the same stuff with speaking to the Kubernetes API. Probably similar to what @Kaiede does with the Docker socket. But I'm not a programmer. Therefore I can't do some programming magic and would need to rely on the standard Kubernetes CLI: kubectl attach <pod> gives me access to the I/O of the container. But it's not directly scriptable. I would have to use something like expect. But I'm not sure how mature and reliable this is.

May I ask you for your opinion on this? Not necessary on the expect thing, more on the issue in general.

[itzg]

@tuxpeople you beat me to my follow-up / redaction on some of my comments and concerns 😄 . The idea of a bundled script that is exec'ed is growing on me -- I was jumping to a conclusion that the periodic scheduling would be part of the introduced behavior, but that can be driven externally. Like what you're saying, there's not really an ideal solution without a remote access interface on Bedrock's part. So co-execution makes the most sense given the constraints.

So, go for it! We can iterate and discuss more via a PR when you're ready.

[Kaiede]

Personally, I’d still vote for some sort of console shim rather than baking functionality into the container for a specific use case.

But that’s because it means I can move my container to it, which makes docker/kubernetes support optional for containers trying to do something similar (like mine), rather than fracturing stuff into different techniques.

I can’t guarantee much, but I could try to make time to whip some sort of prototype up using python or the like.

[itzg]

Thanks @Kaiede, that's where I'm still wanting to lean in order to keep the separation of concerns between containers.

Speaking of a shim, perhaps there's room for improvement in the "console shim" I have so far: https://github.com/itzg/docker-minecraft-bedrock-server/blob/master/bin/send-command . Reading out from the process' stdout seems non-trivial since I'm not sure it'll multiplex across the container's stdout and an arbitrary read from there.

[tuxpeople]

I stumbled upon this, which says Bedrock has rcon: https://help.craftingstore.net/games/minecraft-bedrock But I assume that's wrong.

@itzg Currently, I use your Helm chart to deploy a Kubernetes CronJob which starts the script within your container. The script is being added to the container via a ConfigMap. But using a Kubernetes CronJob to do kubectl exec is also something not winning a beauty contest, IMHO. But it's not our fault Bedrock can't speak with us :-) Btw. I didn't reinvent the wheel for the script, some of the logic is borrowed from here.

Curent version of my script is here. CronJob for scheduling and conversion to zip (.mcworld) is here. Note to myself: remove the for-loop around the conversion in the CronJob :-)

I also found this: https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/ which would allow moving the whole backup stuff into its own container to be added as a sidecar. But this needs elevated privileges as normally different containers in a pod have segregated process namespaces. The question would be: what about users who do not have the possibility to grant such privileges?

@Kaiede There's an official Python client library for Kubernetes (the normal CLI client is written in Go) here: https://github.com/kubernetes-client/python there's a short note about attach calls at the end of the readme (Why Exec/Attach calls doesn't work), just in case you're curious.

[Kaiede]

Speaking of a shim, perhaps there's room for improvement in the "console shim" I have so far: https://github.com/itzg/docker-minecraft-bedrock-server/blob/master/bin/send-command . Reading out from the process' stdout seems non-trivial since I'm not sure it'll multiplex across the container's stdout and an arbitrary read from there.

Yeah, going after the stdout/in file descriptors does complicate things a little. It's great for something like what you are doing here, but it's probably easier to let a process simply sit between the container's out/in and the server's out/in and expose a socket from there. Go would probably be perfect for this, and maybe it's the excuse I need to give it a try. Swift would be easy for me, but the binaries aren't nearly as portable and easy to integrate into a container right now.

I stumbled upon this, which says Bedrock has rcon: https://help.craftingstore.net/games/minecraft-bedrock But I assume that's wrong.

It's wrong. I suspect someone copy-pasted some Java content or maybe even asked GPT to write it (you never know these days).

@Kaiede There's an official Python client library for Kubernetes (the normal CLI client is written in Go) here

I was talking about writing a shim that implements rcon so that you can use it with Bedrock. The main thing having rcon built in provides you is that you get a bit more clarity on what command triggered what output. If you don't care about that, you can use rcon as a pure console.

I could directly support Kubernetes in my backup tool, but if I'm going to spend effort to help enable Kubernetes, I think I'd rather spend effort making the backup process free from Docker-isms. I have gotten reports from users that are in situations where they can't give my tool access to Docker/Kubernetes on the host in order to do the attach, but they can configure a private network between containers. So "fixing" the lack of rcon in Bedrock has wider impacts and helps more folks in the long run, based on what I've seen so far.

[tuxpeople]

Speaking of a shim, perhaps there's room for improvement in the "console shim" I have so far: https://github.com/itzg/docker-minecraft-bedrock-server/blob/master/bin/send-command . Reading out from the process' stdout seems non-trivial since I'm not sure it'll multiplex across the container's stdout and an arbitrary read from there.

Yeah, going after the stdout/in file descriptors does complicate things a little. It's great for something like what you are doing here, but it's probably easier to let a process simply sit between the container's out/in and the server's out/in and expose a socket from there. Go would probably be perfect for this, and maybe it's the excuse I need to give it a try. Swift would be easy for me, but the binaries aren't nearly as portable and easy to integrate into a container right now.

I also just access the processes stdin and stdout to do the communication, what you're describing here would be much better. I'm a big fan of go binaries when creating containers etc. Can't speak for the language though, as I'm not a programmer. I tried to learn go, but not doing programming stuff it's hard. I need something "real" to learn something, I'm not good at learning things the academic way.

@Kaiede There's an official Python client library for Kubernetes (the normal CLI client is written in Go) here

I was talking about writing a shim that implements rcon so that you can use it with Bedrock. The main thing having rcon built in provides you is that you get a bit more clarity on what command triggered what output. If you don't care about that, you can use rcon as a pure console.

Sorry, my bad. As you may have noticed, Engish is not my primary language and sometimes I missunderstand things while reading.

I could directly support Kubernetes in my backup tool, but if I'm going to spend effort to help enable Kubernetes, I think I'd rather spend effort making the backup process free from Docker-isms. I have gotten reports from users that are in situations where they can't give my tool access to Docker/Kubernetes on the host in order to do the attach, but they can configure a private network between containers. So "fixing" the lack of rcon in Bedrock has wider impacts and helps more folks in the long run, based on what I've seen so far.

I'd like the idea. Sounds good to have a runtime-agnostic, stable solution that does not need elevated privileges. Networking access between Bedrock container and a "backup" container would be easy in many environments. Whether it's Docker, Kubernetes, or anything else running containers.

[tuxpeople]

Speaking of a shim, perhaps there's room for improvement in the "console shim" I have so far: https://github.com/itzg/docker-minecraft-bedrock-server/blob/master/bin/send-command . Reading out from the process' stdout seems non-trivial since I'm not sure it'll multiplex across the container's stdout and an arbitrary read from there.

I wasn't aware of that and reinvented that in my script 😇

[itzg]

Yeah, going after the stdout/in file descriptors does complicate things a little. It's great for something like what you are doing here, but it's probably easier to let a process simply sit between the container's out/in and the server's out/in and expose a socket from there. Go would probably be perfect for this, and maybe it's the excuse I need to give it a try

Funny thing is that https://github.com/itzg/mc-server-runner (used by the Java edition image) partly exists to wrap the stdin when rcon isn't enabled. It only writes a "stop" line, but perhaps that would be an enhancement vector to add a simple REST API to enable remote stdin/stdout access.

[Kaiede]

So, I've added initial support for rcon into my backup container. It will let you use rcon for both Bedrock and Java containers, but clearly Bedrock won't work. The downside to rcon is that it doesn't support the server pushing anything to the client. It doesn't really need to, and so that's generally fine, but it does mean triggering backups on events doesn't work. Once I get a chance to let the validation run a bit, I'll merge the PR and update the documentation: Kaiede/Bedrockifier#73

Thinking on it more, I wonder if the better approach is use rlogin for the wrapper? It at least aligns properly with exposing stdout/in directly which would simplify things considerably. It's not uncommon for folks to use the server username field as the password field for rlogin, which would make it about as secure as rcon in this case (i.e. terrible). It would also make it possible to monitor the output of the server in ways that rcon doesn't allow or make sense for, meaning it would be something that makes sense to include in mc-server-runner for both bedrock/java and enable it if a password is provided to configure it?

REST is also an idea, although if we wanted to support monitoring of the service output for certain events, that's a whole thing that we could bike shed for a while.

Funny thing is that https://github.com/itzg/mc-server-runner (used by the Java edition image) partly exists to wrap the stdin when rcon isn't enabled. It only writes a "stop" line, but perhaps that would be an enhancement vector to add a simple REST API to enable remote stdin/stdout access.

Just to make sure I understand, you are suggesting maybe forking this for building out the wrapper and then folding it back via a PR once it works?

[itzg]

Just to make sure I understand, you are suggesting maybe forking this for building out the wrapper and then folding it back via a PR once it works?

No forking needed -- that one is my project also 😀. I'm saying, it could be enhanced with a REST endpoint and I'll bump the version used by the image to pick that up.

[Kaiede]

I think we are saying the same thing. The fork would be mine so I can add the functionality before creating a PR to merge it back into your repo.

I'm playing around a bit with a copy of mc-server-runner at the moment. I've got the skeleton of a "RESTCON" server implemented that uses basic auth and re-uses the RCON password (it ignores the username for now). The bit that I need to figure out at the moment is when to consider the response from stdout "complete" and send it back to the client for Bedrock. Need to experiment with that a bit locally.

I've also drafted up a small API for registering listeners with the REST API. This would let containers like mine provide matching patterns to mc-server-runner and have it issue a callback to a URL when the pattern is matched with the content of the matching line. This would allow my event-based backups to still work with this approach, unlike RCON.

[itzg]

Yes we are 😀. Awesome!

[Kaiede]

I’m bailing on the REST approach BTW. The issue is race conditions. Because I’m trying to wrap the server in/out directly, most of the time the next output is going to be what is returned from the command, but that’s not a guarantee. You can get races where some other logged event makes it to the console first. So you either need to do some sort of mapping, or risk that sometimes you’ll get something entirely different than expected and get random “failures” from issuing a command. Not a great foundation for things that need to be reliable over days and weeks, unfortunately. Ultimately, you need such functionality baked into the game itself (i.e. rcon) to avoid this.

So really some way to export the interactive session over the network is needed. I see that gliderlabs has an SSH package to make writing a server or client as easy as using net/http, which is promising. And it should be possible to disable/ignore things like port forward requests from the client to minimize the exposure of the container’s internals.

[itzg]

...and I think that's why I never started on that endeavor 😀. I think anything stream/ssh, etc based is going run into challenges of non-deterministic delineation of the output.

[itzg]

...in the grander scheme of things, I'm starting to recommend people just move away from the bedrock server software and use Java edition with Geyser for equivalent client support

https://docker-minecraft-server.readthedocs.io/en/latest/misc/examples/#bedrock-compatible-server

[Kaiede]

...and I think that's why I never started on that endeavor 😀. I think anything stream/ssh, etc based is going run into challenges of non-deterministic delineation of the output.

Which is why my service has an expect-like processing flow with timeouts.

I’m trying to also be cognizant that anything added to mc-server-runner should be light-weight. Not looking to add a ton of complexity that has to be managed/maintained. One advantage of rlogin/ssh and pushing the complexity to the client is that the server is essentially just exposing the in/out streams as a connectable PTY, and keeps things fairly clean on your end.

...in the grander scheme of things, I'm starting to recommend people just move away from the bedrock server software and use Java edition with Geyser for equivalent client support

https://docker-minecraft-server.readthedocs.io/en/latest/misc/examples/#bedrock-compatible-server

I might have to play with that, but it does still mean that rcon + Bedrockifier can’t do things like wait for your server to be empty to kick off backups which works well in closed servers with a small number of users. So I’m still interested in something like the above for Java as well, to be able to start phasing out the need for Docker integration. That would solve something like 80% of my support requests right there.

[Kaiede]

Just as an FYI, I setup a draft PR here with a couple questions since I am relatively new to Go: itzg/mc-server-runner#56

It enables SSH for remote console access on the container as part of mc-server-runner. In principle it should handle multiple simultaneous clients (including someone attached via docker as normal). It's a little over-engineered, but at the very least this should be easier to keep secure than rcon.

I'd love to be able to finish this up and add support for it to Bedrockifier's container. It'll make it so that I can deprecate direct access to docker.

[Kaiede]

Thanks for the quick review @itzg. Now we just need a means to trigger the command-line argument for mc-server-runner on the bedrock/java container and we should be good to go there.

I've made changes to my container to support connecting via SSH, which means the next release of my container will support docker, ssh, and rcon as means to talk to the Minecraft server. rcon just isn't fully featured like the other two.

I just need to resolve an issue where ssh + entry point demoter don't like each other this weekend, and that will be ready to test the whole E2E on my local Linux VM. That should help unblock Kubernetes users like @tuxpeople, folks on NAS boxes, and really just anyone where the docker method is tricky or just not feasible.

[tuxpeople]

@Kaiede let me know if I can be helpful. Time is currently limited, but I'm happy to help as good as I can.

[Kaiede]

@Kaiede let me know if I can be helpful. Time is currently limited, but I'm happy to help as good as I can.

I got lucky last night and was able to do some end-to-end testing of all this and was able to successfully backup Java and Bedrock servers, and trigger it via player logout over SSH, demonstrating that both listening and control functionality worked as expected between my backup container and the changes I made to @itzg’s containers.

What’s left is:

• Do some validation on the bedrock container changes to be more confident that adding mc-server-runner to the container doesn’t regress anything there.
• Do some validation on the backup container changes to ensure a fix I had to make for SSH to work (SSH requires carriage-return + linefeed, but rcon-cli and docker do not) doesn’t regress the docker and rcon functionality.

Once that’s done, then I should have a build on my test tag you can try with some docker-compose examples checked into the repo’s Examples folder of how to configure things for SSH. Unfortunately, translating that to Kubernetes will be up to you.

I’ll need a bit of time to update my wiki before I move the changes from test to latest, as I will be recommending people use SSH by default going forward, with docker as a fallback for a while. As docker amounts to around half of my container size, I may even look at building a “slim” tag that omits docker support entirely.

[Kaiede]

@Kaiede let me know if I can be helpful. Time is currently limited, but I'm happy to help as good as I can.

I'll just ping this one last time to say this work is now in the latest tag for itzg/minecraft-bedrock-server, itzg/minecraft-server and Kaiede/minecraft-bedrock-backup (which is increasingly poorly named these days). So it should be possible to configure something using SSH with Kubernetes.

[tuxpeople]

@Kaiede Sorry for the delay. I haven't seen it. I'm currently trying to set it up. Not yet successful, but I'm working on it.

I've some feedback/observations. My logs currently looks like this:

/usr/sbin/deluser: Only root may remove a user or group from the system.
useradd: Permission denied.
useradd: cannot lock /etc/passwd; try again later.
adduser: Only root may add a user or group to the system.
time="2024-02-24T12:28:58+01:00" level=debug msg="Using /backups to match uid and gid"
time="2024-02-24T12:28:58+01:00" level=debug msg="Resolved UID=1024 from match path"
time="2024-02-24T12:28:58+01:00" level=debug msg="Resolved GID=100 from match path"
time="2024-02-24T12:28:58+01:00" level=info msg="Skipping uid/gid change since current user is not root"
[12:28:58.496][info    ] Initializing Bedrockifier Daemon
[12:28:58.507][info    ] Configuring Bedrockifier Daemon
[12:28:58.513][info    ] Loading Configuration From: /backups/config.yml
[12:28:58.546][info    ] Configuration Loaded, Running Service...
[12:28:58.560][info    ] Host PTY Handle Opened: /dev/pts/0  (minecraft-server-insel - 6)
[12:28:58.561][info    ] Child PTY Handle Opened: /dev/pts/0 (minecraft-server-insel - 7)
[12:28:58.564][info    ] Checking for servers that might not be cleaned up
[12:28:58.569][info    ] Backup Interval: 10800.0 seconds
[12:28:58.571][info    ] Service Started Successfully.
[12:28:58.574][info    ] Starting Full Backup
[12:29:58.586][error   ] SSH connection doesn't seem to have been made properly.
[12:29:58.588][error   ] Docker process didn't start successfully, or has died
[12:29:58.589][error   ] Container minecraft-server-insel failed to backup properly
[12:29:58.589][warning ] Attempted to stop a terminal process that isn't running. (container: minecraft-server-insel, kind: ssh)
[12:29:58.589][info    ] Performing Trim Jobs
[12:29:58.729][info    ] Full Backup Completed

Last line says Full Backup Completed but it hasn't done one (because of the errors earlier in the log, which I'm working on). I assume the last line should be something like Backup failed.