Crash when using AutoPause in the game container

[wolfen351]

I've been struggling with this on and off now for a bit, and I've finally narrowed the crashes of this container to the fact that the main game container is "paused". Could we add the rcon connection port to the wake up sequence? So the game can be woken up for backups?

[wolfen351]

Here are some logs to show what I mean:

Game Container:

[23:49:02] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 11949ms or 238 ticks behind
[Autopause loop] No client connected since startup / knocked - stopping
[23:53:34] [RCON Listener #1/INFO]: Thread RCON Client /0:0:0:0:0:0:0:1 started
[23:53:34] [Server thread/INFO]: [Rcon: Saved the game]
[23:53:34] [RCON Client /0:0:0:0:0:0:0:1 #5/INFO]: Thread RCON Client /0:0:0:0:0:0:0:1 shutting down
[2022-05-04T23:53:34+00:00] [Autopause] Pausing Java process

Backup Container:

2022-05-04T23:47:20+0000 INFO waiting initial delay of 1m...
2022-05-04T23:48:20+0000 INFO waiting for rcon readiness...
2022-05-04T23:48:20+0000 ERROR Unable to execute rcon-cli save-on - try 0/5. Retrying in 10s
2022-05-04T23:48:20+0000 ERROR Failure reason: 2022/05/04 23:48:20 Failed to connect to RCON serverdial tcp [::1]:25575: connect: connection refused
2022-05-04T23:48:30+0000 ERROR Unable to execute rcon-cli save-on - try 1/5. Retrying in 10s
2022-05-04T23:48:31+0000 ERROR Failure reason: 2022/05/04 23:48:30 Failed to connect to RCON serverdial tcp [::1]:25575: connect: connection refused
2022-05-04T23:48:41+0000 INFO Command executed successfully rcon-cli save-on
2022-05-04T23:48:41+0000 INFO Command executed successfully rcon-cli save-off
2022-05-04T23:49:00+0000 INFO Command executed successfully rcon-cli save-all flush
2022-05-04T23:49:00+0000 INFO Command executed successfully sync
2022-05-04T23:49:00+0000 INFO Backing up content in /data to /backups/world-20220504-234900.tgz
tar: ./world: file changed as we read it
2022-05-04T23:57:40+0000 WARN tar exited with code 1. Ignoring
2022-05-04T23:57:50+0000 ERROR Unable to execute rcon-cli save-on - try 0/5. Retrying in 10s
2022-05-04T23:57:50+0000 ERROR Failure reason: 2022/05/04 23:57:50 Failed to connect to RCON serverread tcp [::1]:44092->[::1]:25575: i/o timeout
2022-05-04T23:58:10+0000 ERROR Unable to execute rcon-cli save-on - try 1/5. Retrying in 10s
2022-05-04T23:58:10+0000 ERROR Failure reason: 2022/05/04 23:58:10 Failed to connect to RCON serverread tcp [::1]:44094->[::1]:25575: i/o timeout
2022-05-04T23:58:30+0000 ERROR Unable to execute rcon-cli save-on - try 2/5. Retrying in 10s
2022-05-04T23:58:30+0000 ERROR Failure reason: 2022/05/04 23:58:30 Failed to connect to RCON serverread tcp [::1]:44096->[::1]:25575: i/o timeout
2022-05-04T23:58:50+0000 ERROR Unable to execute rcon-cli save-on - try 3/5. Retrying in 10s
2022-05-04T23:58:50+0000 ERROR Failure reason: 2022/05/04 23:58:50 Failed to connect to RCON serverread tcp [::1]:44098->[::1]:25575: i/o timeout
2022-05-04T23:59:10+0000 ERROR Unable to execute rcon-cli save-on - try 4/5. Retrying in 10s
2022-05-04T23:59:10+0000 ERROR Failure reason: 2022/05/04 23:59:10 Failed to connect to RCON serverread tcp [::1]:44100->[::1]:25575: i/o timeout
2022-05-04T23:59:30+0000 ERROR Unable to execute rcon-cli save-on - try 5/5. Retrying in 10s
2022-05-04T23:59:30+0000 ERROR Failure reason: 2022/05/04 23:59:30 Failed to connect to RCON serverread tcp [::1]:44102->[::1]:25575: i/o timeout
2022-05-04T23:59:50+0000 ERROR Unable to execute rcon-cli save-on - try 0/5. Retrying in 5s
2022-05-04T23:59:50+0000 ERROR Failure reason: 2022/05/04 23:59:50 Failed to connect to RCON serverread tcp [::1]:44106->[::1]:25575: i/o timeout
2022-05-05T00:00:05+0000 ERROR Unable to execute rcon-cli save-on - try 1/5. Retrying in 5s
2022-05-05T00:00:05+0000 ERROR Failure reason: 2022/05/05 00:00:05 Failed to connect to RCON serverread tcp [::1]:44108->[::1]:25575: i/o timeout
2022-05-05T00:00:20+0000 ERROR Unable to execute rcon-cli save-on - try 2/5. Retrying in 5s
2022-05-05T00:00:20+0000 ERROR Failure reason: 2022/05/05 00:00:20 Failed to connect to RCON serverread tcp [::1]:44110->[::1]:25575: i/o timeout
2022-05-05T00:00:35+0000 ERROR Unable to execute rcon-cli save-on - try 3/5. Retrying in 5s
2022-05-05T00:00:35+0000 ERROR Failure reason: 2022/05/05 00:00:35 Failed to connect to RCON serverread tcp [::1]:44112->[::1]:25575: i/o timeout
2022-05-05T00:00:50+0000 ERROR Unable to execute rcon-cli save-on - try 4/5. Retrying in 5s
2022-05-05T00:00:50+0000 ERROR Failure reason: 2022/05/05 00:00:50 Failed to connect to RCON serverread tcp [::1]:44114->[::1]:25575: i/o timeout
2022-05-05T00:01:05+0000 ERROR Unable to execute rcon-cli save-on - try 5/5. Retrying in 5s
2022-05-05T00:01:05+0000 ERROR Failure reason: 2022/05/05 00:01:05 Failed to connect to RCON serverread tcp [::1]:44116->[::1]:25575: i/o timeout```

[wolfen351]

I did some more digging, and it appears the rcon port IS configured for knocking:

root@minecraft-freecrafting-game-deployment-58dc585565-ctgx5:/# cat /tmp/knockd-config.cfg
[options]
 logfile = /dev/null
[unpauseMCServer-server]
 sequence = 25565
 seq_timeout = 1
 command = /autopause/resume.sh
 tcpflags = syn
[unpauseMCServer-rcon]
 sequence = 25575
 seq_timeout = 1
 command = /autopause/resume.sh
 tcpflags = syn
[unpauseMCServer-bedrock]
 sequence = 19132:udp
 command = /autopause/resume.sh

However the backup container still crashes instead of making the backup :/

[itzg]

Can you provide your container configs, such as compose file? It looks like you went with the option to run the two containers in the same networking namespace. That's the approach I recommend; however, knockd isn't listening on the loopback interface.

Knockd support was a contributed feature so I would need to research if that can be configured to listen to multiple/all interfaces.

[wolfen351]

Thanks, that was the piece of info I needed to fix my issue, much appreciated! I deployed them in the same networking namespace, however I did it in k8s and not docker compose. Knowing that it does not listen on lo0, I made a k8s service to handle the inter container communications of RCON and then used the internal k8s dns entry for the communication. This resolved the problem for me.

The changes to fix this issue were exposing the RCON port in the load balancer service, and then using the service name in the RCON_HOST env variable.

For reference, here is my working config:

apiVersion: v1
kind: Service
metadata:
  name: minecraft-freecrafting-game-service
spec:
  type: LoadBalancer
  ports:
    - name: game
      port: 25565
    - name: rcon
      port: 25575
  selector:
    app: minecraft-freecrafting-game
---
apiVersion: v1
kind: Service
metadata:
  name: minecraft-freecrafting-game-voice-service
spec:
  type: LoadBalancer
  ports:
    - name: voice
      port: 24454
      protocol: UDP
  selector:
    app: minecraft-freecrafting-game
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minecraft-freecrafting-game-deployment
  labels:
    app: minecraft-freecrafting-game
spec:
  replicas: 1
  selector:
    matchLabels:
      app: minecraft-freecrafting-game
  template:
    metadata:
      labels:
        app: minecraft-freecrafting-game
    spec:
      imagePullSecrets: 
      - name: registry-credentials
      # Add the server as an NFS volume for the pod
      volumes:
        - name: nfs-volume
          nfs: 
            # URL for the NFS server
            server: xxxx
            path: /k3s/minecraft-freecrafting
        - name: nfs-volume-bk
          nfs: 
            server: xxxx
            path: /k3s/minecraft-freecrafting-backups

      containers:
      - name: minecraft-freecrafting-game
        image: itzg/minecraft-server:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 25575 # rcon
        - containerPort: 25565 # game
        - containerPort: 24454 # voicechat
          protocol: UDP
        # Mount the NFS volume in the container
        volumeMounts:
          - name: nfs-volume
            mountPath: /data
        env:
            - name: EULA
              value: "TRUE"
            - name: TYPE
              value: "FABRIC"
            - name: MEMORY
              value: "1500M"
            - name: VERSION
              value: "1.18.1"
            - name: ENABLE_AUTOPAUSE
              value: "TRUE"
            - name: AUTOPAUSE_TIMEOUT_EST
              value: "300"
            - name: AUTOPAUSE_TIMEOUT_INIT
              value: "300"
      - name: minecraft-freecrafting-backup
        image: itzg/mc-backup
        imagePullPolicy: Always
        # Mount the NFS volume in the container
        volumeMounts:
          - name: nfs-volume
            mountPath: /data
          - name: nfs-volume-bk
            mountPath: /backups
        env:
            - name: RCON_HOST
              value: minecraft-freecrafting-game-service
            - name: BACKUP_INTERVAL
              value: "1d"
            - name: INITIAL_DELAY
              value: "5h"
            - name: PRUNE_BACKUPS_DAYS
              value: "14"

[wolfen351]

If you'd like, you could add that to the documentation on how to deploy these 2 containers together into a kubernetes server

[itzg]

Thanks. Yes, that would be a good note to add to the docs.