Kubernetes support

[alexevladgabriel]

This thread is intended to discuss benefits & drawbacks of having it implemented.

Are there specific aspects of Kubernetes functionality you believe would be the most beneficial for the project to support?
What are the pain points or gaps that tighter integration with Kubernetes could solve for your use cases?

[PseudoResonance]

One of the things I've been thinking about is logging. I use Kubernetes, and all my applications log to standard logs, so in Rancher, Lens, kubectl, whatever I can easily check the logs. Additionally, I have Vector collect and push those logs to a log aggregator, so that I have a nice web UI with dropdowns to choose what service I want to view logs for, etc.

With the current file logging, it can still be done, but I would need to setup a dedicated Vector instance to pull from the Pelican PV, do some custom labeling to fit with the other apps, and then aggregate it. I personally believe it'd make more sense if Pelican was built with Docker in mind, not necessarily Kubernetes, but if it's built to follow Docker standards, then it should be easy to integrate into a K8s solution. By this, I am specifically referring to again standard logging instead of files, and additionally, a more segmented approach, instead of a single monolithic container helps, as each process' logs will be divided up and labeled by the container's name automatically in K8s and Docker.

I feel like these 2 aspects are the things that most greatly affect how suitable Pelican is for running on Kubernetes, and I've been wondering why the current approach is being taken instead?

[parkervcp]

We don't log to a file. We just use dockers standard logging feature which outputs to a file.

The log config hasn't been used since 0.7 and is a left over from that.

[PseudoResonance]

It's not a direct reply to my comment, but it sounds like it might be in response to it?

Unless it's changed in the past 2 weeks since I was working on #932, Laravel logs are not output to standard logging, but are logged in storage/logs. Additionally, Supervisor logs to /var/www/html/storage/logs/supervisord/supervisord.log, which is also in that log folder. The Dockerfile is setup to have a volume mounted there to collect all the log files in one place. When I would run the container in testing, I would get effectively 0 useful logs and I had to check the logging folder for everything. That's what I am referring to.

I'm so sorry if I misunderstood something though and this wasn't a reply to my question or anything!

[parkervcp]

I was discussing the game server logging and not the panels.

[PseudoResonance]

I was discussing the game server logging and not the panels.

Ahh, I'm sorry, thank you for clarifying it for me!

[KittyKatt]

Personally, the benefit for Kubernetes comes from the scalability and resiliency. The architecture that I'd like to see (take note that this is very idealized and extremely high level without consideration for the work involved to get there) would look something like this:

• A deployment for the panel itself. This could 1 to N replicas. In my case, I'd like 3-5 replicas spaced across my worker nodes to ensure the panel is always available even through node maintenance/downtime.
• An operator that would actually orchestrate the game server deployments themselves. This would fill in a lot of what Wings currently does. Game servers could have potentially one or many replicas, depending on resources and game server type. The option to have a PVC attached to the pod hosting the game server would at least allow me resiliency when a node goes down because I'm currently using NFS-attached storage for my main PV StorageClass in my cluster. If a node were to go down for any reason, the game server pod could just pop back up on any worker node that has access to that PV.
• A plus would be reading Kubernetes node annotations/labels to determine node size/resources. This could be a dropdown selection when creating a server so if I had something particularly resource hungry, I could choose a node labelled "big". If I had a small server that probably won't take up much resources, I'd choose one of the nodes labelled "small".

This is most of what I'm looking for out of Kubernetes support in Pelican. Nearly all of my infrastructure is on my Kubernetes cluster at this point, as well, so not having to have a whole separate box/VM just for Pelican. No project (at least none that are active) that I've found currently integrates both the panel and operator functionality together to give a really good, seamless Kubernetes solution.

Hopefully that helps to outline at least one benefit to tighter integration with k8s.

[richard-callis]

The main benefit from Kubernetes that I want to take advantage is distributing the workload across the cluster, versus manually creating VMs as nodes and assigning multiple servers to one node. Plus the added benefit from longhorn managing the volumes across the cluster, it works with my already established rsync backup strategy. Claude Code has been a godsend, it has helped me transition from a traditional docker setup strategy on VMs to a distributed workload on 6 Raspberry PIs, and 2 AMD machines.

Right now the only viable path to running Wings on K8s is Docker-in-Docker (DinD) — a DinD sidecar provides a private Docker daemon, and Wings connects to it via tcp://localhost:2375. This works, but it comes with significant costs:

• Security: Both the DinD sidecar and Wings itself must run with privileged: true and runAsUser: 0. That's a full container escape surface on every game server pod. For home labs this is acceptable; for anyone running shared or multi-tenant infrastructure, it's a blocker.
• Resource visibility: Game containers launched inside DinD are completely invisible to Kubernetes. K8s cannot enforce resource limits, won't kill an OOM game process, and has no way to surface metrics. The node can be silently starved without the scheduler knowing.
• Cold start penalty: Docker image layers live in an emptyDir volume — ephemeral per pod. When Kubernetes reschedules a Wings pod to a different node (routine maintenance, node failure, etc.), DinD must re-pull all game container images from scratch. For large game servers this can mean 30–60+ minute downtime windows.
• Update downtime: Because game data lives on a ReadWriteOnce PVC, rolling updates are impossible — strategy: Recreate is
  required, meaning every Wings update drops players.

Instead of Wings managing game containers via Docker, it would speak the Kubernetes API directly — spawning game servers as Pods (or Jobs) in a namespace, with:

• No privileged: true required — game Pods get standard K8s security contexts
• Full resource enforcement — CPU/memory limits on game containers visible to the scheduler
• Native image caching — K8s node image cache survives pod rescheduling; no re-pull on restart
• Proper lifecycle management — K8s handles restarts, health probes, graceful termination
• Metrics — game Pod resource usage flows naturally into Prometheus/Grafana
• Rolling updates — Wings daemon updates without touching game Pods at all

Below is a full manifest set if useful — currently running 7 Wings instances (ARK SA, Satisfactory, Valheim, Enshrouded, 7D2D, RoR2, Moria) all on K8s with this pattern. It's stable, but the DinD privileged requirement and cold-start problem are the two things that would most benefit from a native solution.

manifests.txt

[lancepioch]

Not planned.

[KittyKatt]

So I take that to mean that all of this work is going to be dropped? pelican/wings#193

[Claiyc]

I believe it would be a mistake to just close this topic as "not planned"... I'm pretty confident to say that EVERYTHING in the "(mass) hosted services" world is going towards kubernetes. Even for the gameserver niche. Kubernetes just has so many advantages that no provider of gameservers (let it be a hosting company or just some guild/org that happens to run a fleet of gameservers) would want to miss... Portability, declarativity, expandability, isolation, observability, standardization for everything...

Tbf, I am currently having a bit of a look at pelican/wings#193 and I believe I am not the biggest fan of the architecture. It certainly is a working solution and some nice work but for me it's taking a couple of shortcuts and is doing some things that are just a bit of an antipattern for kubernetes imo (Imperativity over declarativity, multiple spofs - eg. the shared RWX PVC (which is a whole topic for itself 😄 ), keeping wings as a central daemon for multiple servers)

Doing a bit of brainstorming right now & talking to the AI for figuring out a fitting architecture / plan on how one could bring pelican to k8s 😃

[notAreYouScared]

This has been re-opened at @parkervcp request

[KittyKatt]

I believe it would be a mistake to just close this topic as "not planned"... I'm pretty confident to say that EVERYTHING in the "(mass) hosted services" world is going towards kubernetes. Even for the gameserver niche. Kubernetes just has so many advantages that no provider of gameservers (let it be a hosting company or just some guild/org that happens to run a fleet of gameservers) would want to miss... Portability, declarativity, expandability, isolation, observability, standardization for everything...

Tbf, I am currently having a bit of a look at pelican/wings#193 and I believe I am not the biggest fan of the architecture. It certainly is a working solution and some nice work but for me it's taking a couple of shortcuts and is doing some things that are just a bit of an antipattern for kubernetes imo (Imperativity over declarativity, multiple spofs - eg. the shared RWX PVC (which is a whole topic for itself 😄 ), keeping wings as a central daemon for multiple servers)

Doing a bit of brainstorming right now & talking to the AI for figuring out a fitting architecture / plan on how one could bring pelican to k8s 😃

Heavy agree here on all points. I've actually been using that PR locally as a test of some things and it works but I also have some concerns about architecture.

[Claiyc]

Alright so I was fiddeling around a bit / did a bit of thinking and I think (while heavily relying on AI 😄) it produced an architecture idea that I am quite happy with: https://github.com/Claiyc/pelican-k8s

To clarify

• Running the panel on k8s is not the scope here (different, much smaller problem)
• This is about the "pelican backend", eg. wings, how gameservers are launched, and how the panel communicates with the backend / servers

TLDR:

• imo, the core "problem" for getting pelican on k8s is wings. In a cloudnative environment like k8s, wings tries to do "too much" / has too many responsibilities. Still we don't want to completely rewrite wings as it does so many things right and the panel knows exactly how to work with it (I don't want to touch the panel at all).
  Therefore I propose:
• Make pelican-k8s a separate project/repo
• Add 2 new services: pelican-operator and pelican-gateway
• pelican-gateway fully implements the wings api (for the panel, the pelican-gateway is basically the same as a wings instance)
• pelican-operator implements a new CRD "GameServer" (which holds required settings, desired runtime state, status fields, conditions, ...). It constantly watches these CRs and reconciles the gameserver pod to achieve desired state:

• Each gameserver consists of one pod running the game container (slightly modified egg image -> shim inject via init container) and a native sidecar which is running wings (a slightly modified wings, called agent in the architecture document -> goal is to keep this upstream compatible).
• There is also one dedicated PVC for each gs which holds the exact same filestructure that wings is used to today (wings pathlogic works unchanged)
• Wings is running exclusively in single-server mode (new "wings agent" cobra command)

• These single-server wings instances never communicate directly with the panel - instead all typical requests executed from the panel go to the pelican-gateway and are relayed to the wing sidecar the request belongs to. Example:

The full document contains a lot more details. It definitely isn't a small project but I think this is a clean option. And the good thing is, the single-server / agent mode is the only real point of conflict / required change with pelican as the upstream. Keeping the Gateway API def up2date would be annoying though and is a lot of work.

[Claiyc]

Why would you even need a Wings sidecar? If the operator handles reconciliation, the "gateway" can just create the resources with the desired configuration and let the operator figure it out. As for server controls, again update resource, let the operator figure it out.

Only thing that's a little complex is the console, but the gateway can just pull the logs, and send commands to be executed in the container through a little init wrapper or something. The gateway should be completely stateless too, and you can load balance it easily if desired as long as you use some kind of sticky connections or otherwise consistent server selection method.

But regardless, the init wrapper/shim/whatever should be all you need in the pod, and can be incredibly minimal.

As for other ideas, consider looking at how Agones solves it.

Your right, I think it actually doesn't make sense to route something like a power request via the gateway directly to the sidecar - the design already describes patching the desired spec.power in the CR - letting the operator reconcile that should be sufficient. And yes the gateway itself should definitely remain completely stateless.

The wings sidecar still has quite a few reasons to exist though. Also we can't do everything via state definition in the CR, some things are just not "state" that can be reconciled (SFTP, send command, files, stats, ..) - so some things still need to be routed through the gateway. I think it would be very nice to be able to just reuse wings for all of that. Thinking about it, also for power commands you still want to keep the wings sidecar in the loop (think about configured stop modes etc) - I don't want to rewrite all the logic directly in the operator. What should be done though is a channel directly from the operator to the sidecar for exactly these "process" rules (eg. power but also things like "SyncWithConfiguration") -> edit: by this I mean the stock wings http api with a per server token, so operator really just calls eg. POST /power exactly as the panel would call wings

I'll update the design...

Edit2: I just looked at Agones, they actually also run a SDK sidecar next to every gameserver.. My idea is not that far off:smile:

Ah and for the console; shim only is an issue here because who owns stdin? pods/attach would be the only option if I'm not mistaken and I think we don't want that cost going through the api server and kubelet for that...
I propose; shim holds the PTY and exposes it on a unix socket and the wings sidecar attaches to that socket the same way it attaches to docker today. And the browsers ws is wings stock handler proxied through the gateway. No api server no kubelet no attach permissions and we get Done-string detection, stop-command detection, crash handling, ... all remains wings code then

[Claiyc]

I'm not certain I entirely agree with it being a separate project. There's going to be a lot of overlap in both code bases and purely from a maintenance perspective, that can be a pain to work with.

I think what you've got there is certainly interesting, though.

You're right and I actually don't want to completely separate it I think...
First of all (after catching some sleep and thinking about it) I think wings really isn't the painpoint, as mentioned we can basically reuse almost everything and just a couple of minor changes (opt-in) would be needed directly in the wings upstream.

The main painpoint (and I believe this is what you have in mind aswell) will probably be the gateway and all of the api definitions, types etc. But the good thing here is that wings already has all of those as Go modules right - so if we write the gateway in Go, we can just import them and we basically automatically get bumps eg. if the upstream adds a field somewhere. This way the Gateway doesn't own the api definitions anymore. Properly configure a renovate bot and we should be good to go.

I'd still put all of this in a separate repo though and just pin the wings version

[KittyKatt]

I'm not certain I entirely agree with it being a separate project. There's going to be a lot of overlap in both code bases and purely from a maintenance perspective, that can be a pain to work with.

I think what you've got there is certainly interesting, though.

You're right and I actually don't want to completely separate it I think...
First of all (after catching some sleep and thinking about it) I think wings really isn't the painpoint, as mentioned we can basically reuse almost everything and just a couple of minor changes (opt-in) would be needed directly in the wings upstream.

The main painpoint (and I believe this is what you have in mind aswell) will probably be the gateway and all of the api definitions, types etc. But the good thing here is that wings already has all of those as Go modules right - so if we write the gateway in Go, we can just import them and we basically automatically get bumps eg. if the upstream adds a field somewhere. This way the Gateway doesn't own the api definitions anymore. Properly configure a renovate bot and we should be good to go.

I'd still put all of this in a separate repo though and just pin the wings version

Yeah, you've already got a really solid and stable API contract established between the panel and Wings. No need to reinvent the wheel.

I think that, in an ideal world, the changes to panel that need to happen to make this all work are going to be extremely minimal. As far as the panel and the end user is concerned, that should all be almost entirely transparent. It's still just Wings, in the end.

[Claiyc]

So…
I may or may not have been a little bit bored yesterday and basically implemented the entire thing; https://github.com/Claiyc/pelican-k8s/tree/master

0 changes to the panel were necessary
some very few opt-in changes were necessary in wings (pelican/wings@main...Claiyc:wings:pelican-k8s-hooks)

I think it turned out really cool 😄 Let me know if you had some time to take a look at it and what you think

[KittyKatt]

I'll give it a spin likely this weekend.