Asking advice on our current infrastructure

[nckenn]

Overview

Currently, we are using Proxmox for our virtual environment. I have installed 4 different virtual machines with Docker Swarm(manager) on each of them. 1 is used for the portainer(I want to add all of our docker hosts to one portainer) only, and 3 is used for the jamstack application. The Jamstack app is pretty straightforward with 1 stack and consists of 4 services running in each virtual machine.

Jamstack App

• Frontend App
• Backend API
• Caddy
• Database

Now on our portainer panel we have 4 environments :

• primary
• jamstack_app1 (running only 1 stack)
• jamstack_app2 (running only 1 stack)
• jamstack_app3 (running only 1 stack)

Question

Is our current approach, that of having each jamstack as a separate stack on a different virtual machine, the correct one? Or should we put our 3 jamstack applications into 1 virtual machine with Docker Swarm and run 3 stacks on it and add the other 2 virtual machines as a worker? 

[BretFisher]

Hey there.

• You should only have odd numbers of Swarm Managers. I recommend 3 for HA. All other nodes would just be workers. No need to treat Managers special in this small environment.
• One of the points of orchestrators is that you shouldn't need to manage "which VMs run which workloads". Just run the three stacks, set memory/cpu reservations and healthchecks in YAML, and let Swarm assign them to nodes.
• In your case of this small environment, without knowing more, I'd just have three VMs, all Swarm Managers, and deploy portainer and app stacks without controlling which node they run on. If you run out of resources, it's up to you if you grow vertically (bigger VMs) or horizontally (more VMs).

[nckenn]

Hey there.

• You should only have odd numbers of Swarm Managers. I recommend 3 for HA. All other nodes would just be workers. No need to treat Managers special in this small environment.
• One of the points of orchestrators is that you shouldn't need to manage "which VMs run which workloads". Just run the three stacks, set memory/cpu reservations and healthchecks in YAML, and let Swarm assign them to nodes.
• In your case of this small environment, without knowing more, I'd just have three VMs, all Swarm Managers, and deploy portainer and app stacks without controlling which node they run on. If you run out of resources, it's up to you if you grow vertically (bigger VMs) or horizontally (more VMs).

Thanks, I really appreciate your answer!

Lastly, should the manager and workers have the same resources(memory/cpu/storage)?

[BretFisher]

Not really in a small cluster (under 10 nodes and 500 workloads)

[nckenn]

Thanks!

What persistent storage do you recommend?

[BretFisher]

Due to Swarm's decline in popularity, there are not many options outside node local. Generally I advise that in 2022 if you need persistent storage for your container workloads, it's time to move to K8s. Here's some of the remaining Swarm options:

GlusterFS
https://thenewstack.io/tutorial-create-a-docker-swarm-with-persistent-storage-using-glusterfs/
https://florianmuller.com/setup-a-scalable-high-availability-glusterfs-network-filesystem-with-docker-swarm-on-ubuntu-server-20-04-lts

PortWorx
https://docs.portworx.com/install-portworx/install-with-other/docker/swarm/

There's Ceph and other ways to use off-server storage, but I don't have a good list for 2022 of projects that are still maintained. I'd google "docker swarm persistent storage" and filter for only 2021 and newer, and see if you can find a solution that fits for you.

One hope is that the next major version of Docker Engine has CSI support for Docker and Swarm, which will help bring Swarm into the Kubernetes world of 3rd party storage solutions, but I have no news on when/if that'll happen. (PR's have been accepted but no idea when a release hits)