This repository contains minimal examples for Fedora CoreOS configurations and scripts that help deploy them on VMWare vSphere (or Fusion).
The base setup contains an etcd cluster consisting of three members, a Traefik edge router, and a hello-world application each in their own Fedora CoreOS VM. Service 'discovery' is done by pushing service information into the etcd cluster on VM startup which is then read by Traefik. On VM shutdown the information is deleted from etcd and Traefik stops serving traffic there.
Each VM will be provisioned with SSH certificates by default, the configuration can be extended with client certificates as outlined here.
The shell script template used for the generator scripts is the MIT licensed script-template.sh by Maciej Radzikowski.
- bash scripting environment
- butane Fedora CoreOS configuration converter
- curl curl to download files off of the Internet
- govc vSphere client software
- gpg OpenPGP implementation for signature checks
- jq JSON parser
- ssh SSH implementation
A way to provide TLS and SSH certificates. You can use simple-ca to get started quickly and without modifying the scripts if you simply want to play around.
GNU compatible base64
is required, for example available via:
brew info coreutils
# Documentation and Caveats...
If you need to use these commands with their normal names, you can add a "gnubin" directory to your PATH with:
PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
# ...
This means that base64 should be available at /usr/local/opt/coreutils/libexec/gnubin/base64
:
/usr/local/opt/coreutils/libexec/gnubin/base64 --version
base64 (GNU coreutils) 9.0
If you don't want to put this permanently onto your path you can simply prefix any ./scripts/*.sh
invocations with
PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
, i.e.:
PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH" ./scripts/deploy.sh
- For a Bash based environment it is easiest to use Git for Windows
- make sure to select the Windows Terminal Profile Fragment during installation for a better user experience later on
- also make sure to use the Windows Secure Channel library if you plan on rolling out certificates to your machine otherwise you'll have to manually patch the bundled certificate bundle
- make sure to use "Checkout as-is, commit as-is" to not break line endings of existing files
- this includes a compatible curl, base64, GPG, and OpenSSH version by default
- Instead of using the MinTTY console installed by Git consider use Windows Terminal instead for a better user experience
- For a simple installation consider using winget
- alternatively, you have to manually add butane, govc, and jq to your
$PATH
environment variable
- alternatively, you have to manually add butane, govc, and jq to your
The Butane configuration files contain pieces for the following tools alongside the actual service configurations:
- Docker
- NetworkManager, NetworkManager CLI Documentation
- rpm-ostree, rpm-ostree manpage
- SSH
- Systemd Unit
- Zincati
Note: the VM configs contain references to additional disks in the storage
section – they have to be removed in
case you want to launch on VMWare Fusion (or Workstation). The OVA conversion doesn't account for them.
- Deploy the etcd cluster after replacing the
CLUSTER_TOKEN_PLACEHOLDER
in thedocker-app.service
file located in theetcd/includes
directory. - Once provisioning is finished and the VM started the second time (required for installing VMWare tools), log into
each member machine and change the cluster state from
new
toexisting
in the systemd default environment found in thedocker-app.service
unit file. - In the meanwhile you can deploy Traefik
- The base infrastructure should now be in place to add additional services, such as the
hello-world
example.
Because of the dynamic nature of the SSH host key pairs and certificates the passphrase for the root key pair and the
path to the private key has to be provided either as environment variable (SIMPLE_CA_SSH_PASSWORD
) or as inline shell
parameter (-i
).
The following command will generate an Ignition configuration using the TLS
certificates provided by a simple-ca
based certificate authority and the aforementioned root key pair for the SSH host
certificates for the hello-world
Butane configuration. During the script run the latest stable CoreOS version will be
downloaded, verified, and uploaded to the default vSphere/vCenter template library. Once done, the template item will be
deployed as hello-world
VM with the hardware specification derived from the resources.json
and the Ignition
configuration applied. In the end the VM will be powered on and start the provisioning process.
export GOVC_URL='vcenter.example.local'
export GOVC_USERNAME='username@vsphere.local'
export GOVC_PASSWORD='password'
./scripts/deploy.sh -s stable -d ~/Downloads/coreos/ \
-n hello-world -b ./hello-world/hello-world.bu.yaml \
-t /Volumes/simple-ca/data/intermediate-ca-name \
-g '/Volumes/simple-ca/data/ssh-ca/ca' -i 'sshpassword' \
-o
Don't forget to read the documentation via --help
to see what other flags and settings can be specified.
Simply deleting VMs via the vCenter/vSphere management UI will cause all attached disks to be deleted, including ones
you may want to keep. There is no confirmation or selection dialog to prevent this. To prevent this an undeploy.sh
script was added that unmounts the non-system disks after a clean shutdown of the VM and allows you to reuse them. This
is handy during a redeployment of an "existing" VM:
export GOVC_URL='vcenter.example.local'
export GOVC_USERNAME='username@vsphere.local'
export GOVC_PASSWORD='password'
# dry-run
./scripts/undeploy.sh -n fcos-hello-world
# List of resources to be removed
#...
# apply removal of VM but keep data volumes
./scripts/undeploy.sh -n fcos-hello-world -a
Running a deploy.sh
run afterwards for hello-world
will reattach the existing disks.
If you want to remove all data, either do so via the vSphere/vCenter UI or run remove.sh
. This will remove all VM
related information including all disks.
export GOVC_URL='vcenter.example.local'
export GOVC_USERNAME='username@vsphere.local'
export GOVC_PASSWORD='password'
# dry-run
./scripts/remove.sh -n fcos-hello-world
# List of resources to be removed
# ...
# apply removal of VM and data volumes
./scripts/remove.sh -n fcos-hello-world -a