infra-shared

Software Defined Git Operated Infrastructure

Reusables of a learning project by rewriting parts of my home infrastructure as

a Pulumi and Fedora Coreos based Gitops Project in Python.

• See safe for usage in an example project

Quick start

create a base project, lock and install build requirements, install and configure a simulation of the targets

mkdir -p example; cd example; git init
git submodule add https://github.com/wuxxin/infra-shared.git infra
infra/scripts/create_skeleton.sh --yes
make sim-up

Congratulations!

You have just created two TLS Certificates and an SSH Keypair in a very fancy way!

See the examples for code of what else can be done with it

Features

• Fedora-CoreOS Linux - updating, minimal, monolithic, container-focused operating system
  • Setup: Bootstrap and Reconfiguration of CoreOS with Jinja templated butane files
  • Reconfiguration: update-system-config*
    • Fast (~4s) reconfiguration using saltstack and butane to salt translation
  • Single Container: podman-systemd.unit
    • container* - run systemd container units using podman-quadlet
  • Compose Container: compose.yml
    • compose* - run multi-container applications defined using a compose file
  • nSpawn OS-Container: systemd-nspawn
    • nspawn* - run any linux OS in a light-weight system container
  • tls/http/web FrontEnd: traefik
    • using container, compose and nspawn labels for dynamic configuration
  • DNS Resolver: unbound
    • using container for local DNSSEC capable recursive DNS-Resolver
• TLS Certificate-Authority, TLS Certificates and SSH-Certificates
• SSH copy/deploy/execute functions, local and remote Salt-Call
• serve configuration HTTPS payloads, request a port forwarding
• write image to removable storage specified by serial_number
• build Embedded-OS Images and IOT Images
  • Raspberry Extras - U-Boot and UEFI Bios Files for Rpi3 and Rpi4
  • Openwrt Linux - Network Device Distribution for Router and other network devices

Technologies

Need to know technologies (to write Deployment and Docs):

• Basic Knowledge of Python, Yaml, Jinja, Systemd Service, Containerfile, Markdown

Advanced functionality available with knowledge of:

• Pulumi, Butane, more Systemd, Fcos, Saltstack, Podman, compose.yml, makefile, Pipfile, libvirt, Bash, Mkdocs, Mermaid, Jupyter Notebooks

Provision can be run on Arch Linux, Manjaro Linux or as Container Image.

Tools used

• pulumi - imperativ infrastructure delaration using python
• fcos - Fedora-CoreOS, minimal OS with clevis (sss,tang,tpm) storage unlock
• butane - create fcos ignition configs using jinja enhanced butane yaml
• systemd - service, socker, path, timer, nspawn machine container
• podman - build Container images, run Container using quadlet systemd container
• saltstack
  • local build environments and local services
  • remote fcos config update using butane to saltstack translation and execution
• mkdocs - documentation using markdown and mermaid
• libvirt - simulation of machines using the virtualization api supporting qemu and kvm
• tang - server used for getting a key shard for unattended encrypted storage unlock on boot
• mkosi - build nspawn OS container images
• age - ssh keys based encryption of production files and pulumi master password
• pipenv - virtualenv management using Pipfile and Pipfile.lock

Usage

List available Makefile targets/commands

make

Bootstrap skeleton files to a new repo

• from current directory, eg. pwd=~/code

project_name=example
current_dir=$(pwd)
project_dir=${current_dir}/${project_name}
mkdir -p ${project_dir}
cd ${project_dir}
git init
git submodule add https://github.com/wuxxin/infra-shared.git infra
infra/create_skeleton.sh --yes

• create_skeleton.sh creates default dirs and files in the project_dir
  • use cat infra/create_skeleton.sh to inspect script before running it
  • directories created:
    • docs, state, target with an empty .gitkeep file inside
  • files created:
    • README.md, __main__.py, Pulumi.yaml, Makefile, Pipfile
    • config-template.yaml, .gitignore, mkdocs.yml, empty authorized_keys

Install build requirements

• on arch or manjaro linux

make install-requirements

• on other linux, use a provision container.

This needs podman or docker already installed on host.

For the simulation environment with libvirt the host system must also have a configured libvirt.

# Either: build container using `sudo podman build`
make provision-client

# Or: build container using any other container tool
# - replace "docker" with the preferred container build call
cd infra/Containerfile/provision-client && \
    docker build -t provision-client:latest $(pwd)

# call provision shell(defaults to /usr/bin/bash interactive shell)
# defaults to podman, but can be overriden with DOCKER=executable
DOCKER=docker infra/scripts/provision_shell.sh
# use exit to return to base shell

Build documentation

make docs
# build infra-shared documentation
make docs-infra

Create/build/install simulation target

make sim-up

Show/use root and provision cert

make sim-show args="ca_factory" | jq ".root_cert_pem" -r | \
    openssl x509 -in /dev/stdin -noout -text
make sim-show args="ca_factory" | jq ".provision_cert_pem" -r | \
    openssl x509 -in /dev/stdin -noout -text

Manual pulumi invocation

export PULUMI_SKIP_UPDATE_CHECK=1
export PULUMI_CONFIG_PASSPHRASE=sim

pulumi stack select sim
pulumi about

Execute in provision python environment

pipenv run ipython

Sim stack: destroy, cleanup, re/create

make sim-clean
make sim-create

test if changes would compute before applying

make sim-preview
# if list of changes looks good, apply them
make sim-up

cancel an currently running/stuck pulumi update

# "error: the stack is currently locked by 1 lock(s)."
# "Either wait for the other process(es) to end or delete the lock file with `pulumi cancel`."
make sim__ args="cancel"

show resource output as json

make sim-show

show resource output key list as yaml

make sim-list

show resource output data as colorized formatted json or yaml

# use highlight and less
make sim-show | highlight --syntax json -O ansi | less
# use bat for integrated highlight plus pager
make sim-show | bat -l json

Production

Add SSH Keys of GitOps Developer

# eg. add the own ssh public key in project_dir/authorized_keys
cat ~/.ssh/id_rsa.pub >> authorized_keys

Create stack

make prod-create
make prod__ args="preview --suppress-outputs"
make prod__ args=up

Credits

• Inspired and impressed by deuill/coreos-home-server

License

All code in this repository is covered by the terms of the Apache 2.0 License,
the full text of which can be found in the LICENSE file.