Skip to content

brettinternet/homelab

Repository files navigation

Homelab

Lint

Don't be fooled, having a home server is really just hundreds of hours of badblocks.

sudo badblocks -wsv -b 4096 /dev/sda output

Features

Usage

Setup and usage is inspired heavily by this homelab gitops template and the k8s-at-home community. You can find similar setups with the k8s at home search. Historical revisions of this repository had rootless Podman containers deployed with ansible as systemd units, and a single-node docker compose orchestration before that.

Setup

Install go-task

task init

Then, provision your infrastructure.

task ansible:{list,setup,kubernetes,status}

Edit provision/terraform/cloudflare/secret.sops.yaml with your own values and encrypt with task sops:encrypt -- <filepath>.

Setup Cloudflare DNS.

task terraform:{init,cloudflare-plan,cloudflare-apply}

Deploy

Kubernetes

Verify flux can be installed. Then, push changes to remote repo and install.

task cluster:{verify,install}

Push latest to repo - you can use the wip.sh script for that with task wip.

task cluster:{reconcile,resources}

Bastion server

Edit provision/terraform/bastion/secret.sops.yaml with your own values. Generate WireGuard keys.

Deploy the remote bastion VPN server.

task terraform:{init,plan,apply}

Then, setup VPN services.

task ansible:bastion

Deployments

Most deployments in this repo use an app-template chart with these configuration options.

Update

The Renovate bot will help find updates for charts and images. Install Renovate Bot, add to your repository and view Renovate bot activity, or use the self-hosted option.

Hardware

book cover: Mommy, Why is There a Server is the House?

Resources

Memory

Storage

Controller

I used a widely-known and inexpensive method to add additional SATA storage via a Host Bus Adapter (HBA). I purchased a Dell Perc H310 a long while back. Mine did come from overseas, but it turned out to be legit. This video shows how it can be flashed to an LSI 9211-8i IT (see also 1, 2).

Here are other recommended controllers.

2.5" drive stackers

These printable stackers are great for stacking SSDs in a homelab.

Home automation

IoT

Software

Linux

SSH

Here's a nice convenience for setting up authorized_keys stored on Github or Gitlab:

curl https://github.com/<username>.keys -o authorized_keys

You could pipe the output to sed to only grab a specific line sed '4!d'.

Check disks

Here's a handy script to automatically test disks with badblocks and SMART: Spearfoot/disk-burnin-and-testing.

Testing disks takes a long time for larger drives, but it's worth it to be thorough before determining whether to make a return. This is a destructive test, so it's probably best to use /dev/disk/by-id to be certain you're targeting the correct drive.

Use tune2fs -l <partition> to identify the block size.

sudo badblocks -wsv -b 4096 /dev/sda > sda_badblocks.txt

Here's some additional advice from /r/DataHoarders.

JBOD

MergerFS is a union filesystem for pooling drives together. It's a great pair with SnapRAID. An alternative is SnapRAID-BTRFS.

mkdir /mnt/disk{1,2,3,4}
mkdir /mnt/parity1 # adjust this command based on your parity setup
mkdir /mnt/storage # this will be the main mergerfs mount point (a collection of your drives)

Mount drives to these folders, then add /etc/fstab entries by ID.

ls /dev/disk/by-id

You must also include an entry for the MergerFS union, such as:

/mnt/disk* /mnt/storage fuse.mergerfs allow_other,use_ino,cache.files=partial,dropcacheonclose=true,category.create=mfs,fsname=mergerfs,minfreespace=10G 0 0

See also perfectmediaserver: MergerFS

Remember, for data that's irreplaceable RAID is not a backup.

ZFS

Install zfs-dkms and zfs-utils, and be sure to have linux-headers installed for dkms to work. Update the ZFS libraries together using a AUR helper.

OS Installation

Use Ventoy to bundle bootable ISO and IMG images on a single USB.

Setup Proxmox on the hosts with Arch Linux guests. Post setup for Proxmox.

Media

For a media server, it's a good idea to understand digital video.

Troubleshooting

Network

Debug DNS issues

kubectl run curl --rm=true --stdin=true --tty=true --restart=Never --image=docker.io/curlimages/curl --command -- /bin/sh -

curl -k https://kubernetes:443; echo

Ensure you're using iptables-legacy. See also nftables.

iptables --version
# iptables v1.8.7 (legacy)

Flush the iptables in between installs. Also check the CNI installation for issues (such as configuration for hardware with multiple NICs).

Services