diff --git a/README.md b/README.md index 9f940cf..53e3406 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,7 @@ For slides rendered to PDF or presentation notes for offline reading, download t - Git - [101](git101/index.md), [150](git150/index.md), [201](https://github.com/jmunixusers/git-201) (External), [301](git301/index.md), [401](git401/index.md) - [groff](groff/index.md) - Homelabbing - [Part 1 (Theory)](homelab.md), [Part 2 (Demo)](homelab-2.md) +- [Homelab Storage](homelab-storage/index.md) - [How to build a VM](BuildAVM/index.md) - [Intel Pin](https://github.com/lam2mo/uug-pin) (External) - [LaTeX](LaTeX.md) diff --git a/homelab-storage/cockpit.png b/homelab-storage/cockpit.png new file mode 100644 index 0000000..445e741 Binary files /dev/null and b/homelab-storage/cockpit.png differ diff --git a/homelab-storage/index.md b/homelab-storage/index.md new file mode 100644 index 0000000..f9b13c4 --- /dev/null +++ b/homelab-storage/index.md @@ -0,0 +1,377 @@ +--- +marp: true +theme: default +class: invert +footer: Slides written by Connor Sample - https://tabulate.tech +--- + +# Homelabbing - Storage & Backups + +--- + + + +## Recap: What is homelabbing? + +--- + +## Resources + +- +- +- +- + +--- + +## Storage Issues + +- 3 Problems with Storage: + - **Capacity** + - **Redundancy** + - **Integrity (bit rot)** + +--- + +## Hardware + +- **CMR (Conventional Magnetic Recording)**: tracks are next to each other (this is what you need to use) +- **SMR (Shingled Magnetic Recording)**: tracks are "shingled" and they overlap + - Higher density and cheaper, but struggle during long sustained data writes (like resilvering) + +--- + + + +## RAID + +- Two main types: + - Hardware + - Software (`mdadm`) +- All about "combining drives" and not much more + - Still has different techniques + +--- + + + +## ZFS (Zettabyte File System) + +- **Volume Manager + File System** +- **COW (Copy-on-Write)**: crash consistency +- **Block Checksums (bit rot)**: auto-heal from parity +- **Smart ARC Caching** +- **Snapshots**: fast & space efficient +- **Fully software** + +--- + +## BTRFS + +- Evil (ripley made me include this even though it always breaks for me) +- GPL (can be included in the kernel) vs CDDL (ZFS) +- Similar features like COW, snapshots, checksums +- Can add in drives after the fact + +--- + + + + + +## Common Terminology + +- **Parity**: Extra data used for redundancy +- **Pool**: Top-level collection of VDEVs. One big "pool" of all your drives +- **VDEV (Virtual Device)**: Group of physical disks + - **Stripe**: Just combine disks, no redundancy (RAID 0) + - **Mirror**: Same data on two drives (RAID 1) + - **Striped Mirror**: Stripe pairs of mirrored disks (RAID 10) + - **RAIDZ1**: Single parity (RAID 5) + - **RAIDZ2**: Double parity (RAID 6) +- **Dataset**: A "file system" in the pool + - Metadata (compression, quotas) can differ from pool + - **ZVol**: "raw block device" + +--- + + + +### DIY vs Managed ZFS: TrueNAS + + + +- Ease of use/setup +- Nice monitoring UI out of the box +- True Immutable +- Mainly for NAS: application management not great + +![TrueNAS Dashboard](./truenas_dashboard.png) + +--- + +### DIY vs Managed ZFS: Cockpit (45Drives plugin) + + + +- NOT immutable (cockpit runs anywhere as a web service) +- Less focused +- Still in pretty early development, docs don't exist +- Only built for Rocky, Debian, and Ubuntu + +
+ +![width:700px](cockpit.png) + +
+ +--- + +### DIY vs Managed ZFS: Manual + +- Much simpler than you would think +- Full control & understanding +- Works anywhere +- Doesn't have a monitoring UI out of the box + +--- + +## Creating a ZPool + +```sh +sudo apt install zfsutils-linux +ls -l /dev/disk/by-id/ +sudo zpool create \ + -o ashift=12 \ # for modern 4KB sector drives. should be 13 for 8K sectors + -O compression=lz4 \ # or zstd, off, gzip + -O atime=off \ # performance/drive health + tank raidz1 \ # pool name and zfs config + /dev/disk/by-id/ata-DRIVE1_______ \ # list which drives make up the pool + /dev/disk/by-id/ata-DRIVE2_______ \ + /dev/disk/by-id/ata-DRIVE3_______ +zpool status +``` + +--- + +## Creating a Dataset + +```sh +sudo zpool create tank/backups +# optionally, change metadata: +sudo zpool set compression=zstd tank/backups +zfs list +``` + +--- + +## Maintenance + +- Crontab for scrub/trim should be installed to `/etc/cron.d/zfsutils-linux` (at least on debian) +- Rebuild after a failed disk: + +```sh +# find the degraded drive (may just be a numeric ID if its really dead): +zpool status +# take the drive offline for the "tank" pool: +zpool offline tank ata-OLDDRIVE... +# find the ID of the new disk: +ls -l /dev/disk/by-id/ +# replace the disk: +zpool replace tank ata-OLDDRIVE /dev/disk/by-id/ata-NEWDRIVE +``` + +--- + +## ZFS Zed + +- Receive emails when scrub fails + +--- + +## Backups + +--- + +## RAID IS NOT A BACKUP + +- RAID protects against hardware failures + - RAID will not help if you `sudo rm -rf /tank` +- Backups protect against human error, disaster, etc. + +--- + +## 3-2-1 Backups + +- 3 copies of your data +- 2 different mediums +- 1 offsite + +--- + +## ZFS Snapshots vs Backups + +- Snapshots: + - Incredibly fast and space efficient (just copies pointers to data instead of the data itself) + - Good for accidental deletion of files + - On same drive as data +- Backups: + - Live on different media + +--- + +## ZFS Send/Receive + +- Computes the block-level differences between snapshots and sends a stream of the changes to another machine +- Very fast (incremental) +- Metadata is saved + +```properties +zfs send tank/data@snapshot50 | ssh user@machine zfs recv tank/backup +``` + +--- + +## BorgBackup + +- Space efficient (deduplicating) +- Encryption +- Compression (if not using ZFS) +- Mount backups with FUSE +- Backup diff + +--- + +## Setting Up Borg + + + +```sh +# SERVER: +sudo apt install borgbackup + +# create borg user and adjust permissions (assumes /tank/backups exists) +sudo adduser borg +mkdir -p /tank/backups/reponame +sudo chmod 700 /tank/backups +sudo chown -R borg:borg /tank/backups +``` + +```sh +# HOST MACHINE: +sudo apt install borgbackup +# Create a separate passwordless SSH keypair and copy it to the server +ssh-keygen -t ed25519 -f ~/.ssh/id_borg +ssh-copy-id -i ~/.ssh/id_borg borg@192.168.x.x +``` + +```sh +# SERVER: +# restrict borg SSH key permissions by editing `/home/borg/.ssh/authorized_keys` and modifying the first line to look like this: +command="borg serve --restrict-to-path /tank/backups",restrict ssh-ed25519 AAAAC3Nz... (remainder of key) +``` + +```sh +# HOST MACHINE: +# init the repository +export BORG_REPO=ssh://borg@192.168.x.x/tank/backups/reponame +export BORG_RSH="ssh -i ~/.ssh/id_borg" +borg init --encryption=repokey +``` + +--- + +### Creating a Backup + +```sh +borg create ::$(date +%Y-%m-%d) ~/Downloads ~/Documents +``` + +### Pruning Backups + +```sh +borg prune --list --keep-daily 7 --keep-weekly 4 --keep-monthly 6 +``` + +### Compacting + +```sh +borg compact +``` + +--- + + + +
+ +## Borg Frontends + +- Vorta +- Pika +- UI for the following: + - Repo/ssh key creation + - Scheduled backups + - Pruning + - Diff/mount/restore + - Folder/file selection/exclusion + +
+ +
+ +![](vorta.png) +![](pika.png) + +
+ +--- + +## Questions diff --git a/homelab-storage/pika.png b/homelab-storage/pika.png new file mode 100644 index 0000000..cd8ebc4 Binary files /dev/null and b/homelab-storage/pika.png differ diff --git a/homelab-storage/truenas_dashboard.png b/homelab-storage/truenas_dashboard.png new file mode 100644 index 0000000..2285b8f Binary files /dev/null and b/homelab-storage/truenas_dashboard.png differ diff --git a/homelab-storage/vorta.png b/homelab-storage/vorta.png new file mode 100644 index 0000000..acac4d7 Binary files /dev/null and b/homelab-storage/vorta.png differ