ServeD-OS

"Because you deserve to be ServeD!" - an open source server-system designed on top of HardenedBSD and ZFS.

Authors:

• Daniel (dmilith) Dettlaff (2011-…)

Custom/ manual installation:

0. Install 64bit HardenedBSD or FreeBSD with version: 12.2+. If you wish to install svdOS on local vm created with ZFS and xHyve, you may like to follow steps from README here.

1. Run: fetch -o - https://raw.githubusercontent.com/VerKnowSys/svdOS/master/svdup | sh - to install ServeD release/update on vanilla system, or update older system version, to current stable build.

2. After reboot new system will be booted automatically (beadm list to list all available boot environments).

Dedicated machine installation:

System preparation:

NOTE: I use: EX-LINE server type

0. Login to: Hetzner Robot, select your server on servers list.

1. Activate rescue system with version FreeBSD 12.2. Reboot system from web panel: CB2->Reset->Execute an automatic hardware reset. Confirm, wait ~48s (default boot time 30s + ~18s for kernel).

2. Login to rescue system via SSH using provided root password: ssh root@YOUR-SRV-IP.

3. Run bsdinstallimage (script provided by Hetzner). Follow steps:

   • Pick FreeBSD 12.2, 64bit,
   • Continue with default keymap,
   • Enter desired myhostname,
   • Unselect all system components on list,
   • Pick Auto (ZFS) option:
     • Pool disk - use mirror, with both disks (ada0 + ada1),
     • Use defaults for other ZFS settings,
     • Hit Install, confirm that we wish to create new pool,
   • Installation shouldn't take more than a minute. After that type new root password,
   • Select default network card (em0), don't use DHCP, but manual for IPv4 - use predefined values,
   • Select "NO" when asked about UTC, pick "Europe", "Poland", confirm it's "CEST",
   • Skip date/time settings,
   • Enable only: sshd, ntpd, ntpdate, and dumpdev system services,
   • Select all system-hardening options on list, but "Disable process debugging facilities for unprivileged users",
   • Don't add any system users,
   • Hit Exit from menu, confirm that you wish to open terminal - we need to do few additional modifications of base system configuration,
   • Paste these commands in terminal followed by Enter:

   sed -i '' -e 's|#PermitRootLogin no|PermitRootLogin yes|' /etc/ssh/sshd_config
   mkdir -p ~/.ssh
   cat > ~/.ssh/authorized_keys
   

   • Paste your SSH public key(s), followed by Enter. Hit Ctrl-d to save authorized_keys file,
   • That's all. Now hit Ctrl-d (or type exit) to quit Shell.
   • Installation is now complete!

4. When bsdinstallimage is done, type reboot, to reboot to freshly installed FreeBSD-12.2 (vanilla) system. NOTE: This system is used only once - to bootstrap svdup installer - but it's required (as system configuration resource).

Post installation - Shable tasks:

5. On your local workstation - clone Shable: git clone https://github.com/VerKnowSys/Shable.

6. Do: cd Shable && echo "myhostname ip=11.22.33.44 vpn_network=172.16.123 default_jails_domain='mydomain.com'" >> inventory.

   • ip -> External IP address of the dedicated machine (mandatory),

   • default_jails_domain -> Domain used as default one for jails (mandatory),

   • vpn_network -> Override default VPN network (default is: 172.16.3) (optional),

7. Do ./setup-dedicated-system myhostname to complete system installation. NOTE: Default SSH config file (~/.ssh/config ) will be updated with myhostname entry after installation.

8. After about a minute, type: ssh myhostname to login as superuser to remote system.

9. Use gvr (Governor) to maintain your jails. Use s (Sofin) to install server software.

Built in features:

• ServeD system implies, that a single dedicated machine is part of a "distributted infrastructure". By default, OpenVPN is used to provide internal network access for all jails automatically. NOTE: It's necessary to provide your private VPN keys. Without them installation will not be possible at the moment (overridable - maybe even supported one day).

• Hardened kernel and base system (prebuilt with support for all HardenedBSD features).

• Support for full network stack under jails (VIMAGE).

• Preconfigured PF to NAT external network for jails.

• Root filesystem (/) mounted read-only by default (overrideable).

• Software datasets read-only by default (overrideable).

• Limited superuser access to base system only. Listening to ports <1024 is allowed for jail default user (worker).

• Jailed user (worker), has full control over owned ZFS-datasets. Features allowed by default: xattr,atime,casesensitivity,checksum,copies,logbias,primarycache,secondarycache,snapdir,userused,dedup,mountpoint,canmount,userprop,create,destroy,snapshot,rollback,clone,promote,rename,mount,send,receive,reservation,readonly.

• Preconfigured ZSH Shell with cherry picked modules from Oh My ZSH.

• Sofin hardened software is default, but you can still use pkg utility or you can even use ports (not supported!).

• Beadm driven ZFS boot environments (with binary upgrade support, and easy system rollbacks).

• ZFS driven Jail Governor - gvr - installed by default when setting up dedicated system.

• ZFS driven software and configuration management (very fast, block level software installation - thx to Sofin).

• DTrace hooks built in (when supported by software) - provided by Sofin.

• Unbound DNS server is installed by default - configured as caching nameserver - used by all jails.

• Nginx proxy is installed by default - used to proxy external domains to local ones (by name, using local caching DNS).

A few pictures are better than thousand words!

---

Pic 1. For this example I've set: default_jails_domain='myhost.my-shiny.com' when invoking setup-dedicated-system script, below is list of invoked commands with commentary:

#
# This part is invoked on host system (red "λ" = superuser):
#

# Create new jail. Local IP address will be auto-picked from pool of local addresses:
gvr create myhost

# Show detailed info about newly created jail:
gvr status myhost

# To login to new jail system directly from host system:
gvr login myhost


#
# This part is invoked in jail (blue "λ" = regular user: `worker`):
#

# Install Php72 software bundle with default configuration.
s i Php72

# Use it:
php --version
which php
which php-fpm

# Install latest Rust stable
s i Rust

# Use it:
rustc --version
which cargo

---

Pic 2. Private ZFS datasets organization, full network stack in action - loopback is there (yay). NOTE: newly created jails allocate no disk space since they're clones on deduplicated ZFS filesystem.

---

Pic 3. NOTE: My host system is called "cb2". Set SSH public key as SSH access key for myhost jail (only ED25519 keys are supported at the moment) and login to jail over SSH (using my local OpenVPN connection with my private OpenVPN server).

---

Pic 4. Sofin-installer - help() invoked from my macOS workstation.

---

Pic 5. Governor (gvr) commands and jail attributes listed.

---

Pic 6. List boot environments. NOTE: default is our vanilla FreeBSD-12.2.

---

Pic 7. Each Sofin bundle has own immutable list of dependencies.

---

Pic 8. Limit CPU per jail example: myhost to 40% of a single CPU core. NOTE: pcpu can be grater than 100. For example: pcpu=200 means that software will be limited to use 2x100% CPU cores max.

---

Pic 9. There are several quota attributes - here we're setting three of them, each with immediate effect. Attributes are immediately applied on working jails. NOTE: Try: gvr available - to get list of all supported attributes.

--

Pic 10. Software version switching (and environment management) for an example of Php.

Few words about /Shared/ mounts:

• /Shared/Cache/* - Dataset used as a system cache dir. Usually contains file snapshots from ServeD release process.

• /Shared/IPs/* - Contains pool of files of unique IPv4 address of the local (VPN) network.

• /Shared/NetIDs/* - Contains pool of files of unique Net-IDs (also serves as UID of worker user in Cell, and epair network interface number).

• /Shared/DNS/*.conf - DNS dataset with custom Unbound *.conf (file per zone) DNS configurations.

• /Shared/Igniters/*.igni - Dataset contains service igniters, f.e. vpn.igni will be invoked later with igni vpn some-command (for both regular and super user).

• /Shared/Prison/* - Dataset contains two datasets:

  • /Shared/Prison/Cells/* - Dataset with jail cells.

  • /Shared/Prison/Sentry/* - Dataset with metadata and configuration of each jail cell.

Known issues/ problems:

• Only SSH ED25519 pub-keys are supported at the moment.

• After installation default SSH port (22) is switched to ServeD SSH port (60022).

• setup-dedicated-system manages ~/.ssh/config entries automatically - in case of interruption or crash caused by an error - it doesn't invoke any cleanup, hence config has to be fixed manually.

My other projects that are part of the system:

• Shable - "Ansible" compliant software written in pure sh.

• Sofin - "Software Installer" also written in pure sh.

• Sofin-definitions - Sofin software definitions - all pure sh.

• svdOS - this repo, with: system build script, production kernel configuration, other configuration files and system installer and updater script - majority in pure sh as well.

But… why?

• I've learned a lot about systems, software hardening and security.

• It's a challenge and this is my PoC of "how to provide modern, concise and clean open-source server-system".

• I wish to do a server system, which will get as close in quality as macOS is for workstation systems.

• It took me 7 years to implement all the stuff, so I guess it's a lot of fun too ;)

License

BSD/MIT/IDGAF