Skip to content
Tools for making remote Linux node management easy
Shell Makefile
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
aktos-bash-lib @ f4597f7 more refactoring Jun 2, 2019
doc separated recipes Jun 1, 2019
smith-sync @ 73fcabb fixed "networking reload" issue on restored disks Jun 2, 2019
.gitignore
.gitmodules depends on smith-sync for backup tasks Dec 9, 2018
README.md separated recipes Jun 1, 2019
common.sh cleanup WIP Jun 22, 2018
config-template.sh cleanup WIP Jun 22, 2018
create-ssh-config
create-tunnel cleanup WIP Jun 22, 2018
exclude-list.txt
format-orange-pi.sh improved console output Jun 1, 2019
format-raspberry-pi.sh improved UX Jun 2, 2019
give-new-id fixed key type handling exception, improved output Dec 8, 2018
make-backup fixed missing test for hardlinks method Nov 26, 2017
make-target-settings make-target-settings is a standalone script now Jul 16, 2017
md5dir added md5dir scripts Aug 10, 2018
md5dir-strict
mount-direct cleanup WIP Jun 22, 2018
mount-proxy cleanup WIP Jun 22, 2018
produce-bootable-disk
restore-from-backup improved UX Jun 2, 2019
setup cleanup WIP Jun 22, 2018
ssh-direct added missing create-ssh-config line Jun 23, 2018
ssh-proxy added missing create-ssh-config line Jun 23, 2018
sync-root
sync.sh more refactoring Jun 2, 2019
toolset-make-backup.sh added "make backup-sync" target, resolves #18 Nov 26, 2017
toolset-makefile.mk
umount-node-root
update.sh

README.md

Description

This toolset is intended to use for managing remote Linux devices (RaspberryPi in mind, but any remote Linux system will work) from host Linux systems, by basically simplifying 6 tasks, where you need to:

  1. ..make ssh for performing remote tasks (either directly or over a link up server).
  2. ..use simple drag and drop style file transfers (by sshfs).
  3. ..backup the target's entire root filesystem (by rsync).
  4. ..create fast and efficient differential full backups (by hardlinks or by BTRFS snapshots).
  5. ..create a separate bootable system disk from any of your backups.
  6. ..clone a target with a new identity.

This simplification is achieved by:

  • Placing separate scripts for each task described above and providing a simple Makefile.
  • Keeping the scripts, configuration files and backups in a folder called your-project.

Install

Requirements

  • Linux OS
  • git (for submodule fetching and make update)
  • rsync
  • sshfs

Setup

Follow these steps for every project:

# on your host (eg. your laptop)
mkdir your-project
cd your-project
git clone --recursive https://github.com/aktos-io/dcs-tools

Configuration

Assuming you are in /path/to/your-project folder already,

  1. Create your configuration file and mandatory folders/flags:

    ./dcs-tools/setup
    
  2. (Optional): Send your RSA public key to the target in order to prevent asking password on every connection:

    ./dcs-tools/make-target-settings  
    

Usage

Set connection type

either connect to your target by its direct IP address and port:

make conn-direct

or meet with your target on a link up server (see link-with-server)

make conn-over-proxy

See doc/configuration.md for explanations.

Make ssh

make ssh

Makes ssh connection either directly or via the link up server according to your connection type.

Mount target root

make mount-root

Mounts the root folder to your-project/NODE_ROOT, which you can use for drag-n-drop style file transfers.

You can later unmount with make umount-root without using sudo command.

Sync target's root folder

make sync-root

Sync whole root partition of the target with your-project/sync-root folder. You can use this command consecutively to keep your sync-root folder up to date as much as possible. Only differentiating data will be transmitted (if any).

This command will only copy the current state of your target to your host machine. You will need to create your backups manually, with make backup-sync-root command

Create backups

make backup-sync-root

Create a backup from the sync-root folder into ./backups folder either by hardlinks method or by creating a btrfs subvolume, according to your your-project/method-* flag.

method-* flags can be set by make method-btrfs or make method-hardlinks commands.

Advanced actions:

Following tools are for advanced usage, use them with extreme caution.

Produce bootable disk from a backup

./dcs-tools/produce-bootable-disk [--help]   

Produces a bootable disk that is capable of booting your target hardware.

Restore files from a backup to physical disk

./dcs-tools/restore-from-backup [--help]

Restores all files from backup folder to the SD card. Useful when you want to update your physical backup disk with your latest sync folder.

Give New ID

Make appropriate changes to give new identity to an installation. Useful for creating a new controller/machine based on current installation.

./dcs-tools/give-new-id [--help]

See Also

Advantages

Backups have following properties:

  • portable (you can move your copies around. eg: take first backup locally, remove disk, mound on another computer, make backup-root again)
  • differential (only differences are transmitted)
  • dead simple copies of original files (you can simply copy/paste when you need to restore or move your files around) (see BIG WARNING)
  • versioned : Take any number of full backups as much as you like. You are responsible for deleting old backups.
  • efficient storage usage (if you backup your 10 GB root for 5 times, you end up using 10.2 GB disk space if you have no modified files. But you will see the snapshots folder has a size of 50 GB. (Magic? No: Hardlinks or BTRFS subvolumes)

BIG WARNING

Move your backups around carefully

If you are not using btrfs, "dead simple copies" feature has a problem by its design: As backups are just plain folders, this may lead breaking (unintentionally changing) the ownership of the files if you move/copy your files carelessly (eg. if you mv your/snapshot to/another/location and then interrupt the command in the middle, you will probably end up with moved files having root:root permissions.) That's why you SHOULD always use rsync for such movements.

If you are using --method btrfs, backups are made as readonly snapshots, so you will not have such problems.

Use correct filesystem

Make sure that you are performing make sync-root command on a native Linux filesystem. You will end up having a backup with wrong file ownership and/or permissions otherwise.

You can’t perform that action at this time.