Build a home server that adds high speed storage and upload capacity to a global volunteer science network
Work in progress!
If you want to build a DataSilo, you can use your own hardware or use our recommended hardware below!
If you are ready to run the software, hold on for a little while and keep an eye on svalbard, which is currently being worked on!
Recommended Parts List
You can use any hardware but these are ones that we've tested and work great:
- Server: Intel Compute Stick V2 (~$125)
- Hard Drive: WD MyBook 4TB (~$110)
- Hard Drive: WD MyBook Duo 16TB (~$499)
- (Optional) Anker USB3/Gigabit Ethernet Hub ($25)
If you're in Canada here's the best link we've found for the compute stick: http://www.ncix.com/detail/intel-compute-stick-stk1aw32sc-atom-6e-134117.htm
Additionally to set it up you will need:
- USB Thumb Drive (Compute Stick can't boot from SD Card Slot so must be USB)
- Second computer to flash the USB thumb drive with Ubuntu
- USB or Bluetooth Keyboard + Mouse
- Display with HDMI input
Running Ubuntu on the Compute Stick
The Compute Stick comes with Windows. You can choose to use Windows if you prefer, the Svalbard software should run on it (check the Svalbard readme first, work in progress). You can also dual-boot Ubuntu pretty easily.
- Sandisk branded SD cards don't work under Ubuntu
- Sometimes randonly when booting the ubuntu bootloader hangs on a purple screen and requires a power cycle
Dual boot instructions
- Download Image: http://www.linuxium.com.au/how-tos/runningubuntuontheintelcomputestick
- Use Etcher to put it on a USB Flash Drive: http://etcher.io/
- Boot from USB Stick on Compute Stick using F10 at startup. (You may need to configure the compute stick to boot in Windows 64 mode, which is done by configuring the BIOS using F2 at startup)
- Choose 'Install Ubuntu' and follow instructions to dual boot Windows with a 10GB split partition.
Helpful Youtube: https://www.youtube.com/watch?v=KAMcr2vbOsA
Setting up the external hard drive
These instructions are for MyBook or MyBook DUO
- install WD Drive Utilities on Mac OS
- convert to JBOD/EXFAT and turn off Drive Sleep
- plug into linux
The next phase uses
parted to remove the existing JBOD/ExFAT partitions on the new drive and create an ext4 partition. First we'll find out what the block device name is (usually
/dev/sd2, or the like), then we'll remove the existing partitions.
lsblk # to view drives and get the /dev/sd<num> entries sudo parted /dev/sd<num> print rm 1 (and others) mklabel gpt unit TB mkpart primary 0.00TB 8.00TB print quit
8.00TB above to match the drive size depending on which drive you got
Now we create a new ext4 filesystem:
sudo mkfs.ext4 /dev/sd<num>.
Create a mount point for the drive, e.g.
mkdir -p /media/hd1
Find the UUID of the newly created filesystem with
ls -l /dev/disk/by-uuid/ and use it to add the appropriate entry in
UUID=<UUIDHERE> /media/hd1 ext4 defaults 0 0
sudo mount -a to reload fstab and mount the new disk.
df -H should now show you a new big drive.
On mac, we recommend using Homebrew:
brew install node
On linux use the package manager you prefer
On Windows use the MSI installer
svalbard on your server
NOTE Svalbard is still under construction. If you are an early adopter and here already, try seeding this dat (9GB of White House Open Datasets):
# install node https://github.com/nodesource/distributions#deb curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash - sudo apt-get install -y nodejs # set npm prefix https://docs.npmjs.com/getting-started/fixing-npm-permissions mkdir ~/.npm-global npm config set prefix '~/.npm-global' # add to ~/.profile: export PATH=~/.npm-global/bin:$PATH # install dat and download the data npm install dat -g dat clone 6fa91405f280c30cedd461dfcd3b4fffffb27759e26f8135b7cbdfe08870ccb2 eop-gov cd eop-gov dat sync # to seed
See https://github.com/datproject/svalbard for instructions when svalbard is ready
Here are devices we tested but for various reasons weren't as good of a fit as the Compute Stick, though YMMV and suggestions are always welcome as the DataSilo is a very hackable, DIY concept.
If you have a NAS you can use it! But buying a full NAS might be a little overkill if you are just using your computer solely as a DataSilo.
- Has gigabit ethernet
- Runs Linux and Dat (
npm install dat -g), we use libsodium and leveldb native modules
- Allows JBOD (RAID is not needed with Svalbard/DataSilo as Dat acts as an internet-level RAID)
Mediasonic Probox Enclosure
These are a great way to get a 4 drive JBOD bay (non-RAID). I actually have 2 of them (8 drives total, with 4TB drives). But compared to the 16TB MyBook Duo the price for these wasn't worth it! But could be a great option if you have existing drives you want to use.
Banana Pi M1
MicroSD http://www.banana-pi.org/m1-download.html Ubuntu Mate 16.04 Limitations: Slow CPU encryption
Limitations: Loud Fan, slow CPU encryption
Limitations: 100Mb Ethernet, USB2, slow CPU encryption
~$200 and up
Limitations: Many kits barebones (requires more setup), expensive, but speed/performance was good. A great option for a more general purpose machine, but Compute Stick is same thing in smaller, more limited but cheaper package.