Build a home server that adds high speed storage and upload capacity to a global volunteer science network
Switch branches/tags
Nothing to show
Clone or download
Latest commit e4b0ea8 Feb 17, 2017
Permalink
Failed to load latest commit information.
README.md Update README.md Feb 17, 2017

README.md

datasilo

Build a home server that adds high speed storage and upload capacity to a global volunteer science network

Work in progress!

If you want to build a DataSilo, you can use your own hardware or use our recommended hardware below!

If you are ready to run the software, hold on for a little while and keep an eye on svalbard, which is currently being worked on!

Recommended Parts List

You can use any hardware but these are ones that we've tested and work great:

If you're in Canada here's the best link we've found for the compute stick: http://www.ncix.com/detail/intel-compute-stick-stk1aw32sc-atom-6e-134117.htm

Additionally to set it up you will need:

  • USB Thumb Drive (Compute Stick can't boot from SD Card Slot so must be USB)
  • Second computer to flash the USB thumb drive with Ubuntu
  • USB or Bluetooth Keyboard + Mouse
  • Display with HDMI input

Running Ubuntu on the Compute Stick

The Compute Stick comes with Windows. You can choose to use Windows if you prefer, the Svalbard software should run on it (check the Svalbard readme first, work in progress). You can also dual-boot Ubuntu pretty easily.

Known issues

  • Sandisk branded SD cards don't work under Ubuntu
  • Sometimes randonly when booting the ubuntu bootloader hangs on a purple screen and requires a power cycle

Dual boot instructions

  • Download Image: http://www.linuxium.com.au/how-tos/runningubuntuontheintelcomputestick
  • Use Etcher to put it on a USB Flash Drive: http://etcher.io/
  • Boot from USB Stick on Compute Stick using F10 at startup. (You may need to configure the compute stick to boot in Windows 64 mode, which is done by configuring the BIOS using F2 at startup)
  • Choose 'Install Ubuntu' and follow instructions to dual boot Windows with a 10GB split partition.

Helpful Youtube: https://www.youtube.com/watch?v=KAMcr2vbOsA

Setting up the external hard drive

These instructions are for MyBook or MyBook DUO

  1. install WD Drive Utilities on Mac OS
  2. convert to JBOD/EXFAT and turn off Drive Sleep
  3. plug into linux

The next phase uses parted to remove the existing JBOD/ExFAT partitions on the new drive and create an ext4 partition. First we'll find out what the block device name is (usually /dev/sda or /dev/sdb, /dev/sd1, /dev/sd2, or the like), then we'll remove the existing partitions.

lsblk # to view drives and get the /dev/sd<num> entries
sudo parted /dev/sd<num>
print
rm 1 (and others)
mklabel gpt
unit TB
mkpart primary 0.00TB 8.00TB
print
quit

Change 8.00TB above to match the drive size depending on which drive you got

Now we create a new ext4 filesystem: sudo mkfs.ext4 /dev/sd<num>.

Create a mount point for the drive, e.g.

mkdir -p /media/hd1

Find the UUID of the newly created filesystem with ls -l /dev/disk/by-uuid/ and use it to add the appropriate entry in /etc/fstab:

UUID=<UUIDHERE> /media/hd1 ext4 defaults 0 0

finally sudo mount -a to reload fstab and mount the new disk. df -H should now show you a new big drive.

Installing node

On mac, we recommend using Homebrew: brew install node

On linux use the package manager you prefer

On Windows use the MSI installer

Running svalbard on your server

NOTE Svalbard is still under construction. If you are an early adopter and here already, try seeding this dat (9GB of White House Open Datasets):

# install node https://github.com/nodesource/distributions#deb
curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
sudo apt-get install -y nodejs

# set npm prefix https://docs.npmjs.com/getting-started/fixing-npm-permissions
mkdir ~/.npm-global
npm config set prefix '~/.npm-global'
# add to ~/.profile: export PATH=~/.npm-global/bin:$PATH

# install dat and download the data
npm install dat -g
dat clone 6fa91405f280c30cedd461dfcd3b4fffffb27759e26f8135b7cbdfe08870ccb2 eop-gov
cd eop-gov
dat sync # to seed

See https://github.com/datproject/svalbard for instructions when svalbard is ready

Alternative Hardware

Here are devices we tested but for various reasons weren't as good of a fit as the Compute Stick, though YMMV and suggestions are always welcome as the DataSilo is a very hackable, DIY concept.

NAS Boxes

If you have a NAS you can use it! But buying a full NAS might be a little overkill if you are just using your computer solely as a DataSilo.

Requirements are:

  • Has gigabit ethernet
  • Runs Linux and Dat (npm install dat -g), we use libsodium and leveldb native modules
  • Allows JBOD (RAID is not needed with Svalbard/DataSilo as Dat acts as an internet-level RAID)

Mediasonic Probox Enclosure

These are a great way to get a 4 drive JBOD bay (non-RAID). I actually have 2 of them (8 drives total, with 4TB drives). But compared to the 16TB MyBook Duo the price for these wasn't worth it! But could be a great option if you have existing drives you want to use.

Banana Pi M1

$40

MicroSD http://www.banana-pi.org/m1-download.html Ubuntu Mate 16.04 Limitations: Slow CPU encryption

ODROID XU4

$80

http://odroid.in/ubuntu_16.04lts/ubuntu-16.04-mate-odroid-x2-20160920.img.xz Ubuntu Mate 16.04

Limitations: Loud Fan, slow CPU encryption

Raspberry Pi

$30

Limitations: 100Mb Ethernet, USB2, slow CPU encryption

Intel NUC

~$200 and up

Limitations: Many kits barebones (requires more setup), expensive, but speed/performance was good. A great option for a more general purpose machine, but Compute Stick is same thing in smaller, more limited but cheaper package.