Store 10MB-10GB-ish personal files on cloud
Go HTML JavaScript Shell
Latest commit 1402594 Feb 19, 2017 @nyaxt static build
Permalink
Failed to load latest commit information.
Godeps
blobstore
btncrypt
chunkstore
cmd
debugcmd
doc
facade
filewritecache
flags
fuse
gc
gcloud
go-fuzz
inodedb
logger
metadata
mgmt
out
scheduler
scripts
testutils
util
vendor
version
webdav
webui
.dockerignore
.gitignore
Dockerfile
LICENSE
README.md
circle.yml
filesystem.go
filesystem_test.go
fsutil.go
inodedbchunksarrayio.go

README.md

otaru

Circle CI

Otaru is a cloud-backed filesystem for archiving your files. Otaru is optimized for storing personal collection of 10MB-10GB-ish files, such as book scan pdfs and disk image dumps.

For more details, see Design Doc

Quick Start

Build otaru inside Docker container

Building otaru takes a bit of time (Approx 6 min with decent internet connection). You may want to start building while doing other setup.

$ git clone https://github.com/nyaxt/otaru && cd otaru
$ docker build -t otaru .
$ docker run -ti --rm -v `pwd`/out:/out otaru

Configure Google Cloud Platform

  • Access Google Cloud Console, and have a project ready (preferrably not the default "API Project") with Cloud Datastore and Cloud Storage enabled.
  • Allow the following API usage from "APIs & auth" -> "APIs":
    • Google Cloud Datastore API
    • Google Cloud Storage API
    • Google Cloud Storage JSON API
  • Create two new buckets to store Otaru blobs/metadata. You can create new bucket from "Storage" -> "Cloud Storage" -> "Browser".
    • Otaru supports using separate bucket for storing metadata, which is accessed more frequently compared to blobs.
    • Blob bucket may be any of Standard, Durable Reduced Availability, or Nearline. However, it is recommended to store metadata in Standard class bucket.
    • Metadata bucket name must be blob bucket name + "-meta" suffix. For example, if your bucket for storing blobs is named "otaru-foobar", metadata bucket must be named "otaru-foobar-meta"
  • Issue OAuth 2.0 client ID. "APIs & auth" -> "Credentials" -> "Add credentials" -> "API key"
    • Choose "OAuth 2.0 client ID", Application type "Other".
    • Name the ID "Otaru client" or something distinguishable.
    • Download the client secret by clicking on the "Download JSON" icon button located on the left of the table.

Create {config,password} file & place OAuth 2.0 credentials

$ mkdir ~/.otaru # You may change this dir to any dir you want, but a new directory is needed as otaru has multiple config files to keep.
$ cp doc/config.toml.example ~/.otaru/config.toml
$ $EDITOR ~/.otaru/config.toml # replace placeholders
$ cp [downloaded-client-secret-json] ~/.otaru/credentials.json
$ echo [your-password] > ~/.otaru/password.txt # configure encryption key

Setup Google Cloud SDK

Install Google Cloud SDK

Install gcloud command per instructions: https://cloud.google.com/sdk/

Enable Alpha/Beta commands

$ gcloud components update alpha beta

Authenticate gcloud tool

$ gcloud auth login

Authorize Google Cloud Storage / Datastore access to Otaru

Complete the step "Build otaru inside Docker container" before executing below.

$ OTARUDIR=~/.otaru out/otaru-gcloudauthcli

Visit displayed url, and paste response code. Check that ~/.otaru/tokencache.json is correctly generated.

Setup Google Cloud Datastore index, and verify Google Cloud Storage settings

$ OTARUDIR=~/.otaru scripts/gcloud_setup.bash

Mount!

$ mkdir -p /otaru/foobar && sudo chown `whoami` /otaru/foobar
$ OTARUDIR=~/.otaru out/otaru-mount -mkfs /otaru/foobar # -mkfs option is only for first mount.

Enjoy using /otaru/foobar. Press Ctrl-C to start unmount sequence. Navigate to http://localhost:10246 for webui.