Google Cloud Storage

(in development)

Google storage provides an unordered key/value object store. This plugins supports all DVID datatypes but the latency is likely inadequate for some interactive use cases. Because google storage is relatively cheap, the recommended use case is batch access of large data. It is possible to do parallel volume fetches with multiple DVID services accessing the same google bucket.

Installation

To enable Google Storage Bucket support when building DVID, include 'gbucket' in the DVID_BACKEND flag as in:

cmake -DDVID_BACKEND="gbucket" -DBUILDEM_DIR=$BUILDEM_DIR ..

Create Google Bucket

(requires a google account and a billable project)

Configure DVID

Point to the Google bucket by modifying the DVID toml configuration file. Here is a sample toml that will cause DVID to run on port 8000 and will point to a Google bucket called "nameofbucket".

[server]
httpAddress = "0.0.0.0:8000"
rpcAddress = "0.0.0.0:8001"
webClient = "/usr/local/dvid-console"
# How new data instance ids are generated.
# Is one of "random" or "sequential".  If "sequential" can set "start_instance_id" property.
instance_id_gen = "sequential"
instance_id_start = 100  # new ids start at least from this.
[logging]
logfile = "/tmp/logs/dvid.log"
max_log_size = 500 # MB
max_log_age = 30   # days
[store]  
# if no store.metadata is provided, it defaults to store.mutable configuration
[store.mutable]
    engine = "gbucket"
    bucket= "nameofbucket"

Launching DVID

It is recommended that DVID services are run on Google VMs.

Running DVID on Google VM

Create a VM instance using the desired hardware configuration:

1. Check 'Allow http traffic'
2. Check 'Allow API access to all Google Cloud services in the same project.' so that Google buckets can be accessed from the VM. This assumes that the VM account holder has read/write permission to the bucket.

Running DVID outside of Google

To access Google Cloud Storage, the machine / environment running DVID should have access to google services. One should just use application default credentials so that DVID is given access to everything. There are instructions here: https://developers.google.com/identity/protocols/application-default-credentials. The easiest solution is to set the GOOGLE_APPLICATION_CREDENTIALS environment to point to a JSON that can be downloaded from the developers console.