Skip to content
An object store for scientific data
Branch: master
Clone or download
wgerlach Merge pull request #366 from wltrimbl/master
De-novo tests for compressed outputs
Latest commit f12ccfe Mar 9, 2019
Type Name Latest commit message Commit time
Failed to load latest commit information.
libs Mar 8, 2019
shock-server fix auth Mar 5, 2019


Shock is a platform to support computation, storage, and distribution. Designed from the ground up to be fast, scalable, fault tolerant, federated. (see Shock: Active Storage for Multicloud Streaming Data Analysis, Big Data Computing (BDC), 2015 IEEE/ACM 2nd International Symposium on, 2015)

Shock is RESTful. Accessible from desktops, HPC systems, exotic hardware, the cloud and your smartphone.

Shock is for scientific data. One of the challenges of large volume scientific data is that without often complex metadata it is of little to no value. Shock allows storage and querying of complex metadata.

Shock is a data management system. The long term goals of Shock include the ability to annotate, anonymize, convert, filter, perform quality control, and statically subsample at line speed bioinformatics sequence data. Extensible plug-in architecture is in development.

Shock is actively being developed at


To build Shock manually, use the Makefile. Note that you need golang (>=1.6.0).


You can get the Shock Dockerimage with:

docker pull mgrast/shock

Or, to build the Docker image on you own:

export TAG=`date +"%Y%m%d.%H%M"`
git clone --recursive
cd Shock
docker build --force-rm --no-cache --rm -t mgrast/shock:${TAG} .

If you only need the statically compiled binary, you can extract it from the Dockerimage:

docker create --name shock mgrast/shock:${TAG}
docker cp shock:/go/bin/shock-server .
docker cp shock:/go/bin/shock-client .
docker rm shock


In ubuntu you can simply install mongo with:

sudo apt-get install -y mongodb-server

If you do not want to use a package manager to install mongodb, use:

curl -s | tar -v -C /mongodb/ -xz

If you do not use a service manager such as systemd, you can start mongodb like this, in foreground:

/mongodb/bin/mongod --dbpath /data/

or in background:

nohup /mongodb/bin/mongod --dbpath /mnt/db/ &

You can also run MongoDB in a docker container:

mkdir -p /mnt/shock-server/mongodb
export DATADIR="/mnt/shock-server"
docker run --rm --name shock-server-mongodb -v ${DATADIR}/mongodb:/data/db --expose=27017 mongo mongod --dbpath /data/db


The Shock configuration file is in INI file format. There is a template of the config file located at the root level of the repository.


To run:

shock-server -conf <path_to_config_file>

With docker:

mkdir -p /mnt/shock-server/log
mkdir -p /mnt/shock-server/data
export DATADIR="/mnt/shock-server"
docker run --rm --name shock-server -p 7445:7445 -v ${DATADIR}/shock-server.cfg:/shock-config/shock-server.cfg -v ${DATADIR}/log:/var/log/shock -v ${DATADIR}/data:/usr/local/shock --link=shock-server-mongodb:mongodb mgrast/shock /go/bin/shock-server --conf /shock-config/shock-server.cfg

port 7445: Shock server API (default in config)
"-v" mounts host to container directories
"--link" connects Shock server and mongodb (--link=$imagename:$alias) so you need to put the alias (in the example "mongodb") as the value of the hosts variable in the shock-server.cfg


For further information about Shock's functionality, please refer to our github wiki.

You can’t perform that action at this time.