Go Other
Clone or download
DocSavage make down-res computation for labels synchronous
If async down-res computation is required, we'll need to implement spatial mutexes
or channel solutions that prevent intermixing of mutations at the lowest-resolution
block (for multi-scale labels).  For example, mutex on each lowest-scale block.
Latest commit 89a6bb1 Jun 19, 2018
Permalink
Failed to load latest commit information.
cmd add starting webhook that gets called on dvid startup May 21, 2018
datastore make data instance deletion more resilient to crashes; fixes #216 May 22, 2018
datatype make down-res computation for labels synchronous Jun 19, 2018
dvid make data instance deletion more resilient to crashes; fixes #216 May 22, 2018
images add architecture/use images Oct 13, 2015
rpc refactor sync handling goroutines and shutdown Oct 17, 2016
scripts update to modified lz4 library; fixes #251 May 22, 2018
server add kafka and other info to /server/info endpoint Jun 12, 2018
storage
test_data mutation log support; allow testable kv and log backends Aug 2, 2017
tests_integration add ability to add tags to data instances Mar 16, 2018
.gitignore conda-recipe: Moved into scripts folder Dec 21, 2017
CMakeLists.txt revert to CGo lz4 library due to issues with native Go library Mar 29, 2018
GUIDE.md guide: Clarifications Apr 24, 2018
LICENSE.txt Initial layout of http & cli files. Working open/create of leveldb us… Feb 2, 2013
Makefile Makefile: Added 'install' target, which overwrites the dvid executabl… Apr 23, 2018
README.md Moved install/developer instructions to INSTALL.md Dec 21, 2017
app.yaml Add app.yaml for app engine Dec 17, 2014
circle.yml
doc.go Add ROI datatype Aug 19, 2014

README.md

DVID Picture

Status: In production use at Janelia. See wiki page for outside lab use of DVID.

Go Report Card GoDoc CircleCI

See the DVID Wiki for more information including installation and examples of use.

High-level architecture of DVID

DVID is a distributed, versioned, image-oriented dataservice written to support Janelia Farm Research Center's brain imaging, analysis and visualization efforts. It's goal is to provide:

  • Easily extensible data types that allow tailoring of access speeds, storage space, and APIs.
  • The ability to use a variety of storage systems by either creating a data type for that system or using a storage engine, currently limited to ordered key/value databases.
  • A framework for thinking of distribution and versioning of data similar to distributed version control systems like git.
  • A stable science-driven API that can be implemented either by native DVID data types and storage engines or by proxying to other connectomics services like Google BrainMaps, BOSS, etc.

DVID aspires to be a "github for large image-oriented data" because each DVID server can manage multiple repositories, each of which contains an image-oriented repo with related data like an image volume, labels, annotations, and skeletons. The goal is to provide scientists with a github-like web client + server that can push/pull data to a collaborator's DVID server.

Although DVID is easily extensible by adding custom data types, each of which fulfill a minimal interface (e.g., HTTP request handling), DVID's initial focus is on efficiently handling data essential for Janelia's connectomics research:

  • image and 64-bit label 3d volumes, including multiscale support
  • 2d images in XY, XZ, YZ, and arbitrary orientation
  • multiscale 2d images in XY, XZ, and YZ, similar to quadtrees
  • sparse volumes, corresponding to each unique label in a volume, that can be merged or split
  • point annotations (e.g., synapse elements) that can be quickly accessed via subvolumes or labels
  • label graphs
  • regions of interest represented via a coarse subdivision of space using block indices
  • 2d and 3d image and label data using Google BrainMaps API and other cloud-based services

Each of the above is handled by built-in data types via a Level 2 REST HTTP API implemented by Go language packages within the datatype directory. When dealing with novel data, we typically use the generic keyvalue datatype and store JSON-encoded or binary data until we understand the desired access patterns and API. When we outgrow the keyvalue type's GET, POST, and DELETE operations, we create a custom datatype package with a specialized HTTP API.

DVID allows you to assign different storage systems to data instances within a single repo, which allows great flexibility in optimizing storage for particular use cases. For example, easily compressed label data can be store in fast, expensive SSDs while larger, immutable grayscale image data can be stored in petabyte-scale read-optimized systems.

DVID is written in Go and supports different storage backends, a REST HTTP API, and command-line access (likely minimized in near future). Some components written in C, e.g., storage engines like Leveldb and fast codecs like lz4, are embedded or linked as a library.

DVID has been tested on MacOS X, Linux (Fedora 16, CentOS 6, Ubuntu), and Windows 10+ Bash Shell. It comes out-of-the-box with an embedded leveldb for storage although you can configure other storage backends.

If you just need nd-array access, consider DICED, which provides a simple python numpy interface to the main image/label datatypes in DVID. Because of the limited datatypes within DICED, petabyte-scale distributed computing support for mutations will be available sooner in DICED than for the full range of datatypes in DVID.

Command-line and HTTP API documentation can be found in help constants within packages or by visiting the /api/help HTTP endpoint on a running DVID server.

Web app for 3d inspection being served from and sending requests to DVID