Holographic storage for distributed applications -- a validating monotonic DHT "backed" by authoritative hashchains for data provenance (a Ceptr sub-project)
Latest commit 170a412 Sep 18, 2018
Permalink
Failed to load latest commit information.
apptest remove MIGRATE_REQUEST and VALIDATE_MIGRATE_REQUEST May 26, 2018
bin fixed test example May 7, 2018
cmd Fixed windows path test in common_test.go Jun 3, 2018
docker removed hccore Apr 29, 2018
examples downcase github url Mar 21, 2018
hash lint gx May 24, 2018
peerqueue lint gx May 24, 2018
scriptsAccessibleByDockerfiles moved the offending scripts to its own dir, so that .dockerignore can… Jun 1, 2017
ui accept-header Content-Type && early return OPTIONS May 31, 2018
.dockerignore dockerignore with docker and bin in it Jun 1, 2017
.gitignore Encrypt varaibles for .travis.yml Jan 16, 2018
.travis.yml added a smoke test for docker that should fail tests Apr 29, 2018
LICENSE Missed one Mar 16, 2018
Make.bat update straggling uppercase H Mar 23, 2018
Makefile removed hccore Apr 29, 2018
README.md Update README.md Sep 18, 2018
ReleaseNotes.md fixed gitter url Mar 21, 2018
action.go use Change on the dht for action migrate May 30, 2018
action_addlist.go lint gx May 24, 2018
action_bridge.go Merge commit '0e5fc625d684a784c00f09eb85cab52f15f48580' into 716 May 22, 2018
action_bundle_test.go added action_bundle_test.go May 22, 2018
action_bundleclose.go move APIFnCloseBundle to action_bundleclose.go May 22, 2018
action_bundlestart.go move APIFnStartBundle to action_bundlestart.go May 22, 2018
action_call.go move APIFnCall to action_call.go May 22, 2018
action_commit.go lint gx May 24, 2018
action_commit_test.go move TestActionCommit to action_commit_test.go May 22, 2018
action_debug.go move APIFnDebug to action_debug.go May 22, 2018
action_delete.go lint gx May 24, 2018
action_delete_test.go standardise TestActionDelete to match migrate test May 30, 2018
action_get.go lint gx May 24, 2018
action_get_test.go lint gx May 24, 2018
action_getbridges.go move APIFnGetBridges to action_getbridges.go May 22, 2018
action_getlinks.go lint gx May 24, 2018
action_link.go lint gx May 24, 2018
action_makehash.go move APIFnMakeHash to action_makehash.go May 22, 2018
action_migrate.go fix missing err in Share for action_migrate.go May 30, 2018
action_migrate_test.go remove panics in action_migrate_test.go May 30, 2018
action_property.go move APIFnProperty to action_property.go May 22, 2018
action_put.go lint gx May 24, 2018
action_query.go move APIFnQuery to action_query.go May 22, 2018
action_send.go lint gx May 24, 2018
action_sign.go lint gx May 24, 2018
action_sign_test.go lint gx May 24, 2018
action_test.go lint gx May 24, 2018
action_update.go lint gx May 24, 2018
action_update_test.go lint gx May 24, 2018
action_updateagent.go lint gx May 24, 2018
agent.go lint gx May 24, 2018
agent_test.go lint gx May 24, 2018
app_package.go improve comments on code generated with `hcdev init` Apr 28, 2018
app_package_test.go improve comments on code generated with `hcdev init` Apr 28, 2018
bridge.go added CalleeName to bridge data and normalized ToApp->CalleeApp, closes May 23, 2018
bridge_test.go added CalleeName to bridge data and normalized ToApp->CalleeApp, closes May 23, 2018
bs.go lint gx May 24, 2018
buntdbht.go fixed bug in DHT JSON dump and make format work for DHT too May 24, 2018
buntdbht_test.go holding wip Apr 3, 2018
capabilities.go fixed BridgeCall bug Aug 15, 2017
capabilities_test.go added bridge calling, wip #110 Aug 2, 2017
chain.go added "migrate" function to jsribosome May 25, 2018
chain_test.go fix tests May 27, 2018
change.go moved requries from zome function to DNA attribute, closes #96 Mar 16, 2017
change_test.go updated ShouldLog for multiple messages Dec 7, 2017
dht.go provide a fallback HashType for DNAs Jun 6, 2018
dht_test.go lint gx May 24, 2018
doc.go downcase github url Mar 21, 2018
entry.go fix "user" in migrate May 29, 2018
entry_agent.go fix tests May 23, 2018
entry_agent_test.go move TestAgentEntryToJSON to entry_agent_test.go May 23, 2018
entry_del.go Merge commit 'd92ada979324ece379705378057bd39c3a6a3478' into 684-clos… May 24, 2018
entry_del_test.go WIP on migrate validation tests May 26, 2018
entry_dna.go move DNAEntry to entry_dna.go May 23, 2018
entry_headers.go move HeadersEntrySchema to entry_headers.go May 23, 2018
entry_key.go move KeyEntryType to entry_key.go May 23, 2018
entry_links.go move LinksEntry to entry_links.go May 23, 2018
entry_migrate.go rename User to Key and Chain to DNAHash in migrate May 29, 2018
entry_migrate_test.go lint May 30, 2018
entry_test.go added some tests for constants May 26, 2018
gossip.go bug fix May 24, 2018
gossip_test.go lint gx May 24, 2018
header.go lint gx May 24, 2018
header_test.go lint gx May 24, 2018
holochain.go provide a fallback HashType for DNAs Jun 6, 2018
holochain.pdf update whitepaper pdf Mar 6, 2018
holochain.tex A few changes to HC def, item 12 h) and i). Removed incorrect mention… Mar 5, 2018
holochain_test.go Merge pull request #698 from holochain/684-closing-entries May 30, 2018
ht.go lint gx May 24, 2018
ht_test.go addPeer now gets the PubKey from originator when confrm true Apr 1, 2018
jsribosome.go add HC.SysEntryType.Migrate for JS zomes Jun 7, 2018
jsribosome_test.go add HC.SysEntryType.Migrate for JS zomes Jun 7, 2018
kad_bucket.go lint gx May 24, 2018
kad_lookup.go lint gx May 24, 2018
kad_lookup_test.go lint gx May 24, 2018
kad_query.go lint gx May 24, 2018
kad_rpc.go lint gx May 24, 2018
kad_rpc_test.go lint gx May 24, 2018
kad_table.go lint gx May 24, 2018
kad_table_test.go lint gx May 24, 2018
log.go get requests now really work on dht via kademlia query, #244 Sep 27, 2017
log_test.go #405 removed more color information from the test. Not possible to te… Sep 20, 2017
node.go remove MIGRATE_REQUEST and VALIDATE_MIGRATE_REQUEST May 26, 2018
node_test.go fix broken tests May 30, 2018
node_test_util.go fix broken tests May 30, 2018
notifi.go lint gx May 24, 2018
notifi_test.go refactor multi-node testing setup/teardown Sep 27, 2017
nucleus.go fix cryptic errors from DNA file not containing PropertiesSchemaFile Jun 5, 2018
nucleus_test.go fix double put, and test for mod replaces loops, #444 Oct 8, 2017
package.json downcase github url Mar 21, 2018
revocation.go lint gx May 24, 2018
revocation_test.go added revocation interface with self-revocation implementation, #281 Jul 24, 2017
ribosome.go hash and type validation for MigrateEntry May 26, 2018
ribosome_test.go validate functions now allow return a string failed reason, #530 Mar 16, 2018
service.go Merge pull request #749 from holochain/747-properties-schema-cryptic-… Jun 13, 2018
service_test.go fixed hardcoded / in path May 29, 2018
test.go fix tests May 27, 2018
test_data.go genTestStringHash returns "Qm" hashes now May 30, 2018
test_data_test.go genTestStringHash returns "Qm" hashes now May 30, 2018
utils.go merge branch develop Mar 16, 2018
utils_test.go safer TestTicker without async problems Apr 30, 2018
validate.go remove MIGRATE_REQUEST and VALIDATE_MIGRATE_REQUEST May 26, 2018
validate_test.go downcase github url Mar 21, 2018
warrant.go lint gx May 24, 2018
warrant_test.go lint gx May 24, 2018
world.go lint gx May 24, 2018
world_test.go lint gx May 24, 2018
zome.go updated from/to language to caller/callee for bridging, closes #661 Apr 30, 2018
zome_test.go refactor cleanup code in test Sep 27, 2017
zygosome.go fix tests Jun 13, 2018
zygosome_test.go fix tests Jun 13, 2018

README.md

Holochain

Code Status Travis Codecov Go Report Card License: GPL v3 Twitter Follow

Code Status

Alpha. Not for production use. The code has not yet undergone a security audit. We expect to destructively restructure code APIs and data chains until Beta.

NOT CURRENT REPOSITORY Holochain is being rebuilt in Rust, and this Go codebase has minimal maintenance while that process is underway. Please see the NEW REPOSITORY with the Rust version for updates.

Holographic storage for distributed applications

Holochain uses a monotonic distributed hash table (DHT) where every node enforces validation rules on data before publishing that data against the signed chains where the data originated.

In other words, Holochain apps function very much like a blockchain without bottlenecks when it comes to enforcing validation rules, but is designed to be fully distributed with each node only needing to hold a small portion of the data instead of everything needing a full copy of a global ledger. This makes it feasible to run blockchain-like applications on devices as lightweight as mobile phones.

History

Proof-of-concept was unveiled at our first hackathon (March 2017). Alpha 0 was released October 2017. Alpha 1 was released May 2018.

Holochain Links: FAQ Developer Wiki White Paper GoDocs

Table of Contents

Installation

Developers Only: At this stage, holochain is only for use by developers (either developers of applications to run on holochains, or developers of the holochain software itself). App developers should bundle their app in an installer as either approach below is not for non-technical folks.

There are two approaches to installing holochain:

  1. as a standard Go language application for direct execution on your machine
  2. using docker for execution in a container.

Which you choose depends on your preference and your purpose. If you intend to develop holochain applications, then you should almost certainly use the docker approach as we provide a testing harness for running multiple holochain instances in a docker cluster. If you will be developing in Go on holochain itself then you will probably end up doing both.

Quick Install

  1. Download the most recent release (https://github.com/holochain/holochain-proto/releases/) for your OS.
  2. Unzip it to your directory of choice (or compile it from source).
  3. Add that directory to your PATH if you want to call holochain-proto commands outside of that directory (this step differs by OS).
  4. The commands listed in the 'Usage' section should now be available.

Go Based Install

Unix

(Unix includes macOS and Linux.)

  1. Download Go. Download the "Archive" or "Installer" for version 1.8 or later for your CPU and OS. The "Source" download does not contain an executable and step 3 will fail.

  2. Install Go on your system. See platform specific instructions and hints below for making this work.

  3. Setup your path (Almost all installation problems that have been reported stem from skipping this step.)

    • Export the $GOPATH variable in your shell profile.
    • Add $GOPATH/bin to your $PATH in your shell profile.

    For example, add the following to the end of your shell profile (usually ~/.bashrc or ~/.bash_profile):

        export GOPATH="$HOME/go"
        export PATH="$GOPATH/bin:$PATH"
  1. Install the command line tool suite with:
$ go get -d -v github.com/holochain/holochain-proto
$ cd $GOPATH/src/github.com/holochain/holochain-proto
$ make
  1. Test that it works (should look something like this):
$ hcadmin -v
hcadmin version 0.0.x (holochain y)

Windows

First you'll need to install some necessary programs if you don't already have them.

  • Download Go. Download the "Archive" or "Installer" for version 1.8 or later for Windows and your CPU. The "Source" download does not contain an executable.
  • Install Windows git. Be sure to select the appropriate options so that git is accessible from the Windows command line.
  • Optional: Install GnuWin32 make.

Next, in your Control Panel, select System>Advanced system settings>Environment Variables... and under System Variables do the following:

  1. Add a new entry with the name GOPATH and the value %USERPROFILE%\go (Or your Go workspace folder).
  2. Double-click Path, and in the window that pops up add the following entries:
    • %GOPATH%\bin
    • C:\Go\bin (Or wherever you installed Go to+\bin).
    • C:\Program Files (x86)\GnuWin32\bin (Or wherever you installed GnuWin32 make to+\bin).

Docker Based Install

Using docker, you don't have to install Go first. Our docker scripts manage installation of Go, holochain dependencies and holochain. The docker installation can run alongside Local ("Go") installation of holochain, sharing config directories. See docker usage on our wiki for more on how this works.

  1. Install the latest version of Docker on your machine

    1. Docker Installation. The Community edition; stable is sufficient.
    2. See Docker Getting Started for help.
    3. It is recommended to add your user to the docker group as in: Post Installation Steps, rather than use sudo before all script commands. Holochain Apps cannot exploit the kinds of security concerns mentioned in the Post Installation Steps document.  
  2. Confirm that docker installation and permissions are working by running:

    	$ docker info
  3. Pull our holochain image from docker hub:

    	$ docker pull holochain/holochain-proto:develop
  4. To run holochain in your new environment, suitable to continue the walkthrough below in usage

    	$ docker run --rm -it --name clutter -p 3141:3141 holochain/holochain-proto:develop
  5. This will put you into an new command shell that may behave differently than what you're used to. To exit this holochain (Alpine) shell, press Ctrl-D or type exit

Usage

These instructions are for using the holochain command line tool suite: hcadmin, hcdev and hcd. They should work equally well for Go based or docker based installation.

(Note that since Holochain is intended to be used behind distributed applications, end users should not have to do much through the command or may not have it installed at all, as the application will probably have wrapped up the holochain library internally.)

Each of the tools includes a help command, e.g., run hcadmin help or for sub-commands run hcadmin <COMMAND> help. For more detailed information, see the wiki page

The tool suite include these commands:

  • hcadmin for administering your installed holochain applications
  • hcd for running and serving a holochain application
  • hcdev for developing and testing holochain applications

Getting Started

The instructions below walk you through the basic steps necessary to run a holochain application.

Initializing the Holochain environment

	$ hcadmin init 'your@emailaddress.here'

This command creates a ~/.holochain directory for storing all chain data, along with initial public/private key pairs based on the identity string provided as the second argument.

Joining a Holochain

You can use the hcadmin tool to join a pre-existing Holochain application by running the following command (replacing SOURCE_PATH with a path to an application's DNA and CHAIN_NAME with the name you'd like it to be stored as).

For example: hcadmin join ./examples/chat chat

Note: this command will be replaced by a package management command still in development.

Running a Holochain

Holochains run and serve their UI via local web sockets. This allows interface developers lots of freedom to build HTML/JavaScript files and drop them in that chain's UI directory. You start a holochain and activate it's UI with the hcd command:

$ hcd <CHAIN_NAME> [PORT]

Developing a Holochain

The hcdev tool allows you to:

  1. generate new holochain application source files by cloning from an existing application, from a package file, or a simple empty template.
  2. run stand-alone or multi-node scenario tests
  3. run a holochain and serve it's UI for testing purposes
  4. dump out chain and dht data for inspection

Please see the docs for more detailed documentation.

Note that the hcdev command creates a separate ~/.holochaindev directory for serving and managing chains, so your dev work won't interfere with any running holochain apps you may be using.

Test-driven Application Development

We have designed Holochain around test-driven development, so the DNA should contain tests to confirm that the rest of the DNA is functional. Our testing harness includes two types of testing, stand-alone and multi-instance scenarios. Stand-alone tests allow you to tests the functions you create in your application. However, testing a distributed application requires being able to spin up many instances of it and have them interact. Our docker cluster testing harness automates that process, and enables app developers to specify scenarios and roles and test instructions to run on multiple docker containers.

Please see the App Testing documentation for details.

File Locations

By default holochain data and configuration files are assumed to be stored in the ~/.holochain directory. You can override this with the -path flag or by setting the HOLOPATH environment variable, e.g.:

$ hcadmin -path ~/mychains init '<my@other.identity>'
$ HOLOPATH=~/mychains hcadmin

You can use the form: hcadmin -path=/your/path/here but you must use the absolute path, as shell substitutions will not happen.

Logging

All the commands take a --debug flag which will turn on a number of different kinds of debugging. For running chains, you can also control exactly which of these logging types you wish to see in the chain's config.json file. You can also set the HCDEBUG environment variable to 0 or 1 to temporarily override your settings to turn everything on or off. See also the Environment Variable documentation for more granular logging control.

Architecture Overview and Documentation

Architecture information and application developer documentation is in our developer.holochain.org.

You can also look through auto-generated reference API on GoDocs

Holochain Core Development

We accept Pull Requests and welcome your participation. Please make sure to include the issue number your branch names and use descriptive commit messages.

Contribute

Contributors to this project are expected to follow our development protocols & practices.

Getting started

Once you have followed the basic "getting started" guide above you will have the CLI tools installed locally.

All the commands (hcadmin, hcd, hcdev, etc.) are built from the same repo:

$ cd $GOPATH/src/github.com/holochain/holochain-proto

Go will throw errors complaining about not being on the $GOPATH if you try to run make from a separate copy of the holochain-proto repository. If you want to contribute to Holochain core you must work in the repository created by Go.

The Makefile contains all the build commands for Holochain. If you make an update to a command you will need to rebuild it before the changes take effect at the command line.

E.g. After making an update to cmd/hcdev/hcdev.go run $ make hcdev then run $ hcdev as normal.

Dependencies

This project depends on various parts of libp2p, which uses the gx package manager. All of which will be automatically installed by make by following the setup instructions above.

The package manager rewrites files that are tracked by git to configure imports. Be careful not to commit the generated imports to git!

make work adds the imports to the repository and make pub reverts them.

Every make command should automatically add and remove imports for you. If a make command is leaving mess behind in the repo, please open a bug report.

If you want to use go commands directly (e.g. go test) then you need to run make work manually first and remember to make pub before committing any changes.

Tests

To compile and run all the tests:

$ cd $GOPATH/src/github.com/holochain/holochain-proto
$ make test

go test can be used instead of make test, but only after make work.

The docker setup runs tests automatically during builds.

License

License: GPL v3

Copyright (C) 2018, The MetaCurrency Project (Eric Harris-Braun, Arthur Brock, et. al.)

This program is free software: you can redistribute it and/or modify it under the terms of the license provided in the LICENSE file (GPLv3). This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Note: We are considering other 'looser' licensing options (like MIT license) but at this stage are using GPL while we're getting the matter sorted out.

Acknowledgements

  • MetaCurrency & Ceptr: Holochains are a sub-project of Ceptr which is a semantic, distributed computing platform under development by the MetaCurrency Project.  
  • Ian Grigg: Some of our initial plans for this architecture were inspired in 2006 by his paper about Triple Entry Accounting and his work on Ricardian Contracts.  
  • Juan Benet & the IPFS team: For all their work on IPFS, libp2p, and various cool tools like multihash, multiaddress, etc. We use libP2P library for our transport layer and kademlia dht.  
  • Crypto Pioneers And of course the people who paved the road before us by writing good crypto libraries and preaching the blockchain gospel. Back in 2008, nobody understood what we were talking about when we started sharing our designs. The main reason people want it now, is because blockchains have opened their eyes to the power of decentralized architectures.