README-dev.md

Coda

Coda is a new cryptocurrency protocol with a lightweight, constant-sized blockchain.

• Developer homepage
• Roadmap
• Repository Readme

If you haven't seen it yet, CONTRIBUTING.md has information about our development process and how to contribute. If you just want to build Coda, this is the right file!

Building Coda

Building Coda can be slightly involved. There are many C library dependencies that need to be present in the system, as well as some OCaml-specific setup.

Currently, Coda builds/runs on Linux & macOS. MacOS may have some issues that you can track here.

The short version:

1. Start with Ubuntu 18 or run it in a virtual machine
2. Set github repos to pull and push over ssh: git config --global url.ssh://git@github.com/.insteadOf https://github.com/
   • To push branches to repos in the CodaProtocol or o1-labs organisations, you must complete this step. These repositories do not accept the password authentication used by the https URLs.
3. Pull in our submodules: git submodule update --init
   • This might fail with git@github.com: Permission denied (publickey).. If that happens it means you need to set up SSH keys on your machine.
4. Install Docker, GNU make, and bash
5. make USEDOCKER=TRUE dev
6. make USEDOCKER=TRUE deb

Now you'll have a src/_build/codaclient.deb ready to install on Ubuntu or Debian!

You should also run:

7. git config --local --add submodule.recurse true

so that the submodules get updated automatically when updating your local copy of the repo.

Developer Setup (MacOS)

• Invoke make macos-setup
• Invoke make build
• Jump to customizing your editor for autocomplete

Developer Setup (Linux)

Install or have Ubuntu 18

• VM Images

Setup Docker CE on Ubuntu

• Ubuntu Setup Instructions

Toolchain docker image

• Pull down developer container image (~2GB download, go stretch your legs)

docker pull codaprotocol/coda:toolchain-9924f4c56a40d65d36440e8f70b93720f29ba171

• Create local builder image

make codabuilder

• Start developer container

make containerstart

• Update OPAM packages

make USEDOCKER=TRUE update-opam

• Start a build (go stretch your arms)

make USEDOCKER=TRUE build

Customizing your dev environment for autocomplete/merlin

• If you build in Docker, the files created for merlin will have invalid paths. You can fix those paths after a build

make USEDOCKER=TRUE merlin-fixup

• If you use vim, add this snippet in your vimrc to use merlin. (REMEMBER to change the HOME directory to match yours)

let s:ocamlmerlin="/Users/USERNAME/.opam/4.07/share/merlin"
execute "set rtp+=".s:ocamlmerlin."/vim"
execute "set rtp+=".s:ocamlmerlin."/vimbufsync"
let g:syntastic_ocaml_checkers=['merlin']

• In your home directory opam init

• In this shell, eval $(opam config env)

• Now /usr/bin/opam install merlin ocp-indent core async ppx_jane ppx_deriving (everything we depend on, that you want autocompletes for) for doc reasons

• Make sure you have au FileType ocaml set omnifunc=merlin#Complete in your vimrc

• Install an auto-completer (such as YouCompleteMe) and a syntastic (such syntastic or ALE)

• If you use vscode, you might like these extensions

  • OCaml and Reason IDE
  • Dune

• If you use emacs, besides the opam packages mentioned above, also install tuareg, and add the following to your .emacs file:

(let ((opam-share (ignore-errors (car (process-lines "opam" "config" "var" "share")))))
  (when (and opam-share (file-directory-p opam-share))
    ;; Register Merlin
    (add-to-list 'load-path (expand-file-name "emacs/site-lisp" opam-share))
    (load "tuareg-site-file")
    (autoload 'merlin-mode "merlin" nil t nil)
    ;; Automatically start it in OCaml buffers
    (add-hook 'tuareg-mode-hook 'merlin-mode t)
    (add-hook 'caml-mode-hook 'merlin-mode t)))

Emacs has a built-in autocomplete, via M-x completion-at-point, or simply M-tab. There are other Emacs autocompletion packages; see Emacs from scratch.

Using the makefile

The makefile contains phony targets for all the common tasks that need to be done. It also knows how to use Docker automatically. If you have USEDOCKER=TRUE in your environment, or run make USEDOCKER=TRUE, it will do the real work inside a container. You should probably use USEDOCKER=TRUE unless you've done the building without docker steps.

These are the most important make targets:

• build: build everything
• docker: build the container
• container: restart the development container (or start it if it's not yet)
• dev: does docker, container, and build
• test: run the tests
• libp2p_helper: build the libp2p helper
• web: build the website, including the state explorer

We use the dune buildsystem for our OCaml code.

NOTE: all of the test-* targets (including test-all) won't run in the container. test wraps them in the container.

Building outside docker

Coda has a variety of opam and system dependencies.

You can see Dockerfile-toolchain for how we install them all in the container. To get all the opam dependencies you need, you run opam switch import src/opam.export.

Some of our dependencies aren't taken from opam, and aren't integrated with dune, so you need to add them manually, by running scripts/pin-external-packages.sh.

There are a variety of C libraries we expect to be available in the system. These are also listed in the dockerfiles. Unlike most of the C libraries, which are installed using apt in the dockerfiles, the libraries for RocksDB are automatically installed when building Coda via a dune rule in the library ocaml-rocksdb.

Steps for adding a new dependency

Rarely, you may edit one of our forked opam-pinned packages, or add a new system dependency (like libsodium). Some of the pinned packages are git submodules, others inhabit the git Coda repository.

If an existing pinned package is updated, either in the Coda repository or in the the submodule's repository, it will be automatically re-pinned in CI.

If you add a new package in the Coda repository or as a submodule, you must do all of the following:

1. Update Dockerfile-toolchain as required; there are comments that distinguish the treatment of submodules from other packages
2. Update scripts/macos-setup.sh with the required commands for Darwin systems
3. Bust the circle-ci Darwin cache by incrementing the version number in the cache keys as required inside .circleci/config.yml.jinja
4. Commit your changes
5. Rebuild the container with make docker-toolchain.
6. Re-render the jinja template make update-deps
7. Commit your changes again

Rebuilding the docker toolchain will take a long time. Running circleci for macos once you've busted the cache will also take a long time. However, only you have to do the waiting and all other developers will get the fast path.

The automatic re-pinning of modified packages does take some CI time, so eventually, you'll want to rebuild the Docker toolchain to save that time.

Common dune tasks

To run unit tests for a single library, do dune runtest lib/$LIBNAME.

You can use dune exec coda to build and run coda. This is especially useful in the form of dune exec coda -- integration-tests $SOME_TEST.

You might see a build error like this:

Error: Files src/lib/mina_base/mina_base.objs/account.cmx
       and src/lib/mina_base/mina_base.objs/token_id.cmx
       make inconsistent assumptions over implementation Crypto_params

You can work around it with rm -r src/_build/default/src/$OFFENDING_PATH and a rebuild. Here, the offending path is src/lib/mina_base/mina_base.objs.

Docker Image Family Tree

Container Stages:

• Stage 0: Initial Image ocaml/opam2:debian-9-ocaml-4.07 (opam community image, ~880MB)
• Stage 1: coda toolchain (built by us, stored on docker hub, ~2GB compressed)
• Stage 2: codabuilder (built with make codabuilder, used with make build, ~2GB compressed)

Overriding Genesis Constants

Coda genesis constants consists of constants for the consensus algorithm, sizes for various data structures like transaction pool, scan state, ledger etc. All the constants can be set at compile-time. A subset of the compile-time constants can be overriden when generating the genesis state using runtime_genesis_ledger.exe, and a subset of those can again be overridden at runtime by passing the new values to the daemon.

The constants at compile-time are set for different configurations using optional compilation. This is how integration tests/builds with multiple configurations are run. Currently some of these constants (defined here) cannot be changed after building and would require creating a new build profile (*.mlh files) for any change in the values.

1. Constants that can be overridden when generating the genesis state are:

• k (consensus constant)
• delta (consensus constant)
• genesis_state_timestamp
• transaction pool max size

To override the above listed constants, pass a json file to runtime_genesis_ledger.exe with the format:

{
  "k": 10,
  "delta": 3,
  "txpool_max_size": 3000,
  "genesis_state_timestamp": "2020-04-20 11:00:00-07:00"
}

The exe will then package the overriden constants along with the genesis ledger and the genesis proof for the daemon to consume.

2. Constants that can be overriden at runtime are:

• genesis_state_timestamp
• transaction pool max size

To do this, pass a json file to the daemon using the flag genesis-constants with the format:

{
  "txpool_max_size": 3000,
  "genesis_state_timestamp": "2020-04-20 11:00:00-07:00"
}

The daemon logs should reflect these changes. Also, coda client status displays some of the constants.