[FAB-14086] Chaincode launcher doc

Change-Id: I78dd2dac59e2343b2082fcef3f4c2b31a764f175
Signed-off-by: joe-alewine <Joe.Alewine@ibm.com>

docs/source/cc_launcher.md

@@ -0,0 +1,301 @@
+# External Builders and Launchers
+
+Prior to Hyperledger Fabric 2.0, the process used to build and launch
+chaincode was part of the peer implementation and could not be easily
+customized. All chaincode instantiated on the peer would be "built" using
+language specific logic hard-coded in the peer. This build process would
+generate a Docker container image that would be launched to execute chaincode
+that connected as a client to the peer.
+
+This approach limited chaincode implementations to a handful of languages,
+required Docker to be part of the deployment environment, and prevented
+running chaincode as a long running server process.
+
+External Builders and Launchers address these limitations by enabling
+operators to extend the peer with programs that can build, launch, and
+discover chaincode.
+
+## External Builder Model
+
+Hyperledger Fabric External Builders and Launchers are loosely based on Heroku
+[Buildpacks][buildpacks]. A buildpack implementation is simply a collection of
+programs or scripts that transform application artifacts into something that
+can run. The buildpack model has been adapted for chaincode packages and
+extended to support chaincode execution and discovery.
+
+### External Builder and Launcher API
+
+An external builder and launcher consists of four programs or scripts:
+
+- `bin/detect`: Determine whether or not this buildpack should be used to
+  build the chaincode package and launch it.
+- `bin/build`: Transform the chaincode package into executable chaincode.
+- `bin/release`: Provide metadata to the peer about the chaincode. (optional)
+- `bin/run`: Run the chaincode.  (optional)
+
+#### `bin/detect`
+
+The `bin/detect` script is responsible for determining whether or not a buildpack
+should be used to build a chaincode package and launch it. The peer invokes
+`detect` with two arguments:
+
+```sh
+bin/detect CHAINCODE_SOURCE_DIR CHAINCODE_METADATA_DIR
+```
+
+When `detect` is invoked, `CHAINCODE_SOURCE_DIR` contains the chaincode source
+and `CHAINCODE_METADATA_DIR` contains the `metadata.json` file from the
+chaincode package installed to the peer. If the buildpack should be applied to
+the chaincode source package, `detect` must return an exit code of `0`; any
+other exit code will indicate that the buildpack should not be applied.
+
+The following is an example of a simple `detect` script for go chaincode:
+```sh
+#!/bin/bash
+
+CHAINCODE_METADATA_DIR="$2"
+
+# use jq to extract the chaincode type from metadata.json and exit with
+# success if the chaincode type is golang
+if [ "$(jq -r .type "$CHAINCODE_METADATA_DIR/metadata.json" | tr '[:upper:]' '[:lower:]')" = "golang" ]; then
+    exit 0
+fi
+
+exit 1
+```
+
+#### `bin/build`
+
+The `bin/build` script is responsible for building, compiling, or transforming
+the contents of a chaincode package into artifacts that can be used by
+`release` and `run`. The peer invokes `build` with three arguments:
+
+```sh
+bin/build CHAINCODE_SOURCE_DIR CHAINCODE_METADATA_DIR BUILD_OUTPUT_DIR
+```
+
+When `build` is invoked, `CHAINCODE_SOURCE_DIR` contains the chaincode source
+and `CHAINCODE_METADATA_DIR` contains the `metadata.json` file from the
+chaincode package installed to the peer. `BUILD_OUTPUT_DIR` is the directory
+where `build` must place artifacts needed by `release` and `run`.
+
+When `build` completes with an exit code of `0`, the contents of
+`BUILD_OUTPUT_DIR` will be copied to the persistent storage maintained by the
+peer; any other exit code will be considered a failure.
+
+The following is an example of a simple `build` script for go chaincode:
+```sh
+#!/bin/bash
+
+CHAINCODE_SOURCE_DIR="$1"
+CHAINCODE_METADATA_DIR="$2"
+BUILD_OUTPUT_DIR="$3"
+
+# extract package path from metadata.json
+GO_PACKAGE_PATH="$(jq -r .path "$CHAINCODE_METADATA_DIR/metadata.json")"
+if [ -f "$CHAINCODE_SOURCE_DIR/src/go.mod" ]; then
+    cd "$CHAINCODE_SOURCE_DIR/src"
+    go build -v -mod=readonly -o "$BUILD_OUTPUT_DIR/chaincode" "$GO_PACKAGE_PATH"
+else
+    GO111MODULE=off go build -v  -o "$BUILD_OUTPUT_DIR/chaincode" "$GO_PACKAGE_PATH"
+fi
+
+# save statedb index metadata to provide at release
+if [ -d "$CHAINCODE_SOURCE_DIR/META-INF" ]; then
+    cp -a "$CHAINCODE_SOURCE_DIR/META-INF" "$BUILD_OUTPUT_DIR/"
+fi
+```
+
+#### `bin/release`
+
+The `bin/release` script is responsible for providing metadata chaincode to
+the peer. The peer invokes `release` with two arguments:
+
+```sh
+bin/release BUILD_OUTPUT_DIR RELEASE_OUTPUT_DIR
+```
+
+When `release` is invoked, `BUILD_OUTPUT_DIR` contains the artifacts populated
+by the `build` program. `RELEASE_OUTPUT_DIR` is the directory where `release`
+must place artifacts to be consumed by the peer.
+
+When `release` completes, the peer will consume two types of metadata from
+`RELEASE_OUTPUT_DIR`:
+
+- state database index definitions for CouchDB
+- external chaincode server connection information (`chaincode/server/connection.json`)
+
+If CouchDB index definitions required for the chaincode, `release` is
+responsible for placing the indexes into the `statedb/couchdb/indexes`
+directory under `RELEASE_OUTPUT_DIR`. The indexes must have a `.json`
+extension.  See the [CouchDB indexes][couchdb-indexes] documentation for
+details.
+
+In cases where a chaincode server implementation is used, `release` is
+responsible for populating `chaincode/server/connection.json` with the address
+of the chaincode server and any TLS assets required to communicate with the
+chaincode. When server connection information is provided to the peer, `run`
+will not be called. See the [Chaincode Server][chaincode-server]
+documentation for details.
+
+The following is an example of a simple `release` script for go chaincode:
+```sh
+#!/bin/bash
+
+BUILD_OUTPUT_DIR="$1"
+RELEASE_OUTPUT_DIR="$2"
+
+# copy indexes from META-INF/* to the output directory
+if [ -d "$BUILD_OUTPUT_DIR/META-INF" ] ; then
+   cp -a "$BUILD_OUTPUT_DIR/META-INF/"* "$RELEASE_OUTPUT_DIR/"
+fi
+```
+
+#### `bin/run`
+
+The `bin/run` script is responsible for running chaincode. The peer invokes
+`run` with two arguments:
+
+```sh
+bin/run BUILD_OUTPUT_DIR RUN_METADATA_DIR
+```
+
+When `run` is called, `BUILD_OUTPUT_DIR` contains the artifacts populated by
+the `build` program and `RUN_METADATA_DIR` is populated with a file called
+`chaincode.json` that contains the information necessary for chaincode to
+connect and register with the peer. The keys included in `chaincode.json` are:
+
+- `chaincode_id`: The unique ID associated with the chaincode package.
+- `peer_address`: The address in `host:port` format of the `ChaincodeSupport`
+  gRPC server endpoint hosted by the peer.
+- `client_cert`: The PEM encoded TLS client certificate generated by the peer
+  that must be used when the chaincode establishes its connection to the peer.
+- `client_key`: The PEM encoded client key generated by the peer that must be
+  used when the chaincode establishes its connection to the peer.
+- `root_cert`: The PEM encoded TLS root certificate for the `ChaincodeSupport`
+  gRPC server endpoint hosted by the peer.
+
+When `run` terminates, the peer considers the chaincode terminated. If another
+request arrives for the chaincode, the peer will attempt to start another
+instance of the chaincode by invoking `run` again. The contents of
+`chaincode.json` must not be cached across invocations.
+
+The following is an example of a simple `run` script for go chaincode:
+```sh
+#!/bin/bash
+
+BUILD_OUTPUT_DIR="$1"
+RUN_METADATA_DIR="$2"
+
+# setup the environment expected by the go chaincode shim
+export CORE_CHAINCODE_ID_NAME="$(jq -r .chaincode_id "$RUN_METADATA_DIR/chaincode.json")"
+export CORE_PEER_TLS_ENABLED="true"
+export CORE_TLS_CLIENT_CERT_FILE="$RUN_METADATA_DIR/client.crt"
+export CORE_TLS_CLIENT_KEY_FILE="$RUN_METADATA_DIR/client.key"
+export CORE_PEER_TLS_ROOTCERT_FILE="$RUN_METADATA_DIR/root.crt"
+
+# populate the key and certificate material used by the go chaincode shim
+jq -r .client_cert "$RUN_METADATA_DIR/chaincode.json" > "$CORE_TLS_CLIENT_CERT_FILE"
+jq -r .client_key  "$RUN_METADATA_DIR/chaincode.json" > "$CORE_TLS_CLIENT_KEY_FILE"
+jq -r .root_cert   "$RUN_METADATA_DIR/chaincode.json" > "$CORE_PEER_TLS_ROOTCERT_FILE"
+if [ -z "$(jq -r .client_cert "$RUN_METADATA_DIR/chaincode.json")" ]; then
+    export CORE_PEER_TLS_ENABLED="false"
+fi
+
+# exec the chaincode to replace the script with the chaincode process
+exec "$BUILD_OUTPUT_DIR/chaincode" -peer.address="$(jq -r .peer_address "$ARTIFACTS/chaincode.json")"
+```
+
+## Configuring External Builders and Launchers
+
+Configuring the peer to use external builders involves adding a configuration
+block to `core.yaml` that defines external builders. Each external builder
+definition must include a name (used for logging) and the path to parent of
+the `bin` directory containing the builder scripts.
+
+An optional list of environment variable names to propagate from the peer when
+invoking the external builder scripts can also be provided.
+
+The following example defines two external builders:
+
+```yaml
+chaincode:
+  externalBuilders:
+  - name: my-golang-builder
+    path: /builders/golang
+    environmentWhitelist:
+    - GOPROXY
+    - GONOPROXY
+    - GOSUMDB
+    - GONOSUMDB
+  - name: noop-builder
+    path: /builders/binary
+```
+
+In this example, the implementation of "my-golang-builder" is contained within
+the `/builders/golang` directory and its build scripts are located in
+`/builders/golang/bin`. When the peer invokes any of the build scripts
+associated with "my-golang-builder", it will not propagate the values of any
+environment variables in the whitelist.
+
+> Note: The following environment variables are always propagated to external
+> builders:
+> - LD_LIBRARY_PATH
+> - LIBPATH
+> - PATH
+> - TMPDIR
+
+When an `externalBuilder` configuration is present, the peer will iterate over
+the list of builders in the order provided, invoking `bin/detect` until one
+completes successfully. If no builder completes `detect` successfully, the
+peer will fallback to using the legacy Docker build process implemented within
+the peer. This means that external builders are completely optional.
+
+In the example above, the peer will attempt to use "my-golang-builder",
+followed by "noop-builder", and finally the peer internal build process.
+
+# Chaincode Packages
+
+As part of the new lifecycle introduced with Fabric 2.0, the chaincode package
+format changed from serialized protocol buffer messages to a gzip compressed
+POSIX tape archive. Chaincode packages created with `peer lifecycle chaincode
+package` use this new format.
+
+## Lifecycle Chaincode Package Contents
+
+A lifecycle chaincode package contains two files. The first file,
+`code.tar.gz` is a gzip compressed POSIX tape archive. This file includes the
+source artifacts for chaincode. Packages created by the peer CLI will place
+the chaincode implementation source under the `src` directory and chaincode
+metadata (like CouchDB indexes) under the `META-INF` directory.
+
+The second file, `metadata.json` is a JSON document with three keys:
+- `type`: the chaincode type (e.g. GOLANG, JAVA, NODE)
+- `path`: for go chaincode, the GOPATH or GOMOD relative path to the main
+  chaincode package; undefined for other types
+- `label`: the chaincode label
+
+## Chaincode Packages and External Builders
+
+When a chaincode package is installed to a peer, the contents of `code.tar.gz`
+and `metadata.json` are not processed prior to calling external builders. This
+affords users a great deal of flexibility in how they package source and
+metadata that will be processed by external builders and launchers.
+
+For example, a custom chaincode package could be constructed that contains a
+pre-compiled, implementation of chaincode in `code.tar.gz` with a
+`metadata.json` that allows a _binary buildpack_ to detect the custom package,
+validate the hash of the binary, and run the program as chaincode.
+
+Another example would be a chaincode package that only contains state database
+index definitions and the data necessary for an external launcher to connect
+to a running chaincode server. In this case, the `build` process would simply
+extract the metadata from the process and `release` would present it to the
+peer.
+
+The only requirements are that `code.tar.gz` can only contain regular file and
+directory entries, and that the entries cannot contain paths that would result
+in files being written outside of the logical root of the chaincode package.
+
+If no configured external builder claims a chaincode package, the peer will
+attempt to process the package as if it were created with the peer CLI.

docs/source/ops_guide.rst

@@ -15,6 +15,7 @@ Operations Guides
    idemixgen
    operations_service
    metrics_reference
+   cc_launcher
    error-handling
    logging-control
    enable_tls