Provisioner for the Block R Network
This tool was created to help maintain the regional Block R processing nodes. Here are some notes:
- Each processing node consists of a single orderer and a single peer.
- Optimized for an odd number of nodes to facilitate consensus.
- Hyperledger Fabric will be installed on each node
- The network can be specified to communicate with or without TLS.
- Only one channel is defined.
- Tested with Centos 7, other distributions might need adjustments.
- Local git repo of Hyperledger Fabric in
- git client installed on the servers
- Ability to do "sudo" without having to enter a password on the servers. One way to do this is to add the user account to the
- Ability to ssh without having to enter a password from the machine that runs this script
- Ability of nodes to resolve each other's DNS names
- Installed and operatonal Apache Kafka at least at level 0.10.2.0
- Docker at least at level 17.09.0-ce
This appears to be a complex section, but it is less painful if you are able to build a development system for Hyperledger using the project's instructions. Their instructions will help you identify all of the dependencies. Once you have a working system, the following (detail) instructions can be used to build the system and use the scripts. The
install.sh script referenced below can be used at anytime to rebuild Hyperledger components from the Master Branch.
- Configure you user accould to have sudo capabilities without being prompted for a password.
- Configure your server to execute SSH commands on all servers that will run compoments without being prompted for a password.
- Make sure that the HOSTNAME environmental variable is set for all servers and that their static addresses are captured in the
/etc/hostsfiles on each server.
- Install Kafaka version 0.10.2. This also contains zookeeper.
- Install the systemd definitions in the /scripts directory for zooeeper and kafka.
- There is no package for CouchDB on Centos 7, so find the package on the web and build from source. Use all of the default settings.
- Run the
install.shscript to setup the environment and install Hyperledger Fabric.
- By default, the
MASTER_BRANCHvariable is set to
true. If you do not want to work from the Hyperledger master branch, set this variable to
- Setting the
truewill leave a copy of script in the home directory to give you a chance to instapect the file.
- Copy the
config.sh-samplefile in the root directory to `config.sh'
config.shin the root directory with the following information:
- zookeepers: a space delimited list of server names that run zookeeper. For example, "vm1 vm2".
- zookeeper_accounts: a space delimited list of accounts that run zookeeper. For example, "user1 user2a. When used with
zookeepersfrom above, can be used for ssh arguments like "user1@vm1" and user2@vm2"
- nodes: a space delimited list of server names operating regional nodes. For eample, "west1 central1 east2".
- accounts: a space delimited list of accounts used to administrate regional nodes. For example, "user_west user_central user_east". When used with
nodesabove, can be used for ssh arguments like "user_west@west1", "user_central@central1" and "user_east@east2".
- domains: a space delimited list of domain names representing regional nodes. For example, "west.blockr central.blockr east.blockr".
- orderer_names: a space delimited list of names representing orderers. For example, "WEST-Orderer-0 CENTRAL-Orderer-0 EAST-Orderer-0".
- orderers: a space delimited list of names representing orderer MSPs. For example, "Orderer1MSP Orderer2MSP Orderer3MSP".
- peer_names: a space delimited list of names representing peers. For example, "WEST-Peer-0 CENTRAL-Peer-0 EAST-Peer-0".
- peers: a space delimited list of names representing peer MSPs. For example, "Org1MSP Org2MSP Org3MSP".
- Provision the system using the instructions below.
The following primary scripts can be found in the root directory:
config.sh- configuration file
create_channel.sh- adds the peer on each node to the "blockr" channel
install.sh- install Hyperledget Fabric and systyemd scripts
install_chaincode.sh- installs chaincode on each node
provision.sh- performs the provisioning
run.sh- interactive shell to run scripts
start_nodes.sh- starts both peer and orderer daemons on each node of the system.
stop_nodes.sh- stops both peer and orderer daemons on each node of the system.
There is also a
scripts directory that contains:
common.sh- shared functions
invoke.sh- used to modify contents of the edger for testing
kafka.service- systemd service for Kafka
list_channels.sh- lists the channels that the peer is subscribed to
query.sh- used to inspect the contents of the ledger during testing
zookeeper.service- systemd service for Zookeeper
Configuration templates are found in the
templates directory. These are used to create operational configurations from the definitions provided in the
config.sh file. They contain default settings:
core.yaml- peer configuration template file
orderer.yaml- orderer configuration template file
server.properties- kafka configuration template file
zookeeper.properties- zookeeper configuration template file
Two other files files are included in the
templates directory for reference purposes only. They are not used but can be used to understand what the
provision.sh script is trying to build. They are auto-generated. If you work with the Hyperledger official documentation, these files will be familiar. They represent a two node system.
blockr-config.yaml- network configuration file
configtx.yaml- genesis block and channel configuration file
You can either execute scripts from the
run.sh interactive script or run them individually. If you run them individually, read the rest of this section carefully.
Here is the sequence of operations neeeded to install a network. It is important that theses scripts are run in order. They have been designed to capture particularly sensitive sequences inside of scripts to avoid problems. For instance, the sequence of operations needed to bring up Zookeeper, Kafka and CouchDB daemons is sensitive and thus are contained with the
provision.sh- performs three operations:
- creates configuration information from the
config.shfile. It creates a
- creates blocks and transactions required for the
- packages validation chaincode using the first node specified in the
config.shfile. Make sure you can ssh without passwords from the first node in the
consif.shto all nodes to avoid being prompted for a password.
- distributes configuration information and chaincode to each server.
- creates configuration information from the
create_channel.sh- performs three operations on the peer of each node:
- defines the
- joins the
- defines the peer as an
AnchorPeerto facilitate Hyperledger gossip communications
- defines the
install_chaincode.sh- installs chaincode on each node.
Once the nodes are running, each node can be tested using the
When you are done testing, use the
stop_nodes.sh script to stop the nodes.
All configuration is conducted on files in the
templates directory. The distribution is pre-configured for a two node system. These nodes are called
vm2. You can add more servers by modifying
In order to test run the system, set up two Centos 7 servers and define their addresses in
Each node should have the following components:
- Hyperledger Fabric source code
- Peer instance
- Orderer instance
- CouchDB instance
- Kafka instance
A primary and secondary Zookeeper instance should be defined on seperate servers. The two Zookeeper server do not have to be the same servers used for Hyperledger and in production, you probably would not want them on the same servers anyway.
Other configurations are possible, but this approach results in a regional processing center that can serve to meet the falut-tolerant, high-availability system for membership information. If one regional center is down, others still operate and take over the load. Sensing which nodes are operational is the job of Zookeeper. Kafka provides a queue of messages that allow a node to rejoin a network and "catch up" with messages that have been transmitted while absent.
Here on notes for changing the pre-configured network:
- All configurations and subsequent provisioning should be made on a single computer. This server will be used to provision other servers.
- Ensure that all HOSTNAME environmental variables are set and that they are defined in the
/etc/hsotsfiles of each server.
core.yamlfile contains configuration information for each PEER. It is used as a template for the
provision.shscript so be careful what you change. Elements that are used for templating are in all caps such as PEER_ID, PEER_ADDRESS and PEER_BOOTSTRAP. The top of the file contains log settings. The definition for talking to the underlying CouchDB process is also contained in this file. You should not have to change thigs here, but have fun tuning and please report your results!
orderer.yamlfile contains configuration information for orderers. The primary concerns here are the location of underlying database (not how to talk to CouchDB, but where the repository is located) and how to talk to Kafka. Kafka is a queue of messages between the nodes.
server.propertiesfile is a template file for the Kafka daemon that runs on each node. Be careful editing this file because settings in all caps (like BROKER_ID and SERVER_ADDRESS) are modified by the `provision.sh' script. Feel free to adjust parameters to improve performance and report your improvements.
zookeeper.propertiesfile contains configuration parameters for the zookeer instance that runs on each node. There is very little to configure in this file.
It is in everyone's interest to report changes and improvements in configurtion. Remember to post your results.
Generated Configuration Files
Two files controlling a Hyperledger setup are auto-configured by the contents of the
config.sh file. They are created and distributed by the
blockr-config.yaml- Contains the names and locations of the various peers and orderers in your system.
configtx.yaml- Contains confiigurations for both the genesis block of the system as well as any channels you would like to define.
- Profiles section: You will find the obvious names
Channels. If you would like to define multiple channels, create another section under
Channelsas a pattern.
- Organizations section: Defines the various servers and which encryption keys to use for each. If you would like a node to have nultiple peers (in order to handle heavy transaction loads), use the AnchorPeers section to point to identify the server that takes the lead when exchanging information with other servers.
- Orderer: This section defines how Orderers exchange information and the addresses of each orderer. We are usinf Kafka to enable communication between Orderes, so the address of each Kafka server needs to be defined.
- Application: Special configuration information for applications. No applications configurations are defined in the distribution, but you should have this section defined.
- Capabilities: The distribution ensures that all processing is conducted with Hyyperledger V1.1 capabilities. These capabilities are defined in the distribution, but you should have this section defined.
- Profiles section: You will find the obvious names
The long term goal of this project is to be able to modify these files once created.
The script is destructive to any server running Hyperledger Fabric. It resets
/var/hyperledger/productionand deletes docker images. Only run this script on VMs / servers created solely for Block R installation.
This work is a collaborative effort of six REALTOR(R) Associations.