Skip to content

Commit

Permalink
Problem: It is cumbersome to manage BigchainDB and Tendermint process…
Browse files Browse the repository at this point in the history
…es (#2410).

* Introduce a Monit-based setup for managing processes. Add the corresponding section to the network setup guide.
  • Loading branch information
muawiakh authored and ldmberman committed Jul 27, 2018
1 parent 2f6413f commit c3f5e2a
Show file tree
Hide file tree
Showing 5 changed files with 227 additions and 7 deletions.
2 changes: 1 addition & 1 deletion bigchaindb/config_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ def file_config(filename=None):
'Failed to parse the JSON configuration from `{}`, {}'.format(filename, err)
)

logger.info('Configuration loaded from `{}`'.format(filename))
logger.info('Configuration loaded from `{}`'.format(filename))

return config

Expand Down
10 changes: 5 additions & 5 deletions bigchaindb/log.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@

from bigchaindb.common.exceptions import ConfigurationError
from logging.config import dictConfig as set_logging_config
from os.path import expanduser, join
import os


DEFAULT_LOG_DIR = expanduser('~')
DEFAULT_LOG_DIR = os.getcwd()
BENCHMARK_LOG_LEVEL = 15


Expand Down Expand Up @@ -40,7 +40,7 @@
},
'file': {
'class': 'logging.handlers.RotatingFileHandler',
'filename': join(DEFAULT_LOG_DIR, 'bigchaindb.log'),
'filename': os.path.join(DEFAULT_LOG_DIR, 'bigchaindb.log'),
'mode': 'w',
'maxBytes': 209715200,
'backupCount': 5,
Expand All @@ -49,7 +49,7 @@
},
'errors': {
'class': 'logging.handlers.RotatingFileHandler',
'filename': join(DEFAULT_LOG_DIR, 'bigchaindb-errors.log'),
'filename': os.path.join(DEFAULT_LOG_DIR, 'bigchaindb-errors.log'),
'mode': 'w',
'maxBytes': 209715200,
'backupCount': 5,
Expand All @@ -58,7 +58,7 @@
},
'benchmark': {
'class': 'logging.handlers.RotatingFileHandler',
'filename': 'bigchaindb-benchmark.log',
'filename': os.path.join(DEFAULT_LOG_DIR, 'bigchaindb-benchmark.log'),
'mode': 'w',
'maxBytes': 209715200,
'backupCount': 5,
Expand Down
47 changes: 46 additions & 1 deletion docs/server/source/simple-network-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,14 +220,18 @@ persistent_peers = "<Member 1 node id>@<Member 1 hostname>:26656,\
<Member N node id>@<Member N hostname>:26656,"
```

## Member: Start MongoDB, BigchainDB and Tendermint
## Member: Start MongoDB

If you installed MongoDB using `sudo apt install mongodb`, then MongoDB should already be running in the background. You can check using `systemctl status mongodb`.

If MongoDB isn't running, then you can start it using the command `mongod`, but that will run it in the foreground. If you want to run it in the background (so it will continue running after you logout), you can use `mongod --fork --logpath /var/log/mongodb.log`. (You might have to create the `/var/log` directory if it doesn't already exist.)

If you installed MongoDB using `sudo apt install mongodb`, then a MongoDB startup script should already be installed (so MongoDB will start automatically when the machine is restarted). Otherwise, you should install a startup script for MongoDB.

## Member: Start BigchainDB and Tendermint

If you want to use a process manager, jump to the [next section](member-start-bigchaindb-and-tendermint-using-monit).

To start BigchainDB, one uses the command `bigchaindb start` but that will run it in the foreground. If you want to run it in the background (so it will continue running after you logout), you can use `nohup`, `tmux`, or `screen`. For example, `nohup bigchaindb start 2>&1 > bigchaindb.log &`

The _recommended_ approach is to create a startup script for BigchainDB, so it will start right after the boot of the operating system. (As mentioned earlier, MongoDB should already have a startup script.)
Expand All @@ -240,6 +244,45 @@ Note: We'll share some example startup scripts in the future. This document is a

If you followed the recommended approach and created startup scripts for BigchainDB and Tendermint, then you can reboot the machine now. MongoDB, BigchainDB and Tendermint should all start.


### Member: Start BigchainDB and Tendermint using Monit

This section describes how to manage the BigchainDB and Tendermint processes using [Monit][monit] - a small open-source utility for managing and monitoring Unix processes.

This section assumes that you followed the guide down to the [start MongoDB section](#member-start-mongodb) inclusive.

Install Monit:

```
sudo apt install monit
```

If you installed the `bigchaindb` Python package, you should have the `bigchaindb-monit-config` script in your `PATH` now.

Run the script:

```
bigchaindb-monit-config
```

The script builds a configuration file for Monit.

Run Monit as a daemon, instructing it to wake up every second to check on processes:

```
monit -d 1
```

It will run the processes and restart them when they crash. If the root `bigchaindb_` process crashes, Monit will also restart the Tendermint process.

Check the status by running `monit status` or `monit summary`.

By default, it will collect program logs into the `~/.bigchaindb-monit/logs` folder.

Consult `monit -h` or [the Monit documentation][monit-manual] to know more about the operational power you've just got the taste of.

Check `bigchaindb-monit-config -h` if you want to arrange a different folder for logs or some of the Monit internal artifacts.

## How Others Can Access Your Node

If you followed the above instructions, then your node should be publicly-accessible with BigchainDB Root URL `http://hostname:9984` (where hostname is something like `bdb7.canada.vmsareus.net` or `17.122.200.76`). That is, anyone can interact with your node using the [BigchainDB HTTP API](http-client-server-api.html) exposed at that address. The most common way to do that is to use one of the [BigchainDB Drivers](./drivers-clients/index.html).
Expand Down Expand Up @@ -315,3 +358,5 @@ TBD.
[bdb:software]: https://github.com/bigchaindb/bigchaindb/
[bdb:pypi]: https://pypi.org/project/BigchainDB/#history
[tendermint:releases]: https://github.com/tendermint/tendermint/releases
[monit]: https://www.mmonit.com/monit
[monit-manual]: https://mmonit.com/monit/documentation/monit.html
173 changes: 173 additions & 0 deletions pkg/scripts/bigchaindb-monit-config
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
#!/bin/bash

set -o nounset

# Check if directory for monit logs exists
if [ ! -d "$HOME/.bigchaindb-monit" ]; then
mkdir -p "$HOME/.bigchaindb-monit"
fi

monit_pid_path=${MONIT_PID_PATH:=$HOME/.bigchaindb-monit/monit_processes}
monit_script_path=${MONIT_SCRIPT_PATH:=$HOME/.bigchaindb-monit/monit_script}
monit_log_path=${MONIT_LOG_PATH:=$HOME/.bigchaindb-monit/logs}
monitrc_path=${MONITRC_PATH:=$HOME/.monitrc}

function usage() {
cat <<EOM
Usage: ${0##*/} [-h]
Configure Monit for BigchainDB and Tendermint process management.
ENV[MONIT_PID_PATH] || --monit-pid-path PATH
Absolute path to directory where the the program's pid-file will reside.
The pid-file contains the ID(s) of the process(es). (default: ${monit_pid_path})
ENV[MONIT_SCRIPT_PATH] || --monit-script-path PATH
Absolute path to the directory where the executable program or
script is present. (default: ${monit_script_path})
ENV[MONIT_LOG_PATH] || --monit-log-path PATH
Absolute path to the directory where all the logs for processes
monitored by Monit are stored. (default: ${monit_log_path})
ENV[MONITRC_PATH] || --monitrc-path PATH
Absolute path to the monit control file(monitrc). (default: ${monitrc_path})
-h|--help
Show this help and exit.
EOM
}

while [[ $# -gt 0 ]]; do
arg="$1"
case $arg in
--monit-pid-path)
monit_pid_path="$2"
shift
;;
--monit-script-path)
monit_script_path="$2"
shift
;;
--monit-log-path)
monit_log_path="$2"
shift
;;
--monitrc-path)
monitrc_path="$2"
shift
;;
-h|--help)
usage
exit
;;
*)
echo "Unknown option: $1"
usage
exit 1
;;
esac
shift
done

# Check if directory for monit logs exists
if [ ! -d "$monit_log_path" ]; then
mkdir -p "$monit_log_path"
fi

# Check if directory for monit pid files exists
if [ ! -d "$monit_pid_path" ]; then
mkdir -p "$monit_pid_path"
fi

cat >${monit_script_path} <<EOF
#!/bin/bash
case \$1 in
start_bigchaindb)
pushd \$4
nohup bigchaindb start >> \$3/bigchaindb.out.log 2>> \$3/bigchaindb.err.log &
echo \$! > \$2
popd
;;
stop_bigchaindb)
kill -2 \`cat \$2\`
rm -f \$2
;;
start_tendermint)
pushd \$4
nohup tendermint node --consensus.create_empty_blocks=false >> \$3/tendermint.out.log 2>> \$3/tendermint.err.log &
echo \$! > \$2
popd
;;
stop_tendermint)
kill -2 \`cat \$2\`
rm -f \$2
;;
esac
exit 0
EOF
chmod +x ${monit_script_path}

# Handling overwriting of control file interactively
if [ -f "$monitrc_path" ]; then
echo "$monitrc_path already exists."
read -p "Overwrite[Y]? " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
echo "Overriding $monitrc_path/.monitrc"
else
read -p "Enter absolute path to store Monit control file: " monitrc_path
eval monitrc_path="$monitrc_path"
if [ ! -d "$(dirname $monitrc_path)" ]; then
echo "Failed to save monit control file '$monitrc_path': No such file or directory."
exit 1
fi
fi
fi

# configure monitrc
cat >${monitrc_path} <<EOF
set httpd
port 2812
allow localhost
check process bigchaindb
with pidfile ${monit_pid_path}/bigchaindb.pid
start program "${monit_script_path} start_bigchaindb $monit_pid_path/bigchaindb.pid ${monit_log_path} ${monit_log_path}"
restart program "${monit_script_path} start_bigchaindb $monit_pid_path/bigchaindb.pid ${monit_log_path} ${monit_log_path}"
stop program "${monit_script_path} stop_bigchaindb $monit_pid_path/bigchaindb.pid ${monit_log_path} ${monit_log_path}"
check process tendermint
with pidfile ${monit_pid_path}/tendermint.pid
start program "${monit_script_path} start_tendermint ${monit_pid_path}/tendermint.pid ${monit_log_path} ${monit_log_path}"
restart program "${monit_script_path} start_bigchaindb ${monit_pid_path}/bigchaindb.pid ${monit_log_path} ${monit_log_path}"
stop program "${monit_script_path} stop_tendermint ${monit_pid_path}/tendermint.pid ${monit_log_path} ${monit_log_path}"
depends on bigchaindb
EOF

# Setting permissions for control file
chmod 0700 ${monitrc_path}

echo -e "BigchainDB process manager configured!"
set -o errexit
2 changes: 2 additions & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,8 @@ def check_setuptools_features():

packages=find_packages(exclude=['tests*']),

scripts = ['pkg/scripts/bigchaindb-monit-config'],

entry_points={
'console_scripts': [
'bigchaindb=bigchaindb.commands.bigchaindb:main'
Expand Down

0 comments on commit c3f5e2a

Please sign in to comment.