Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Shared assemblies #328

Merged
merged 3 commits into from
Nov 26, 2021
Merged

Shared assemblies #328

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b8f950f
fix: reusing assemblies for all network presets.
fboucquez Jul 29, 2021
4297a97
fix: reusing assemblies for all network presets
Nov 25, 2021
270c681
feedback fix
Nov 26, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ The changelog format is based on [Keep a Changelog](https://keepachangelog.com/e
- The `bootstrap` preset is not the default anymore. The name must be provided via --preset or as a custom preset field.
- A 'safe' custom preset is cached in the target folder. It's not required when upgrading the node without a configuration change.
- Added `--logger` option to the commands.
- Assemblies are shared between all the presets.

## [1.1.1] - Nov-16-2021

Expand Down
98 changes: 57 additions & 41 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,7 @@ Symbol CLI tool that allows you creating, configuring and running Symbol's c
* [Key benefits:](#key-benefits)
* [Concepts](#concepts)
* [Requirements](#requirements)
* [Usage](#usage)
* [General Usage](#general-usage)
* [Installation](#installation)
* [Wizard](#wizard)
* [E2E Testing support](#e2e-testing-support)
* [Development](#development)
Expand All @@ -27,12 +26,7 @@ Symbol CLI tool that allows you creating, configuring and running Symbol's c

# Why this tool?

This tool has been created to address the problems defined in Symbol's [NIP11](https://github.com/nemtech/NIP/blob/main/NIPs/nip-0011.md).

It replaces:

- [catapult-service-bootstrap](https://github.com/nemtech/catapult-service-bootstrap)
- [symbol-testnet-bootstrap](https://github.com/nemgrouplimited/symbol-service-bootstrap)
This tool has been created to address the original problems defined in Symbol's [NIP11](https://github.com/nemtech/NIP/blob/main/NIPs/nip-0011.md).

# Key benefits:

Expand All @@ -58,23 +52,25 @@ Yaml files that define the configuration and layout of the network and nodes. It
Presets are defined at 4 levels from general to specific:

- Shared: Default configurations for all the networks.
- Network: It defines the main preset of a given network, example: `bootstrap` or `testnet`.
- Assembly: It defines a modification of a network, example: `testnet peer`, `tesnet dual`, `testnet api`. Assembly is required for some networks (like `testnet`).
- Network: It defines the configuration of a given network.
- Assembly: It defines a modification of a network selecting the services that the node will run.
- Custom: A user provided yml file (`--customPreset` param) that could override some or all properties in the out-of-the-box presets.

Properties in each file override the previous values (by object deep merge).

### Out-of-the-box presets and assemblies:
### Network Presets:

- [`mainnet`](presets/mainnet/network.yml): Used to create nodes connected to Symbol's Mainnet network. The [nemesis block](presets/mainnet/seed/00000) is copied over.
- [`testnet`](presets/testnet/network.yml): Used to create nodes connected to Symbol's Testnet network. The [nemesis block](presets/testnet/seed/00000) is copied over.
- [`bootstrap`](presets/bootstrap/network.yml): Used to create new private networks with dual currency configuration, network and harvest currencies. Nemesis block is generated.

- `-p bootstrap`: A [multinode](https://github.com/nemtech/symbol-bootstrap/blob/main/presets/bootstrap/network.yml) local network. A network with 1 mongo database, 2 peers, 1 api and 1 rest gateway. Nemesis block is generated.
- `-p bootstrap -a light`: A [light](https://github.com/nemtech/symbol-bootstrap/blob/main/presets/bootstrap/assembly-light.yml) local network. It's a version of bootstrap preset with 1 mongo database, 1 dual peer and 1 rest gateway. Great for faster light e2e automatic testing. Nemesis block is generated.
- `-p bootstrap -a full`: A [full](https://github.com/nemtech/symbol-bootstrap/blob/main/presets/bootstrap/assembly-full.yml) local network. The bootstrap preset plus 1 wallet, 1 explorer and 1 faucet. Great for demonstration purposes. Nemesis block is generated.
- `-p testnet -a peer`: A [harvesting](https://github.com/nemtech/symbol-bootstrap/blob/main/presets/testnet/assembly-peer.yml) peer node that connects to the current public [testnet](https://github.com/nemtech/symbol-bootstrap/blob/main/presets/testnet/network.yml). [Nemesis block](https://github.com/nemtech/symbol-bootstrap/tree/main/presets/testnet/seed/00000) is copied over.
- `-p testnet -a api`: A [api](https://github.com/nemtech/symbol-bootstrap/blob/main/presets/testnet/assembly-api.yml) peer node that connects to the current public [testnet](https://github.com/nemtech/symbol-bootstrap/blob/main/presets/testnet/network.yml) running its own mongo database and rest gateway. [Nemesis block](https://github.com/nemtech/symbol-bootstrap/tree/main/presets/testnet/seed/00000) is copied over.
- `-p testnet -a dual`: A [dual](https://github.com/nemtech/symbol-bootstrap/blob/main/presets/testnet/assembly-dual.yml) haversting peer node that connects to the current public [testnet](https://github.com/nemtech/symbol-bootstrap/blob/main/presets/testnet/network.yml) running its own mongo database and rest gateway. [Nemesis block](https://github.com/nemtech/symbol-bootstrap/tree/main/presets/testnet/seed/00000) is copied over.
- `-p mainnet -a peer`: A [harvesting](https://github.com/nemtech/symbol-bootstrap/blob/main/presets/mainnet/assembly-peer.yml) peer node that connects to the current public [mainnet](https://github.com/nemtech/symbol-bootstrap/blob/main/presets/mainnet/network.yml). [Nemesis block](https://github.com/nemtech/symbol-bootstrap/tree/main/presets/mainnet/seed/00000) is copied over.
- `-p mainnet -a api`: A [api](https://github.com/nemtech/symbol-bootstrap/blob/main/presets/mainnet/assembly-api.yml) peer node that connects to the current public [mainnet](https://github.com/nemtech/symbol-bootstrap/blob/main/presets/mainnet/network.yml) running its own mongo database and rest gateway. [Nemesis block](https://github.com/nemtech/symbol-bootstrap/tree/main/presets/mainnet/seed/00000) is copied over.
- `-p mainnet -a dual`: A [dual](https://github.com/nemtech/symbol-bootstrap/blob/main/presets/mainnet/assembly-dual.yml) haversting peer node that connects to the current public [mainnet](https://github.com/nemtech/symbol-bootstrap/blob/main/presets/mainnet/network.yml) running its own mongo database and rest gateway. [Nemesis block](https://github.com/nemtech/symbol-bootstrap/tree/main/presets/mainnet/seed/00000) is copied over.
### Assemblies:

- [`peer`](presets/assemblies/assembly-peer.yml): A standard peer-only node that contains 1 peer node.
- [`api`](presets/assemblies/assembly-api.yml): A standard API node that contains 1 Mongo database, 1 API node, 1 REST gateway, and 1 broker.
- [`dual`](presets/assemblies/assembly-dual.yml): A standard dual node that contains 1 Mongo database, 1 API node, 1 REST gateway, 1 broker, and 1 peer node.
- [`demo`](presets/assemblies/assembly-demo.yml): A dual node with an additional explorer, web wallet and faucet for test and demonstration purposes.
Jaguar0625 marked this conversation as resolved.
Show resolved Hide resolved
- [`multinode`](presets/assemblies/assembly-multinode.yml): A special assembly that contains 1 API node and 2 peer-only nodes. This assembly is for testing, it showcases how a private network with 3 nodes runs.

### Custom preset:

Expand All @@ -84,29 +80,29 @@ Custom presets give Symbol Bootstrap its versatility. Check out the custom prese

## Target:

The folder where the generated config, docker files and data are stored. The folder structure is:
The folder where the generated config, docker files and data are stored.

The folder structure is:

- `./preset.yml`: the final generated preset.yml that it's used to configure bootstrap, the nodes, docker, etc.
- `./addresses.yml`: randomly generated data that wasn't provided in the preset. e.g.: SSL keys, nodes' keys, nemesis accounts, generation hash seed, etc.
- `./nodes`: it holds the configuration, data and logs for all the defined node instances.
- `./gateways`: it holds the configuration and logs for all the defined node rest gateways.
- `./preset.yml`: The final generated preset.yml that it's used to configure bootstrap, the nodes, docker, etc.
- `./addresses.yml`: Randomly generated data that wasn't provided in the preset. e.g.: SSL keys, nodes' keys, nemesis accounts, generation hash seed, etc.
- `./nodes`: It holds the configuration, data and logs for all the defined node instances.
- `./gateways`: It holds the configuration and logs for all the defined node rest gateways.
- `./nemesis`: The folder used to hold the nemesis block. Block 1 data is generated via `nemgen` tool for new networks. For existing network, it is copied over.
- `./databases`: the location where the mongo data is stored for the different database instances.
- `./docker`: the generated docker-compose.yml, mongo init scripts and server basic bash scripts.
- `./reports`: the location of the generated reports.
- `./databases`: The location where the mongo data is stored for the different database instances.
- `./docker`: The generated docker-compose.yml, mongo init scripts and server basic bash scripts.
- `./explorers`: The generated explorer configuration.
- `./wallets`: The generated wallet configuration.
- `./reports`: The location of the generated reports.

Note: **The target folder should not be manually modified**. This tool may override any file in the target folder when doing upgrades. Any custom configuration should be provided via a custom preset. Check out the custom preset [guides](docs/presetGuides.md)!

# Requirements

- Node 12.0.0+
- Docker 18.3.0+
- Docker Compose 1.25.0+

Validate your environment by running:

```
symbol-bootstrap verify
```

Check your user can run docker without sudo:

```
Expand All @@ -121,7 +117,7 @@ Got permission denied while trying to connect to the Docker daemon socket at uni

Please follow this [guide](https://www.digitalocean.com/community/questions/how-to-fix-docker-got-permission-denied-while-trying-to-connect-to-the-docker-daemon-socket).

# Usage
# Installation

It's recommended to run the commands from en empty working dir.

Expand All @@ -148,20 +144,24 @@ USAGE
```
<!-- usagestop -->

# General Usage
Validate your environment by running:

```
symbol-bootstrap verify
```

The general usage would be:

```
symbol-bootstrap config -p bootstrap
symbol-bootstrap config -p testnet -a dual
symbol-bootstrap compose
symbol-bootstrap run
```

You can aggregate all these commands with this one liner:

```
symbol-bootstrap start -p bootstrap
symbol-bootstrap start -p testnet -a dual
```

If you need to start fresh, you many need to sudo remove the target folder (docker volumes dirs may be created using sudo). Example:
Expand All @@ -170,6 +170,22 @@ If you need to start fresh, you many need to sudo remove the target folder (dock
sudo rm -rf ./target
```

## Examples

Network presets and assemblies can be combined to generate different types of nodes. Some examples:

- `$ symbol-bootstrap start -p mainnet -a dual -c customPreset.yml`
- `$ symbol-bootstrap start -p testnet -a peer -c customPreset.yml`
- `$ symbol-bootstrap start -p testnet -a demo -c customPreset.yml`
- `$ symbol-bootstrap start -p bootstrap -a multinode -c customPreset.yml`
- `$ symbol-bootstrap start -p bootstrap -a demo -c customPreset.yml`
- `$ symbol-bootstrap start -p bootstrap -a dual -c customPreset.yml`

Although some combinations can be done, they may not be really useful. Examples that are NOT useful:

- `$ symbol-bootstrap start -p mainnet -a multinode`
- `$ symbol-bootstrap start -p testnet -a multinode`

# Wizard

If this is your first time creating a node, it's recommended to use the Wizard. Just follow the instructions:
Expand All @@ -183,7 +199,7 @@ symbol-bootstrap wizard
One use case of this CLI is client E2E testing support. If you are coding a Symbol client, you (Travis or Jenkins) can run e2e tests like:

```
symbol-bootstrap start -p bootstrap --detached
symbol-bootstrap start -p bootstrap -a dual -c my_custom_preset.yml --detached --healthCheck
YOUR TEST (e.g: npm run test, gradle test, selenium etc.)
symbol-bootstrap stop
```
Expand All @@ -208,7 +224,7 @@ Your `package.json` can look like this:
scripts": {
...
"clean-network": "symbol-bootstrap clean",
"run-network": "symbol-bootstrap start -c ./output/my_custom_preset.yml --detached --healthCheck",
"run-network": "symbol-bootstrap start -p bootstrap -a dual -c my_custom_preset.yml --detached --healthCheck",
"run-stop": "symbol-bootstrap stop",
"integration-test": "....some mocha/jest/etc tests running against localhost:3000 network....",
"e2e": "npm run clean-network && npm run run-network && npm run integration-test && npm run stop-network",
Expand All @@ -224,7 +240,7 @@ npm run e2e

## Node client E2E via API:

Alternatively, you can use the [BootstrapService](https://github.com/nemtech/symbol-bootstrap/blob/main/src/service/BootstrapService.ts) facade to programmatically start and stop a server.
Alternatively, you can use the [BootstrapService](src/service/BootstrapService.ts) facade to programmatically start and stop a server.

Example:

Expand Down
2 changes: 1 addition & 1 deletion config/rest-gateway/rest.json.mustache
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"network": {
"name": "{{{networkName}}}",
"description": "{{{description}}}"
"description": "{{{networkDescription}}}"
},

"port": 3000,
Expand Down
56 changes: 29 additions & 27 deletions docs/config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,44 +14,46 @@ USAGE
$ symbol-bootstrap config

OPTIONS
-a, --assembly=assembly The assembly that defines the node(s) layout. It can be provided via custom
preset or cli parameter. If not provided, the value is resolved from the
target/preset.yml file.
-a, --assembly=(api|demo|dual|multinode|peer) The assembly that defines the node(s) layout. It can be provided via
custom preset or cli parameter. If not provided, the value is resolved
from the target/preset.yml file.

-c, --customPreset=customPreset External preset file. Values in this file will override the provided
presets.
-c, --customPreset=customPreset External preset file. Values in this file will override the provided
presets.

-h, --help It shows the help of this command.
-h, --help It shows the help of this command.

-p, --preset=(bootstrap|testnet|mainnet) The network preset. It can be provided via custom preset or cli parameter.
If not provided, the value is resolved from the target/preset.yml file.
-p, --preset=(bootstrap|testnet|mainnet) The network preset. It can be provided via custom preset or cli
parameter. If not provided, the value is resolved from the
target/preset.yml file.

-r, --reset It resets the configuration generating a new one.
-r, --reset It resets the configuration generating a new one.

-t, --target=target [default: target] The target folder where the symbol-bootstrap network is
generated
-t, --target=target [default: target] The target folder where the symbol-bootstrap network
is generated

-u, --user=user [default: current] User used to run docker images when creating
configuration files like certificates or nemesis block. "current" means the
current user.
-u, --user=user [default: current] User used to run docker images when creating
configuration files like certificates or nemesis block. "current" means
the current user.

--logger=logger [default: Console,File] The loggers the command will use. Options are:
Console,File,Silent. Use ',' to select multiple loggers.
--logger=logger [default: Console,File] The loggers the command will use. Options are:
Console,File,Silent. Use ',' to select multiple loggers.

--noPassword When provided, Bootstrap will not use a password, so private keys will be
stored in plain text. Use with caution.
--noPassword When provided, Bootstrap will not use a password, so private keys will
be stored in plain text. Use with caution.

--password=password A password used to encrypt and decrypt private keys in preset files like
addresses.yml and preset.yml. Bootstrap prompts for a password by default,
can be provided in the command line (--password=XXXX) or disabled in the
command line (--noPassword).
--password=password A password used to encrypt and decrypt private keys in preset files
like addresses.yml and preset.yml. Bootstrap prompts for a password by
default, can be provided in the command line (--password=XXXX) or
disabled in the command line (--noPassword).

--report It generates reStructuredText (.rst) reports describing the configuration of
each node.
--report It generates reStructuredText (.rst) reports describing the
configuration of each node.

--upgrade It regenerates the configuration reusing the previous keys. Use this flag
when upgrading the version of bootstrap to keep your node up to date without
dropping the local data. Backup the target folder before upgrading.
--upgrade It regenerates the configuration reusing the previous keys. Use this
flag when upgrading the version of bootstrap to keep your node up to
date without dropping the local data. Backup the target folder before
upgrading.

EXAMPLES
$ symbol-bootstrap config -p bootstrap
Expand Down
Loading