Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement compose to debug local docker-compose clusters #568

Closed
corverroos opened this issue May 20, 2022 · 1 comment
Closed

Implement compose to debug local docker-compose clusters #568

corverroos opened this issue May 20, 2022 · 1 comment
Assignees
@corverroos
Labels
enhancement New feature or request

Comments

@corverroos
Copy link
Contributor

Problem to be solved

Since charon-docker-compose has been repurposed to be more customer/user facing, advanced features has been removed from it. It therefore cannot do things like "run locally built binary", "DKG using docker compose", or "mock validators", or "simnet".

As charon core developers, we need a way to test all the features of charon locally using docker-compose.

Proposed solution

  • Write a go program called compose
  • It is the successor of the "old" charon-docker-compose for developers.
  • The aim is to be able to test charon locally using docker-compose.
  • It is different from the "new" charon-docker-compose in that it supports "advanced" features like build locally binary, split keys, docker-compose DKG, mock validators, etc.
  • It is aimed at Charon core developers, not external customers or users.
  • The "old" charon-docker-compose used a makefile to configure the cluster (make create-cluster, make split-keys, make disable-simnet, etc)
  • This is fine, but since I want to write this tool in Golang, we can improve on the UX a lot.
  • I propose to serve a simple web UI form to configure the cluster
  • The tool would serve this ( via compose define)
  • On submission, the tool will create a compose.yml file which contains the cluster definition and all the other stuff in the webui form.
  • The tool will also be configurable by direct manipulation of the compose.yml
  • The tool will live in the charon repo at package testutil/compose

Example flow:

  • compose define : populate form and submit, creates compose.yml which contains the cluster definition and other config.
  • compose lock && docker-compose up: creates docker-compose.yml for key generation and then creates keys and cluster lock
  • compose run && docker-compose up : creates docker-compose.yml for running the cluster and runs it.
@corverroos
Copy link
Contributor Author

Example web ui
image

@corverroos corverroos self-assigned this May 20, 2022
@corverroos corverroos added the enhancement New feature or request label May 20, 2022
@corverroos corverroos mentioned this issue May 20, 2022
Merged
obol-bulldozer bot pushed a commit that referenced this issue May 20, 2022
@corverroos
testutil/compose: implement v0 define command (#569)
0bfa40b 
Implements the first iteration of the `compose define` command. It just uses default configuration for now and outputs the `compose.yml` file. Adding webui can be done later.

category: feature 
ticket: #568
This was referenced May 20, 2022
Merged
Merged
Merged
obol-bulldozer bot pushed a commit that referenced this issue May 21, 2022
@corverroos
testutil/compose: add lock command (#570)
ac67cd8 
Adds the first version of the lock command that creates a docker-compose file to run `charon create cluster` to generate keys and cluster lock file.

category: feature
ticket: #568
obol-bulldozer bot pushed a commit that referenced this issue May 21, 2022
@corverroos
testutil/compose: add run command (#571)
6d97de5 
Implements the `compose run` command that generates a docker-compose.yml for running a cluster.

category: feature
ticket: #568
obol-bulldozer bot pushed a commit that referenced this issue May 21, 2022
@corverroos
testutil/compose: add dkg lock (#575)
c7db045 
Adds support for DKG key generation to compose lock command.

category: feature
ticket: #568
This was referenced May 21, 2022
Merged
Merged
Merged
obol-bulldozer bot pushed a commit that referenced this issue May 23, 2022
@corverroos
testutil/compose: align define with the rest (#576)
4a78292 
Aligns the `define` step with the other two steps (`lock` and `run`) by also creating a `docker-compose.yml` file as output.
This also ensures we test `charon create dkg` with compose. Just do noop for other keygens in this step.

category: refactor
ticket: #568
obol-bulldozer bot pushed a commit that referenced this issue May 23, 2022
@corverroos
testutil/compose: add auto up (#577)
a4b3913 
Add automatically calling of docker-compose up after each command. Adding support for chaining commands: `compose define && compose lock && compose run`

category: feature
ticket: #568
obol-bulldozer bot pushed a commit that referenced this issue May 23, 2022
@corverroos
testutil/compose: add clean command (#578)
7f3f30b 
Extracts `clean` into its own command. Add `Step` to config and verify that config is incorrect state for each step.

category: refactor 
ticket: #568
@corverroos corverroos mentioned this issue May 23, 2022
Merged
obol-bulldozer bot pushed a commit that referenced this issue May 23, 2022
@corverroos
testutil/compose: add monitoring static files (#581)
969dc9a 
Adds prometheus/grafana/jaeger monitoring to `compose run` command. The static assets are embedded into the `compose` binary and written to disk on `compose define`. 

Also rename `NodeOnly` to `Bootnode`.

category: feature
ticket: #568
@corverroos corverroos mentioned this issue May 23, 2022
Merged
obol-bulldozer bot pushed a commit that referenced this issue May 23, 2022
@corverroos
testutil/compose: rename compose config (#590)
16b2c35 
Renames `charon-compose.yml` to `config.json` in order to differentiate and distinguish it from `docker-compose.yml`. 

category: refactor
ticket: #568
@corverroos corverroos mentioned this issue May 23, 2022
Merged
obol-bulldozer bot pushed a commit that referenced this issue May 23, 2022
@corverroos
testutil/compose: add support for locally built binaries (#591)
7eea302 
Adds support for locally built binaries via `compose define --build-local`. Thereby adding support for changing the source code inside the container. This is required for proper debugging. 

Note this requires the `CHARON_REPO` env var to be set. Developers are advised to put this in their bash profile.

category: feature 
ticket: #568
@corverroos corverroos mentioned this issue May 23, 2022
Merged
obol-bulldozer bot pushed a commit that referenced this issue May 23, 2022
@corverroos
testutil/compose: add readme (#593)
f8f08ec 
Adds a readme to compose

category: docs
ticket: #568
@corverroos corverroos mentioned this issue May 23, 2022
Merged
obol-bulldozer bot pushed a commit that referenced this issue May 23, 2022
@corverroos
testutil/compose: add support for real vcs (#595)
3c09b13 
Adds support for real VCs. 

category: feature
ticket: #568
@corverroos corverroos mentioned this issue May 24, 2022
Merged
obol-bulldozer bot pushed a commit that referenced this issue May 24, 2022
@corverroos
testutil/compose: introduce new and auto commands (#600)
6e537c4 
Refactors compose:
 - Decouple `new` from `define`: `new` only creates config.json file, `define` is now identical to `lock` and `run`; it only creates docker-compose files.
 - This adds support for using existing config files, and generating clusters from them
 - Add convenience function `compose auto` that runs `compose define && compose lock && compose run`, since this will be the most common use-case.
 
 Result:
 ```
 compose new --keygen==dkg && compose auto
 ``` 

category: refactor 
ticket: #568
@corverroos corverroos mentioned this issue May 24, 2022
Merged
obol-bulldozer bot pushed a commit that referenced this issue May 24, 2022
@corverroos
testutil/compose: safe clean (#601)
5c4dd41 
Make `compose clean` safe, by checking if config.json is present (and double check that no go files are present).

Note: I deleted my whole charon repo by running `compose clean` in the wrong folder  🤦 

Also pull latest docker image in define step.

category: bug 
ticket: #568
@corverroos corverroos mentioned this issue May 24, 2022
Merged
obol-bulldozer bot pushed a commit that referenced this issue May 24, 2022
@corverroos
testutil/compose: add split-keys-dir and beacon-node flags (#605)
c6d6daf 
Adds `split-keys-dir` and `beacon-node` flags to compose config to support splitting keys and connecting to real beacon node.

category: feature
ticket: #568
@corverroos corverroos mentioned this issue May 24, 2022
Merged
obol-bulldozer bot pushed a commit that referenced this issue May 24, 2022
@corverroos
testutil/compose: add splitting to readme (#606)
858032a 
Add splitting public testnet keys to readme.

category: docs
ticket: #568
@corverroos corverroos mentioned this issue May 24, 2022
Merged
obol-bulldozer bot pushed a commit that referenced this issue May 25, 2022
@corverroos
testutil/compose: add feature set flags (#607)
27414e7 
Adds the ability to enable different feature sets to compose.

Also fix issue with teku vc config.

category: feature
ticket: #568
@corverroos corverroos mentioned this issue May 26, 2022
Merged
obol-bulldozer bot pushed a commit that referenced this issue May 26, 2022
@corverroos
testutil/compose: fix dkg flow (#621)
c260692 
Fixes DKG flow in compose.

category: bug
ticket: #568
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@corverroos corverroos

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@corverroos