Skip to content

Commit e5b40af

Browse files
varlvarl
authored
feat: decouple configs from cluster (#53)
* fix: each cluster has its own compose cache * fix: use multipe cluster caches * fix: store a cache obj * fix: use the dirname of the cache loc * fix: log cache obj * refactor: extra constant for cache config * fix: shorten the composeProject name after isolation * fix: simplify command cache handling * test: fix tests * docs: fix desc for changed arg * chore: new yarn.lock file after master was merged in * feat: allow cluster configuration to be stored in ~/.config/d2/config.js * docs: update readme for cluster * docs: update readme * docs: fix broken syntax * refactor: break up path and clean up imports * refactor: clean up imports * docs: add note about config order * chore: add repo config files * refactor: address pr comments * refactor: prefix docker-compose projects * refactor: rearrange noop change * chore: correct log variable
1 parent 1ff1975 commit e5b40af

14 files changed

Lines changed: 554 additions & 113 deletions

File tree

‎.commitlintrc.js‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
module.exports = {
2+
extends: ['@commitlint/config-conventional'],
3+
}

‎.dependabot/config.yml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
version: 1
2+
3+
update_configs:
4+
- package_manager: "javascript"
5+
directory: "/"
6+
update_schedule: "live"
7+
version_requirement_updates: "increase_versions"
8+
- package_manager: "java:maven"
9+
directory: "/"
10+
update_schedule: "monthly"
11+
- package_manager: "docker"
12+
directory: "/"
13+
update_schedule: "weekly"
14+
- package_manager: "submodules"
15+
directory: "/"
16+
update_schedule: "weekly"

‎.eslintrc.js‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
const SEVERITY = 2
2+
3+
module.exports = {
4+
root: true,
5+
6+
parser: 'babel-eslint',
7+
8+
env: {
9+
browser: true,
10+
node: true,
11+
jest: true,
12+
},
13+
14+
parserOptions: {
15+
// latest standard is ok, eq. to 9
16+
ecmaVersion: 2018,
17+
ecmaFeatures: {
18+
jsx: true,
19+
modules: true,
20+
},
21+
},
22+
23+
rules: {
24+
'max-params': [
25+
SEVERITY,
26+
{
27+
max: 3,
28+
},
29+
],
30+
'prefer-const': [
31+
SEVERITY,
32+
{
33+
destructuring: 'any',
34+
ignoreReadBeforeAssign: false,
35+
},
36+
],
37+
'no-mixed-spaces-and-tabs': [SEVERITY],
38+
},
39+
}

‎.github/semantic.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
titleOnly: true
2+
commitsOnly: false
3+
titleAndCommits: false
4+
allowMergeCommits: false

‎package.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,8 @@
1313
"devDependencies": {
1414
"@dhis2/cli-style": "3.3.4",
1515
"husky": "^2.7.0",
16-
"tape": "^4.10.2"
16+
"tape": "^4.10.2",
17+
"tape-await": "^0.1.2"
1718
},
1819
"husky": {
1920
"hooks": {

‎packages/cluster/README.md‎

Lines changed: 289 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -19,21 +19,298 @@ While you can install and run `d2-cluster` from `@dhis2/cli-cluster`, the prefer
1919

2020
Depending on your installation method, the following examples which use `d2 cluster` may need to be modified to use `d2-cluster`, or `npx @dhis2/cli-cluster`.
2121

22-
## Spin up a development cluster
22+
## Common concepts
2323

24-
This sequence of commands will install the latest bleeding-edge DHIS2 core instance in a Docker container and seed it with the relevant Sierra Leone [demo database](https://github.com/dhis2/dhis2-demo-db/)
24+
### Release channels
2525

26-
```sh
27-
> d2 cluster up dev --seed
28-
> d2 cluster logs dev core
29-
# Wait for the line "Catalina.start Server startup in XXXX ms", then ctrl+c / cmd+c to terminate
30-
> d2 cluster restart dev gateway # this is a hack necessary to rehup the gateway
31-
# DHIS2 is available with Sierra Leone db at http://localhost:8080
32-
# Run the Analytics Table export task from the Data Administration app
26+
DHIS2 has several release channels, such as **dev** and **stable**.
27+
28+
To each channel several artifacts can be published, so the **stable**
29+
channel contains all the stable releases of DHIS2, such as 2.32.0,
30+
2.32.1, etc.
31+
32+
The **dev** channel is the latest build straight from the development
33+
branches. There is one development branch per supported release of
34+
DHIS2.
35+
36+
For our Docker images, that means that we have one repo on Docker Hub
37+
per channel:
38+
39+
- Stable: https://cloud.docker.com/u/dhis2/repository/docker/dhis2/core
40+
- Dev: https://cloud.docker.com/u/dhis2/repository/docker/dhis2/core-dev
41+
42+
### Tags
43+
44+
Within each Docker repo, we have multiple tags. The channel coupled with
45+
the tag uniquely identifies a built DHIS2 Docker image. This is
46+
important for how the `cluster` command works.
47+
48+
For the **stable channel** each tag represents a formally released version
49+
of DHIS2.
50+
51+
- [2.32.0](https://github.com/dhis2/dhis2-core/tree/2.32.0)
52+
- [2.31.3](https://github.com/dhis2/dhis2-core/tree/2.31.3)
53+
54+
For the **dev channel**, each tag represents the last build from the
55+
development branches in
56+
[dhis2/dhis2-core](https://github.com/dhis2/dhis2-core):
57+
58+
- [master](https://github.com/dhis2/dhis2-core/tree/master)
59+
- [2.32](https://github.com/dhis2/dhis2-core/tree/2.32)
60+
- [2.31](https://github.com/dhis2/dhis2-core/tree/2.31)
61+
- [2.30](https://github.com/dhis2/dhis2-core/tree/2.30)
62+
63+
### Database dumps
64+
65+
For development DHIS2 provides a [set of database
66+
dumps](https://github.com/dhis2/dhis2-demo-db) which are essential in
67+
getting a usable environment up and running quickly.
68+
69+
Most often we use the [Sierra
70+
Leone](https://github.com/dhis2/dhis2-demo-db/tree/master/sierra-leone)
71+
dumps.
72+
73+
## Usage
74+
75+
### Getting help
76+
77+
Remember that the help is your friend:
78+
79+
```bash
80+
d2 cluster --help
81+
```
82+
83+
### Command layout
84+
85+
There are two arguments that are always required for the `cluster` to
86+
command to be able to do anything at all: `{command}` and `{name}`.
87+
88+
```bash
89+
d2 cluster {command} {name}
3390
```
3491

35-
Note that any Docker image in [amcgee/dhis2-core](https://cloud.docker.com/u/amcgee/repository/docker/amcgee/dhis2-core) can be used, just omit the `-alpine` suffix. Sierra Leone demo databases are automatically downloaded from the [dhis2-demo-db](https://github.com/dhis2/dhis2-demo-db) repository.
92+
The command refers to an action, like `up` or `down` and the name is the
93+
name of the cluster to operate on, which can be anything you like, like
94+
`mydev`, `superfly`, or `2.32`.
95+
96+
### Arguments
97+
98+
In addition to the command and name, there are more arguments you can
99+
pass to `cluster` to customize your environment. If the arguments are
100+
omitted there is some fallback logic, so even if they are not used, they
101+
are important to know about.
102+
103+
- `--channel`: This matches to the Docker Hub repository mentioned above
104+
in [Release channels](#release-channels). E.g. `dev`.
105+
106+
- `--dhis2-version`: This matches to the [tag name within a Docker
107+
Hub repo](#tags). E.g.
108+
[`2.32`](https://cloud.docker.com/u/dhis2/repository/docker/dhis2/core-dev/tags)
109+
110+
- `--db-version`: This matches to the database dumps mentioned in
111+
[Database dumps](#database-dumps). E.g. `dev` or `2.32`.
36112

37-
## Known Issues
113+
So through a combination of these arguments: `channel`, `dhis2-version`,
114+
and `db-version` we can spin up a cluster.
115+
116+
### Spin up a stable version
117+
118+
First up, in the best case scenario where you want to run DHIS2 2.32.0 on
119+
an empty database, you are able to run:
120+
121+
```bash
122+
d2 cluster up 2.32.0
123+
124+
# result
125+
# ---
126+
# channel: stable
127+
# dhis2Version: 2.32.0
128+
# dbVersion: empty
129+
```
130+
131+
Usually you want to `seed` your database with a database dump from
132+
Sierra Leone to have an instance set up with data. If you add the
133+
`--seed` command to the command above, it will try to find the database
134+
dump 2.32.0 in the
135+
[dhis2-db-demo](https://github.com/dhis2/dhis2-demo-db/tree/master/sierra-leone)
136+
repo. That doesn't exist, but 2.32 does.
137+
138+
```bash
139+
d2 cluster up 2.32.0 --seed
140+
# fail: there's no db dump with 2.32.0
141+
142+
d2 cluster up 2.32.0 --db-version 2.32 --seed
143+
144+
# result
145+
# ---
146+
# channel: stable
147+
# dhis2Version: 2.32.0
148+
# dbVersion: 2.32
149+
```
150+
151+
### Spin up a development version
152+
153+
Let's switch to the **dev** channel as we want the bleeding edge build
154+
from 2.32. We want it seeded with a 2.32 dump so we are going to run it
155+
with `--seed`.
156+
157+
```bash
158+
d2 cluster up 2.32 --channel dev --seed
159+
160+
# result
161+
# ---
162+
# channel: dev
163+
# dhis2Version: 2.32
164+
# dbVersion: 2.32
165+
```
166+
167+
Since the 2.32 branch exists in
168+
[dhis2-core](https://github.com/dhis2/dhis2-core/tree/2.32) and the 2.32
169+
dump exists in
170+
[dhis2-demo-db](https://github.com/dhis2/dhis2-demo-db/tree/master/sierra-leone/2.32)
171+
the tool doesn't need more information to create an environment.
172+
173+
Now, let's run a `master` build from the **dev** channel:
174+
175+
```bash
176+
d2 cluster up master --channel dev --db-version dev --seed
177+
178+
# result
179+
# ---
180+
# channel: dev
181+
# dhis2Version: master
182+
# dbVersion: dev
183+
```
184+
185+
Since the `--dhis2-version` argument was omitted, it used the `{name}`
186+
as fallback. Since we used `master` as the name, and the `master` tag
187+
exists in the
188+
[dhis2/core-dev](https://cloud.docker.com/u/dhis2/repository/docker/dhis2/core-dev/tags)
189+
it is able to resolve a complete environment.
190+
191+
We could also have run:
192+
193+
```bash
194+
d2 cluster up master --channel dev --db-version dev --dhis2-version master --seed
195+
```
196+
197+
The name can be anything you wish, but remember to specify `channel`,
198+
`dhis2-version`, and `db-version` in that case.
199+
200+
## Configuration
201+
202+
### Cached configuration
203+
204+
To avoid having to pass in all arguments over and over when using the
205+
`up` and `down` commands often, the `cluster` command caches your
206+
configuration per cluster in a `config.json` file.
207+
208+
```bash
209+
d2 debug cache list cluster/2.32.0
210+
┌────────────────┬──────┬─────────────────────┐
211+
│ Name │ Size │ Modified │
212+
├────────────────┼──────┼─────────────────────┤
213+
│ config.json │ 171 │ 2019-06-06 11:07:37 │
214+
├────────────────┼──────┼─────────────────────┤
215+
│ docker-compose │ 512 │ 2019-06-06 11:07:32 │
216+
└────────────────┴──────┴─────────────────────┘
217+
```
218+
219+
And it looks like this:
220+
221+
```bash
222+
cat ~/.cache/d2/cache/cluster/2.32.0/config.json
223+
{
224+
"channel": "dev",
225+
"dbVersion": "2.32",
226+
"dhis2Version": "2.32.0",
227+
"customContext": false,
228+
"image": "dhis2/core{channel}:{version}",
229+
"port": 8080
230+
}
231+
```
232+
233+
This means that if you run a command sequence like:
234+
235+
```bash
236+
d2 cluster up superfly --db-version 2.31 --dhis2-version master --seed --custom-context --port 9999 --channel dev
237+
238+
d2 cluster down superfly
239+
240+
d2 cluster up superfly
241+
```
242+
243+
The second time you run `up superfly` it will use the configuration from
244+
the first run:
245+
246+
```bash
247+
cat ~/.cache/d2/cache/cluster/superfly/config.json
248+
{
249+
"channel": "dev",
250+
"dbVersion": "2.31",
251+
"dhis2Version": "master",
252+
"customContext": true,
253+
"image": "dhis2/core{channel}:{version}",
254+
"port": "9999"
255+
}
256+
```
257+
258+
To purge the `config.json` file you can use:
259+
260+
```bash
261+
d2 debug cache purge cluster/superfly/config.json
262+
? Are you sure you want to remove cache item "cluster/superfly/config.json"? Yes
263+
Purged cache item cluster/superfly/config.json
264+
```
265+
266+
### Persistent configuration
267+
268+
It is also possible to set up your clusters in the `d2` configuration
269+
file, e.g. `~/.config/d2/config.js`:
270+
271+
```js
272+
module.exports = {
273+
cluster: {
274+
channel: 'stable',
275+
clusters: {
276+
superfly: {
277+
channel: 'dev',
278+
dbVersion: '2.31',
279+
dhis2Version: 'master',
280+
customContext: true,
281+
image: 'dhis2/core{channel}:{version}',
282+
port: 9999
283+
}
284+
}
285+
}
286+
}
287+
```
288+
289+
```bash
290+
d2 cluster up superfly
291+
292+
# saves the configuration to `config.json`
293+
cat ~/.cache/d2/cache/cluster/superfly/config.json
294+
{
295+
"channel": "dev",
296+
"dbVersion": "2.31",
297+
"dhis2Version": "master",
298+
"customContext": true,
299+
"image": "dhis2/core{channel}:{version}",
300+
"port": 9999
301+
}
302+
```
303+
304+
From here it's possible to override the configuration file properties
305+
for a cluster as well:
306+
307+
```
308+
# port is 9999 in ~/.config/d2/config.js:clusters.superfly.port
309+
d2 cluster up superfly --port 8888
310+
311+
# port is saved as 8888 in ~/.cache/d2/cache/cluster/superfly/config.json:port
312+
```
38313

39-
- The 2.31 patch releases (2.31.0, 2.31.1, etc.) are nested, so they cannot be downloaded automatically. You can download any .sql.gz file manually and specify it with the `--seedFile` option to the `d2 cluster up` command or the `--path` option to the `d2 cluster seed` command.
314+
Now for each subsequence `down` and `up` command, the cached config will
315+
take priority over the persistent configuration. When you clear the
316+
cache, the persistent configuration will come into effect again.

0 commit comments

Comments
 (0)