Decouple configs from cluster by varl · Pull Request #53 · dhis2/cli

varl · 2019-05-23T15:37:03Z

Creates a new Cache for d2-cluster based on the global cache.

Would be nice to pair with https://github.com/dhis2/cli-helpers-engine/pull/15/files to remove the layered cache directory.

amcgee

This is cool! I'm not sure if we want to go all the way to implementing a generic key-value store, but if we don't (yet) this is a great solution.

I don't think the multi-tiered Caches are required, which would make this even simpler!

varl · 2019-05-27T17:57:31Z

Rebased on master, variation on the multi-layered Cache.

Produces this structure:

tree ~/.cache/d2/ -L 4
/home/varl/.cache/d2/
└── cache
    ├── cluster
    │   ├── dev
    │   │   ├── config.json
    │   │   └── docker-compose
    │   └── master
    │       ├── config.json
    │       └── docker-compose
    ├── cluster-db-dev.sql.gz
    └── cluster-db-master.sql.gz

edit: updated with simpler paths.

Depends on dhis2/cli-helpers-engine#16

varl · 2019-05-27T18:49:01Z

varl@veronika:~/dev/dhis2/libs/cli$ ./packages/main/bin/d2 debug cache list
┌──────────────────────────┬──────────┬─────────────────────┐
│ Name                     │ Size     │ Modified            │
├──────────────────────────┼──────────┼─────────────────────┤
│ cluster                  │ 512      │ 2019-05-27 17:50:36 │
├──────────────────────────┼──────────┼─────────────────────┤
│ cluster-db-dev.sql.gz    │ 98904634 │ 2019-05-27 17:50:42 │
├──────────────────────────┼──────────┼─────────────────────┤
│ cluster-db-master.sql.gz │ 165532   │ 2019-05-27 17:50:20 │
└──────────────────────────┴──────────┴─────────────────────┘

varl@veronika:~/dev/dhis2/libs/cli$ ./packages/main/bin/d2 debug cache list cluster
┌───────────────────┬──────┬─────────────────────┐
│ Name              │ Size │ Modified            │
├───────────────────┼──────┼─────────────────────┤
│ dev               │ 512  │ 2019-05-27 17:50:37 │
├───────────────────┼──────┼─────────────────────┤
│ master            │ 512  │ 2019-05-27 17:44:02 │
└───────────────────┴──────┴─────────────────────┘

varl@veronika:~/dev/dhis2/libs/cli$ ./packages/main/bin/d2 debug cache list cluster/dev/docker-compose
┌────────────────────┬──────┬─────────────────────┐
│ Name               │ Size │ Modified            │
├────────────────────┼──────┼─────────────────────┤
│ config             │ 512  │ 2019-05-27 19:01:22 │
├────────────────────┼──────┼─────────────────────┤
│ docker-compose.yml │ 918  │ 2019-05-23 09:53:06 │
├────────────────────┼──────┼─────────────────────┤
│ README.md          │ 3641 │ 2019-05-23 09:53:06 │
├────────────────────┼──────┼─────────────────────┤
│ scripts            │ 512  │ 2019-05-27 19:01:22 │
└────────────────────┴──────┴─────────────────────┘

I think I'll rename the d2-cluster-dev to just dev since it's already subdired.

edit: updated.

varl · 2019-05-30T10:15:13Z

Updated dhis2/cli-helpers-engine#16 with a read/write implementation for the cache, and could simplify this PR as a result.

The cached configuration reading/writing is handled after the runtime configuration has been resolved. Only a subset of the configuration object is cached for subsequent runs.

varl · 2019-06-05T14:05:05Z

@amcgee I think this is ready for review and testing without using links.

amcgee · 2019-06-12T14:13:43Z

@varl looks like there's been a bunch of activity since you said this was ready for review, are you still iterating on it?

varl · 2019-06-12T14:19:07Z

@amcgee No, I just rebased on master to bring it up to date. Looks like there's another conflict so I'll do it once more. 😇

varl · 2019-06-17T06:36:35Z

Rebased on master so incorporates #61.

varl · 2019-06-17T07:16:05Z

@amcgee This is ready for review.

amcgee

Looking good! Made a few comments

varl · 2019-06-18T10:52:43Z

@amcgee changed the code with regards to your comments.

amcgee

@varl made some few minor comments, will test and approve

amcgee · 2019-06-26T09:40:01Z

+    const cfg = await resolveConfiguration(argv)
+
    const cacheLocation = await initDockerComposeCache({
+        composeProjectName: name,


Should this also be makeComposeProject(name) or the parameter just called name? Otherwise maybe initDockerComposeCache::composeProjectName is a bit misleading...

Yeah, I've considered that too.

The main reason I want name here is to make the cache a lot easier to browse: d2 debug cache list clusters/dev/docker-compose

If we use makeComposeProject: d2 debug cache list clusters/d2-cluster-dev/docker-compose

Since all the compose projects are already namespaced to clusters/{name} I don't think it makes sense to prefix it again with d2-cluster-{name}.

It's still the cache directory for the compose project, so the function name names sense (initDockerComposeCache), it's just the argument that is slightly misleading. I can rename it to composeCacheName if that makes it clearer.

@amcgee Alternatively initDockerComposeCache::cacheDirName?

We can rename this later but I think name might even be fine

amcgee

Testing and looks good but found one undefined variable!

amcgee

Nice work @varl!! Tested and approved, sorry it took so much back-and-forth

varl · 2019-07-01T12:34:26Z

No worries, the PR is not that big but the implications of it are huge so good to be thorough!

# [1.3.0](v1.2.4...v1.3.0) (2019-07-01) ### Features * decouple configs from cluster ([#53](#53)) ([e5b40af](e5b40af))

dhis2-bot · 2019-07-01T12:37:44Z

🎉 This PR is included in version 1.3.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

varl mentioned this pull request May 25, 2019

feat: support official docker images #54

Merged

amcgee reviewed May 27, 2019

View reviewed changes

Comment thread packages/cluster/src/common.js Outdated

amcgee reviewed May 27, 2019

View reviewed changes

Comment thread packages/cluster/src/common.js Outdated

varl force-pushed the decouple-configs-from-cluster branch from e565467 to 91337d9 Compare May 27, 2019 17:55

varl mentioned this pull request May 27, 2019

feat: expose read and write commands for files in cache dhis2/cli-helpers-engine#16

Merged

varl marked this pull request as ready for review May 30, 2019 09:39

varl commented May 30, 2019

View reviewed changes

Comment thread packages/cluster/src/commands/down.js Outdated

varl commented May 30, 2019

View reviewed changes

Comment thread packages/cluster/src/common.js Outdated

varl commented May 30, 2019

View reviewed changes

Comment thread packages/cluster/src/common.js Outdated

varl commented May 30, 2019

View reviewed changes

Comment thread packages/cluster/src/common.js

varl force-pushed the decouple-configs-from-cluster branch 2 times, most recently from 046590e to 5e42b86 Compare June 6, 2019 13:02

varl force-pushed the decouple-configs-from-cluster branch from 601fb46 to 83d6504 Compare June 12, 2019 14:21

varl mentioned this pull request Jun 12, 2019

fix: use dhis2/docker-compose instead of amcgee/dhis2-backend as default #61

Merged

varl and others added 10 commits June 17, 2019 00:11

fix: each cluster has its own compose cache

e6067d2

fix: use multipe cluster caches

cdec88f

fix: store a cache obj

de99312

fix: use the dirname of the cache loc

ddbe035

fix: log cache obj

4544cb7

refactor: extra constant for cache config

0f9cc6a

fix: shorten the composeProject name after isolation

baef8a8

fix: simplify command cache handling

a0e0699

test: fix tests

0387b57

docs: fix desc for changed arg

a22dd5b

varl and others added 7 commits June 17, 2019 00:38

docs: update readme for cluster

e736081

docs: update readme

a55bbb4

docs: fix broken syntax

4935876

refactor: break up path and clean up imports

2d6209c

refactor: clean up imports

43cbcef

docs: add note about config order

18c1f6c

chore: add repo config files

6712b6e

varl force-pushed the decouple-configs-from-cluster branch from 3f849e8 to 2d6209c Compare June 17, 2019 06:35

amcgee suggested changes Jun 18, 2019

View reviewed changes

Comment thread packages/cluster/src/commands/up.js Outdated

Comment thread packages/cluster/src/common.js Outdated

Comment thread packages/cluster/src/common.js Outdated

Comment thread packages/cluster/src/commands/status.js Outdated

varl added 2 commits June 18, 2019 12:33

refactor: address pr comments

4f7d7d2

refactor: prefix docker-compose projects

88e7702

Merge branch 'master' into decouple-configs-from-cluster

daa147b

amcgee reviewed Jun 26, 2019

View reviewed changes

varl and others added 5 commits June 26, 2019 13:11

refactor: rearrange noop change

eaccd9c

Merge branch 'master' into decouple-configs-from-cluster

b4d4362

Merge branch 'master' into decouple-configs-from-cluster

06be935

Merge branch 'master' into decouple-configs-from-cluster

53fca03

Merge branch 'master' into decouple-configs-from-cluster

82a19d0

amcgee suggested changes Jul 1, 2019

View reviewed changes

Comment thread packages/cluster/src/common.js Outdated

chore: correct log variable

2a4ac2b

amcgee approved these changes Jul 1, 2019

View reviewed changes

varl merged commit e5b40af into master Jul 1, 2019

varl deleted the decouple-configs-from-cluster branch July 1, 2019 12:34

dhis2-bot pushed a commit that referenced this pull request Jul 1, 2019

chore(release): cut 1.3.0 [skip ci]

3b0eec5

# [1.3.0](v1.2.4...v1.3.0) (2019-07-01) ### Features * decouple configs from cluster ([#53](#53)) ([e5b40af](e5b40af))

dhis2-bot added the released label Jul 1, 2019

Conversation

varl commented May 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amcgee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

varl commented May 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

varl commented May 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

varl commented May 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

varl commented Jun 5, 2019

Uh oh!

amcgee commented Jun 12, 2019

Uh oh!

varl commented Jun 12, 2019

Uh oh!

varl commented Jun 17, 2019

Uh oh!

varl commented Jun 17, 2019

Uh oh!

amcgee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

varl commented Jun 18, 2019

Uh oh!

amcgee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

amcgee Jun 26, 2019

Choose a reason for hiding this comment

Uh oh!

varl Jun 26, 2019

Choose a reason for hiding this comment

Uh oh!

varl Jun 26, 2019

Choose a reason for hiding this comment

Uh oh!

amcgee Jul 1, 2019

Choose a reason for hiding this comment

Uh oh!

amcgee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

amcgee left a comment

Choose a reason for hiding this comment

Uh oh!

varl commented Jul 1, 2019

Uh oh!

dhis2-bot commented Jul 1, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

varl commented May 23, 2019 •

edited

Loading

varl commented May 27, 2019 •

edited

Loading

varl commented May 27, 2019 •

edited

Loading

varl commented May 30, 2019 •

edited

Loading