Add expansion feature #37

damoncro · 2021-11-30T06:54:17Z

Expand .env if specified in yaml configuration

damoncro · 2021-11-30T07:07:52Z

This PR would not affect the old yaml setting. It only reads the dotenv field (if exists), and expand the setting. System Env is also expanded.

Read dotenv file and expand based on env and system env

pystarport/cluster.py

pystarport/expansion.py

yihuang

I recommend to allow user to override dotenv path in cli parameter and init_cluster parameter. it'll be more flexible for user to handle path issues.

damoncro · 2021-11-30T08:01:03Z

The dotenv field will be poped out in the setting after the expansion to make minimal impact of the original design. And keep it very simple usage: just read the .env file and expand it.

damoncro · 2021-11-30T08:03:09Z

I'll try override dotenv path in cli.

After crypto-com/pystarport#37, would change back to crypto-com/pystarport.

yihuang · 2021-12-01T02:11:23Z

The dotenv field will be poped out in the setting after the expansion to make minimal impact of the original design. And keep it very simple usage: just read the .env file and expand it.

I suggest don't expand at all when dotenv is not specified, to keep it backward compatible.

damoncro · 2021-12-01T03:01:10Z

The dotenv field will be poped out in the setting after the expansion to make minimal impact of the original design. And keep it very simple usage: just read the .env file and expand it.

I suggest don't expand at all when dotenv is not specified, to keep it backward compatible.

Yes, it is what it can do now. It is very simple, just one if statement. https://github.com/crypto-com/pystarport/pull/37/files#diff-aa7a1723d585e834919ecbcc08a9aacad7402b6d0dc1a1f5b478ad17c44657b5R911. And I will add cli support as well.

1. `--dotenv`: an optional cli option for overriding the dotenv option in yaml file 2. move the expansion codes to expansion.py and handle it in `expand_yaml`

lgtm-com · 2021-12-01T10:50:16Z

This pull request introduces 2 alerts when merging f17ff16 into a127111 - view on LGTM.com

new alerts:

1 for Redundant assignment
1 for Wrong number of arguments in a call

To make diff easy.

lgtm-com · 2021-12-01T11:08:47Z

This pull request introduces 2 alerts when merging 946d5e9 into a127111 - view on LGTM.com

new alerts:

1 for Redundant assignment
1 for Wrong number of arguments in a call

After crypto-com/pystarport#37, would change back to crypto-com/pystarport.

damoncro · 2021-12-02T07:02:23Z

Please check the test result here: https://github.com/damoncro/pystarport/runs/4391700415?check_suite_focus=true

damoncro · 2021-12-02T09:18:37Z

It is ready to merge. Please check the test result: https://github.com/crypto-com/pystarport/runs/4391781936?check_suite_focus=true

yihuang · 2021-12-03T04:02:00Z

pyproject.toml

@@ -22,6 +22,9 @@ supervisor = "^4.2.1"
 docker = "^4.3.1"
 bech32 = "^1.1.0"
 multitail2 = "^1.5.2"
+python-dotenv = "^0.19.2"
+deepdiff = "^5.6.0"
+pytest = "^6.2.5"


this should be in the dev-dependencies?

yihuang · 2021-12-03T04:03:19Z

pystarport/cluster.py

@@ -902,9 +903,9 @@ def relayer_chain_config(data_dir, chain, relayer_chains_config):


 def init_cluster(
-    data_dir, config_path, base_port, image=IMAGE, cmd=None, gen_compose_file=False
+    data_dir, config_path, base_port, dotenv, image=IMAGE, cmd=None, gen_compose_file=False


Suggested change

data_dir, config_path, base_port, dotenv, image=IMAGE, cmd=None, gen_compose_file=False

data_dir, config_path, base_port, dotenv=None, image=IMAGE, cmd=None, gen_compose_file=False

to keep it backward compatible.

yihuang · 2021-12-03T04:03:35Z

pystarport/cluster.py

@@ -1095,7 +1096,7 @@ def format_value(v, ctx):
 if __name__ == "__main__":
    interact("rm -r data; mkdir data", ignore_error=True)
    data_dir = Path("data")
-    init_cluster(data_dir, "config.yaml", 26650)
+    init_cluster(data_dir, "config.yaml", 26650, None)


Suggested change

init_cluster(data_dir, "config.yaml", 26650, None)

init_cluster(data_dir, "config.yaml", 26650)

yihuang

Looks great, only minor suggestions, and could you add a changelog item.

damoncro · 2021-12-03T04:07:03Z

Looks great, only minor suggestions, and could you add a changelog item.

Good

It is more general and backward compatible.

…tests (#233) * Add .env * Load dotenv and expand it in all related files 1. Scritps 2. yaml 3. Python files * Update to read full path of config files * Test: Replace with test pystarport at first After crypto-com/pystarport#37, would change back to crypto-com/pystarport. * Test expansion Test the expansion feature * Update the test pystarport * Update test pystarport * Update test_expansion * Move the test_expansion to pystarport * Back to crypto-com/pystarport@main and remove deepdiff dependency. * Use Path to get path It is more general and work with full path as well * Use full path of .env * Use pystarport v0.2.3 * Override dotenv in start-cronos/start-chainmain This makes these two script-bins defined in `scripts.nix` always use the same yaml and dotenv file, for metting the "re-producibility" requirement during one/same `nix-shell` environment. These script-bins are wrappers of pystarport with fixed yaml and dotenv. If you want to create the cronos/chainmain instance with different configurations, you should avoid using the script-bins. Please run the scripts directly, or use `pystarport` step by step. Besides, if you change the yaml or dotenv, you need to exit and re-enter the nix-shell. So that the modified yaml or dotenv are copied to nix store, and their paths can be rewritten into these two script-bins.

damoncro mentioned this pull request Nov 30, 2021

Problem: there are duplicated hard coded informations in integration tests crypto-org-chain/cronos#233

Merged

13 tasks

Add expansion feature

ee589d2

Read dotenv file and expand based on env and system env

damoncro force-pushed the feature/expansion branch from fb642b9 to ee589d2 Compare November 30, 2021 07:16

yihuang reviewed Nov 30, 2021

View reviewed changes

pystarport/cluster.py Outdated Show resolved Hide resolved

yihuang reviewed Nov 30, 2021

View reviewed changes

pystarport/expansion.py Outdated Show resolved Hide resolved

yihuang reviewed Nov 30, 2021

View reviewed changes

pystarport/expansion.py Outdated Show resolved Hide resolved

yihuang reviewed Nov 30, 2021

View reviewed changes

damoncro added a commit to damoncro/cronos that referenced this pull request Nov 30, 2021

Test: Replace with test pystarport at first

68675b5

After crypto-com/pystarport#37, would change back to crypto-com/pystarport.

Add --dotenv cli option and expand_yaml

f17ff16

1. `--dotenv`: an optional cli option for overriding the dotenv option in yaml file 2. move the expansion codes to expansion.py and handle it in `expand_yaml`

Back to original even it is too long

946d5e9

To make diff easy.

damoncro force-pushed the feature/expansion branch from 261b008 to 946d5e9 Compare December 1, 2021 10:59

damoncro added a commit to damoncro/cronos that referenced this pull request Dec 2, 2021

Test: Replace with test pystarport at first

993b0a7

After crypto-com/pystarport#37, would change back to crypto-com/pystarport.

Fix LGTM alerts

5948b55

damoncro force-pushed the feature/expansion branch from 23fbc32 to 5948b55 Compare December 2, 2021 03:56

Remove _str_to_python_value and remove {}

19fe003

damoncro added a commit to damoncro/cronos that referenced this pull request Dec 2, 2021

Test: Replace with test pystarport at first

bc2ae1a

After crypto-com/pystarport#37, would change back to crypto-com/pystarport.

Add test: test_expansion

380df94

damoncro added 2 commits December 2, 2021 15:06

Add Makefile and test action

2f14333

Update Makefile to test all tests

a08f9b9

yihuang reviewed Dec 3, 2021

View reviewed changes

yihuang requested changes Dec 3, 2021

View reviewed changes

damoncro added 2 commits December 3, 2021 12:22

Move dotenv=None to init_cluster definition

996fbef

It is more general and backward compatible.

Move pytest and deepdiff to dev

a1dcda5

yihuang self-requested a review December 3, 2021 04:27

yihuang approved these changes Dec 3, 2021

View reviewed changes

yihuang merged commit ed52d36 into crypto-com:main Dec 3, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add expansion feature #37

Add expansion feature #37

damoncro commented Nov 30, 2021

damoncro commented Nov 30, 2021 •

edited

Loading

yihuang left a comment •

edited

Loading

damoncro commented Nov 30, 2021

damoncro commented Nov 30, 2021

yihuang commented Dec 1, 2021

damoncro commented Dec 1, 2021 •

edited

Loading

lgtm-com bot commented Dec 1, 2021

lgtm-com bot commented Dec 1, 2021

damoncro commented Dec 2, 2021

damoncro commented Dec 2, 2021

yihuang Dec 3, 2021

yihuang Dec 3, 2021 •

edited

Loading

yihuang Dec 3, 2021

yihuang left a comment •

edited

Loading

damoncro commented Dec 3, 2021

	data_dir, config_path, base_port, dotenv, image=IMAGE, cmd=None, gen_compose_file=False
	data_dir, config_path, base_port, dotenv=None, image=IMAGE, cmd=None, gen_compose_file=False

	init_cluster(data_dir, "config.yaml", 26650, None)
	init_cluster(data_dir, "config.yaml", 26650)

Add expansion feature #37

Add expansion feature #37

Conversation

damoncro commented Nov 30, 2021

damoncro commented Nov 30, 2021 • edited Loading

yihuang left a comment • edited Loading

Choose a reason for hiding this comment

damoncro commented Nov 30, 2021

damoncro commented Nov 30, 2021

yihuang commented Dec 1, 2021

damoncro commented Dec 1, 2021 • edited Loading

lgtm-com bot commented Dec 1, 2021

lgtm-com bot commented Dec 1, 2021

damoncro commented Dec 2, 2021

damoncro commented Dec 2, 2021

yihuang Dec 3, 2021

Choose a reason for hiding this comment

yihuang Dec 3, 2021 • edited Loading

Choose a reason for hiding this comment

yihuang Dec 3, 2021

Choose a reason for hiding this comment

yihuang left a comment • edited Loading

Choose a reason for hiding this comment

damoncro commented Dec 3, 2021

damoncro commented Nov 30, 2021 •

edited

Loading

yihuang left a comment •

edited

Loading

damoncro commented Dec 1, 2021 •

edited

Loading

yihuang Dec 3, 2021 •

edited

Loading

yihuang left a comment •

edited

Loading