Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: move save/load config to JAML module #1533

Merged
merged 31 commits into from Dec 23, 2020
Merged

Conversation

hanxiao
Copy link
Member

@hanxiao hanxiao commented Dec 22, 2020

Details see docstring

Support substitutions in YAML:
- Environment variables: ${{ENV.VAR}} (recommended), ${{VAR}}, $VAR.
- Context dict (context): ${{VAR}}(recommended), $VAR.
- Internal reference via this and root: ${{this.same_level_key}}, ${{root.root_level_key}}

    Substitutions are carried in the order and multiple passes to resolve variables with best effort.

    .. highlight:: yaml
    .. code-block:: yaml

        !BaseEncoder
        metas:
            name: ${{VAR_A}}  # env or context variables
            workspace: my-${{this.name}}  # internal reference

    .. highlight:: python
    .. code-block:: python

        # load Executor from yaml file
        BaseExecutor.load_config('a.yml')

        # load Executor from yaml file and substitute environment variables
        os.environ['VAR_A'] = 'hello-world'
        b = BaseExecutor.load_config('a.yml')
        assert b.name == hello-world

        # load Executor from yaml file and substitute variables from a dict
        b = BaseExecutor.load_config('a.yml', context={'VAR_A': 'hello-world'})
        assert b.name == hello-world

        # disable substitute
        b = BaseExecutor.load_config('a.yml', substitute=False)

@hanxiao hanxiao requested a review from a team as a code owner December 22, 2020 15:26
@jina-bot jina-bot added size/M area/core This issue/PR affects the core codebase area/helper This issue/PR affects the helper functionality area/network This issue/PR affects network functionality area/testing This issue/PR affects testing component/executor component/flow component/logging component/peapod executor/meta component/jaml labels Dec 22, 2020
@hanxiao hanxiao marked this pull request as draft December 22, 2020 15:30
@github-actions
Copy link

github-actions bot commented Dec 22, 2020

Latency summary

Current PR yields:

  • 😶 index QPS at 1728, delta to last 3 avg.: -1%
  • 😶 query QPS at 27, delta to last 3 avg.: -5%

Breakdown

Version Index QPS Query QPS
current 1728 27
0.8.12 1755 28
0.8.11 1726 28
0.8.10 1775 28

Backed by latency-tracking. Further commits will update this comment.

@jina-bot jina-bot added size/L and removed size/M labels Dec 22, 2020
@hanxiao hanxiao marked this pull request as ready for review December 23, 2020 16:04
@codecov
Copy link

codecov bot commented Dec 23, 2020

Codecov Report

Merging #1533 (7cd3c34) into master (d45af86) will increase coverage by 0.24%.
The diff coverage is 96.52%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1533      +/-   ##
==========================================
+ Coverage   84.35%   84.60%   +0.24%     
==========================================
  Files         109      108       -1     
  Lines        6277     6359      +82     
==========================================
+ Hits         5295     5380      +85     
+ Misses        982      979       -3     
Impacted Files Coverage Δ
jina/executors/compound.py 85.60% <ø> (ø)
jina/logging/logger.py 82.25% <ø> (+0.14%) ⬆️
jina/jaml/__init__.py 95.65% <94.52%> (+1.31%) ⬆️
jina/jaml/helper.py 94.01% <98.48%> (+5.78%) ⬆️
jina/enums.py 96.52% <100.00%> (+0.09%) ⬆️
jina/excepts.py 100.00% <100.00%> (ø)
jina/flow/base.py 89.03% <100.00%> (-0.14%) ⬇️
jina/jaml/parsers/executor/legacy.py 87.09% <100.00%> (+0.21%) ⬆️
jina/logging/profile.py 68.33% <0.00%> (-0.58%) ⬇️
jina/drivers/craft.py 100.00% <0.00%> (ø)
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d45af86...c6009c4. Read the comment docs.

@hanxiao hanxiao merged commit f670690 into master Dec 23, 2020
@hanxiao hanxiao deleted the refactor-load-config branch December 23, 2020 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/core This issue/PR affects the core codebase area/helper This issue/PR affects the helper functionality area/network This issue/PR affects network functionality area/testing This issue/PR affects testing component/executor component/flow component/jaml component/logging component/peapod component/resource executor/indexer executor/meta size/L
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants