Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add note on compound executor #3406

Merged
merged 4 commits into from Sep 15, 2021
Merged

docs: add note on compound executor #3406

merged 4 commits into from Sep 15, 2021

Conversation

cristianmtr
Copy link
Contributor

No description provided.

@github-actions github-actions bot added size/S area/docs This issue/PR affects the docs labels Sep 14, 2021
@github-actions
Copy link

github-actions bot commented Sep 14, 2021

Latency summary

Current PR yields:

  • 😶 index QPS at 1097, delta to last 2 avg.: -1%
  • 😶 query QPS at 44, delta to last 2 avg.: -2%
  • 😶 dam extend QPS at 40059, delta to last 2 avg.: -3%
  • 😶 avg flow time within 1.6938 seconds, delta to last 2 avg.: +2%
  • 🐢🐢 import jina within 0.4201 seconds, delta to last 2 avg.: -13%

Breakdown

Version Index QPS Query QPS DAM Extend QPS Avg Flow Time (s) Import Time (s)
current 1097 44 40059 1.6938 0.4201
2.1.0 1106 44 41335 1.7998 0.4737
2.0.24 1113 45 41809 1.5042 0.4929

Backed by latency-tracking. Further commits will update this comment.

@codecov
Copy link

codecov bot commented Sep 14, 2021

Codecov Report

Merging #3406 (ce0a7ba) into master (c939bf3) will increase coverage by 0.41%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3406      +/-   ##
==========================================
+ Coverage   89.85%   90.26%   +0.41%     
==========================================
  Files         152      152              
  Lines       10841    10908      +67     
==========================================
+ Hits         9741     9846     +105     
+ Misses       1100     1062      -38     
Flag Coverage Δ
daemon 45.70% <ø> (+0.15%) ⬆️
jina 90.25% <ø> (+0.41%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
jina/hubble/hubio.py 88.07% <0.00%> (-0.78%) ⬇️
jina/peapods/peas/__init__.py 85.89% <0.00%> (-0.09%) ⬇️
jina/parsers/base.py 100.00% <0.00%> (ø)
jina/hubble/helper.py 90.06% <0.00%> (ø)
jina/jaml/__init__.py 95.57% <0.00%> (+0.18%) ⬆️
jina/peapods/zmq/__init__.py 89.04% <0.00%> (+0.23%) ⬆️
jina/types/arrays/neural_ops.py 89.91% <0.00%> (+0.92%) ⬆️
jina/peapods/runtimes/gateway/http/app.py 92.40% <0.00%> (+1.26%) ⬆️
jina/clients/base/__init__.py 92.42% <0.00%> (+1.51%) ⬆️
jina/peapods/runtimes/zmq/asyncio.py 92.98% <0.00%> (+1.75%) ⬆️
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c939bf3...ce0a7ba. Read the comment docs.

Copy link
Member

@hanxiao hanxiao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't this here already?
image
if the current tab not enough, please put your new content in that tab.

Copy link
Member

@hanxiao hanxiao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like the guide in docs/advanced/experimental/compound_executors.md is not complete? it introduces what is CompoundExecutor, but it does not give any MWU to showcase it, or reference.

@cristianmtr
Copy link
Contributor Author

I feel like the guide in docs/advanced/experimental/compound_executors.md is not complete? it introduces what is CompoundExecutor, but it does not give any MWU to showcase it, or reference.

Expanded with a basic example.

Copy link
Member

@alexcg1 alexcg1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo

docs/advanced/experimental/compound_executors.md Outdated Show resolved Hide resolved
Copy link
Member

@hanxiao hanxiao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor revision

docs/advanced/experimental/compound_executors.md Outdated Show resolved Hide resolved
Comment on lines 8 to 13
- combining a Storage and a Search Indexer in one, for easier access, scaling etc.
- combining a custom ranker and an Indexer in one, for cases where `top_k` is not achieved after a ranking process filters out results
- having Indexers that store multiple level of granularity in one Executor, for easier lookup and custom logic

If you want to develop a `CompoundExecutor`-type Executor based on one of the Executors existing in Jina Hub, you have to copy-paste the code of the classes you need into your own Executor's package.
This is because, for now, the Executors in Jina Hub can **not** be imported as Python modules.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- combining a Storage and a Search Indexer in one, for easier access, scaling etc.
- combining a custom ranker and an Indexer in one, for cases where `top_k` is not achieved after a ranking process filters out results
- having Indexers that store multiple level of granularity in one Executor, for easier lookup and custom logic
If you want to develop a `CompoundExecutor`-type Executor based on one of the Executors existing in Jina Hub, you have to copy-paste the code of the classes you need into your own Executor's package.
This is because, for now, the Executors in Jina Hub can **not** be imported as Python modules.
- combining a Storage and a Search Indexer in one, for easier access, scaling etc.
- combining a custom ranker and an Indexer in one, for cases where `top_k` is not achieved after a ranking process filters out results
- having Indexers that store multiple levels of granularity in one Executor, for easier lookup and custom logic
````{caution}
If you want to develop a `CompoundExecutor`-type Executor based on one of the Executors existing in Jina Hub, you have to copy-paste the code of the classes you need into your own Executor's package.
This is because, for now, the Executors in Jina Hub can **not** be imported as Python modules.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems my "` * 4" does not work on Github IDE, the closing quote is eaten by it

docs/advanced/experimental/compound_executors.md Outdated Show resolved Hide resolved
docs/advanced/experimental/compound_executors.md Outdated Show resolved Hide resolved
docs/advanced/experimental/compound_executors.md Outdated Show resolved Hide resolved
cristianmtr and others added 2 commits September 15, 2021 15:55
Co-authored-by: Han Xiao <artex.xh@gmail.com>
Co-authored-by: Han Xiao <artex.xh@gmail.com>
Copy link
Member

@hanxiao hanxiao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few glitches, let me directly working on this branch

@hanxiao hanxiao merged commit 7c67bc9 into master Sep 15, 2021
@hanxiao hanxiao deleted the docs-compound-executors branch September 15, 2021 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docs This issue/PR affects the docs size/S
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants