New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add specific executor param #4934
Conversation
8f9dd24
to
0eb2675
Compare
Codecov Report
@@ Coverage Diff @@
## master #4934 +/- ##
==========================================
- Coverage 87.86% 85.12% -2.75%
==========================================
Files 111 111
Lines 8763 8794 +31
==========================================
- Hits 7700 7486 -214
- Misses 1063 1308 +245
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for the logic u mentioned it should be 'executorname__param` and not the other way around right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure this solution can be handled. Since the name of the Executor w.r.t the Flow is not known by the Deployment. Imagine an External Executor cannot have the same name for every Flow
:param executor_name: name of the Executor | ||
:returns: the parsed parameters after applying filtering for the specific Executor | ||
""" | ||
parsed_params = copy.deepcopy(parameters) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no need to copy
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is already a dict generated from protobuf
0aacd14
to
1582e8a
Compare
This reverts commit 492b25f.
Context
ATM when you pass a parameters to an Executor
client.index(docs, parameters = {...}
every Executor will receive all of them.Meaning that if you want to pass different parameters to each Executor you will need to change the name.That's problematic because some Executor except the same parameters, like
traversal_path
.For instance you can't embed on the chunk with one executor and embed on the root with an other one because you can only pass one
traversal_path
.We need to be able to send parameters to different Executor.
original issue here : #4919
this PR take over the old proposition which we decide not to go for : #4923
Proposal for a Solution
Using the
key__executor_name
instead ofkey
to precise that it should only be sent to the given Executor.The
__
operator is use for this kind of things in the python world. We use it in docarray : da[:,'tags__key'].This is actually the exact same syntax of scikit-learn pipeline.
Parameters of the estimators in the pipeline can be accessed using the <estimator>__<parameter> syntax
Moreover : regarding performance issue of deserialization on the gateway side I suggest to avoid doing any deserialization on the gateway side by sending the full parameters to the Worker and only passing the one relevant to the Executor
What this PR do:
__
syntax to pass executor to one ExecutorWhat still need to be fix/done:
__results__
or__something__
should not be considerate while parsing