[ML] Preserve order of inference results #100143
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Pinging @elastic/ml-core (Team:ML) |
|
Hi @davidkyle, I've created a changelog YAML for you. |
davidkyle
commented
Oct 2, 2023
davidkyle
commented
Oct 2, 2023
jonathan-buttner
approved these changes
Oct 2, 2023
.../main/java/org/elasticsearch/xpack/ml/action/TransportInferTrainedModelDeploymentAction.java
Outdated
Show resolved
Hide resolved
| * the listener will never call {@code finalListener::onFailure} | ||
| * instead failures are returned as inference results. | ||
| */ | ||
| private ActionListener<InferenceResults> orderedListener( |
There was a problem hiding this comment.
nit: Can we make this static?
There was a problem hiding this comment.
👍 and I've added a test
| if (result instanceof ErrorInferenceResults errorResult) { | ||
| // Any failure fails all requests | ||
| // TODO is this the correct behaviour for batched requests? | ||
| finalListener.onFailure(errorResult.getException()); |
There was a problem hiding this comment.
I don't know the code well enough but maybe in the future we could make the response similar to a bulk response where an entry in the results array can either be a failure or a successful result?
There was a problem hiding this comment.
That the idea. The rest response does not have to change but internal users (such as ingest) can make better decisions about how to handle a response which is partially successful
…on/TransportInferTrainedModelDeploymentAction.java Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a request contains multiple inputs the order in which those inputs are processed is not deterministic if the C++ process is using more than one allocation. This change ensures the inference results are returned in the same order as the request inputs so that a caller knows result 1 is for input 1 etc.
Another change is to return all results even if there was a failure. Failures are returned as
ErrorInferenceResults, the caller should check for instances ofErrorInferenceResultsand handle them appropriately.This is labelled as a bug because the
_inferAPI accepts multiple inputs and previously the returned order was not guaranteed to match the input order.