Inference Graph error response handling #3039

rachitchauhan43 · 2023-07-19T16:31:45Z

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2484

Type of changes
Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Feature/Issue validation/testing:

Please describe the tests that you ran to verify your changes and relevent result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

Test A
Test B
Logs

Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Checklist:

Have you added unit/e2e tests that prove your fix is effective or that this feature works?
Has code been commented, particularly in hard-to-understand areas?
Have you made corresponding changes to the documentation?

Release note:

Add dependency field(Soft/Hard) to the inference graph steps.

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

kserve-oss-bot · 2023-07-19T16:31:48Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rachitchauhan43
To complete the pull request process, please assign njhill after the PR has been reviewed.
You can assign the PR to them by writing /assign @njhill in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

rachitchauhan43 · 2023-07-25T15:48:17Z

/test ?

kserve-oss-bot · 2023-07-25T15:48:22Z

@rachitchauhan43: No presubmit jobs available for kserve/kserve@master

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kserve-oss-bot · 2023-07-25T22:58:29Z

@rachitchauhan43: you cannot LGTM your own PR.

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

config/crd/serving.kserve.io_clusterservingruntimes.yaml

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

…sier debugging Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

… tests Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

…r job as well as new tests have been added Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

cmd/router/main_test.go

python/test_resources/graph/error_404_isvc/requirements.txt

test/e2e/graph/test-resources/ig_test_ensemble_scenario_7.yaml

test/e2e/graph/test-resources/ig_test_ensemble_scenario_8.yaml

cmd/router/main.go

Co-authored-by: Dan Sun <dsun20@bloomberg.net> Signed-off-by: Rachit Chauhan <rachitchauhan43@gmail.com>

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

cmd/router/utils.go

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

yuzisun · 2023-09-07T11:04:09Z

@rachitchauhan43 There seems to be resource issues possibly after changing test parallelism to 2?

isvc-sklearn-runtime-predictor-00001" failed with message: 0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod...

https://github.com/kserve/kserve/actions/runs/6097857892/job/16547655921?pr=3039

…urce contention issues. Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

rachitchauhan43 · 2023-09-07T16:24:31Z

@rachitchauhan43 There seems to be resource issues possibly after changing test parallelism to 2?
isvc-sklearn-runtime-predictor-00001" failed with message: 0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod...
https://github.com/kserve/kserve/actions/runs/6097857892/job/16547655921?pr=3039

Ok. Removed parallelism and bumping timeout to 120 mins to avoid job timeout issue.

yuzisun · 2023-09-10T10:03:58Z

Awesome work @rachitchauhan43 !

/lgtm

yuzisun · 2023-09-10T10:07:11Z

/approve

yuzisun · 2023-09-10T10:20:38Z

@rachitchauhan43 Created a doc issue for you kserve/website#289

Signed-off-by: Edgar Hernández <23639005+israel-hdez@users.noreply.github.com>

* First commit Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Removing unused code Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Added handling for ensemble use case Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Bumping up resources for ISVCs to come up quickly Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Using latest version of controller-gen Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Adding in testing resources Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Reverting res req and limits Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Enabling live logging for pytest to enable logs to prin to cli for easier debugging Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Adding images build step in e2e tests. Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Fixing test to use service_name Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Adding build of router image to use latest router image everytime for tests Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Adding router enrty for kserve deps setup to update router for IGs Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Fixed linting errors Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Asserting in pythonic way Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Updating kserve version to 0.11.0 Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Adding more e2e tests Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Some refactoring to run tests locally easily Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Adding some test deps and adding more tests Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Fixing a bug and Adding tests for switch use cases Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Fixing a bug in e2e-test script Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Added e2e test cases for ensemble. Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Reducing parallelism to avoid pressure on minikube and timeouts Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Bump up timeout interval for the action. Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Bump up timeout interval for the test_triton_runtime_with_transformer as this ISVC has both predictor and transformer that needs to come up and it fails sometime with ISVC Timeout error. Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Running path-based routing tests as their own job Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Moved path-based-routing test as it's own job Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Reducing parallelism to 2 Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Reducing parallelism to 2 for test-fast job and increasing timeout for job as well as new tests have been added Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> * Update test/e2e/graph/test-resources/ig_test_ensemble_scenario_7.yaml Co-authored-by: Dan Sun <dsun20@bloomberg.net> Signed-off-by: Rachit Chauhan <rachitchauhan43@gmail.com * Removing parallelism and bumping up timeout to 120 mins to avoid resource contention issues. Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> --------- Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com> Signed-off-by: Rachit Chauhan <rachitchauhan43@gmail.com> Co-authored-by: Dan Sun <dsun20@bloomberg.net> Signed-off-by: iamlovingit <freecode666@gmail.com>

First commit

958d737

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

kserve-oss-bot requested a review from Iamlovingit July 19, 2023 16:31

kserve-oss-bot requested a review from yuzisun July 19, 2023 16:31

rachitchauhan43 changed the title ~~First commit~~ IG response code. Jul 19, 2023

rachitchauhan43 added 3 commits July 19, 2023 11:06

Removing unused code

632d6ee

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Merge branch 'kserve:master' into ig_response_code

507d268

Added handling for ensemble use case

3cfaf4b

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

rachitchauhan43 mentioned this pull request Jul 25, 2023

[WIP - Do Not Merge]Initial Set of changes for step level dependency in Inference Graphs #2542

Closed

9 tasks

rachitchauhan43 added 2 commits July 25, 2023 21:07

Merge branch 'kserve:master' into ig_response_code

70e7add

Bumping up resources for ISVCs to come up quickly

97fa4e2

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

sivanantha321 reviewed Jul 26, 2023

View reviewed changes

config/crd/serving.kserve.io_clusterservingruntimes.yaml Outdated Show resolved Hide resolved

rachitchauhan43 added 15 commits July 26, 2023 01:44

Using latest version of controller-gen

d549f1c

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Adding in testing resources

397f2ea

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Reverting res req and limits

84a8398

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Fixing linting errors

33b13bd

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Merge branch 'kserve:master' into ig_response_code

79220fe

Merge branch 'kserve:master' into ig_response_code

a5b18fd

Refactoring

3121bb8

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Enabling live logging for pytest to enable logs to prin to cli for ea…

592e2ef

…sier debugging Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Merge branch 'kserve:master' into ig_response_code

ff2a22d

Adding images build step in e2e tests.

895e9a9

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Adding images build step in e2e tests.

57bba8b

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Adding e2e test

bc0a7be

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Fixing linting errors

ba92a18

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Fixing test to use service_name

30aceac

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Adding build of router image to use latest router image everytime for…

cfd2acd

… tests Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

rachitchauhan43 added 2 commits August 27, 2023 07:22

Merge branch 'kserve:master' into ig_response_code

f02913e

Reducing parallelism to 2 for test-fast job and increasing timeout fo…

47e0e32

…r job as well as new tests have been added Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

yuzisun reviewed Aug 30, 2023

View reviewed changes

cmd/router/main_test.go Outdated Show resolved Hide resolved

yuzisun reviewed Aug 30, 2023

View reviewed changes

python/test_resources/graph/error_404_isvc/requirements.txt Outdated Show resolved Hide resolved

test/e2e/graph/test-resources/ig_test_ensemble_scenario_7.yaml Outdated Show resolved Hide resolved

test/e2e/graph/test-resources/ig_test_ensemble_scenario_8.yaml Outdated Show resolved Hide resolved

yuzisun reviewed Aug 30, 2023

View reviewed changes

cmd/router/main.go Show resolved Hide resolved

cmd/router/main.go Outdated Show resolved Hide resolved

cmd/router/main.go Outdated Show resolved Hide resolved

rachitchauhan43 and others added 5 commits August 30, 2023 12:49

Merge branch 'kserve:master' into ig_response_code

c656b8e

Update test/e2e/graph/test-resources/ig_test_ensemble_scenario_7.yaml

83c0847

Co-authored-by: Dan Sun <dsun20@bloomberg.net> Signed-off-by: Rachit Chauhan <rachitchauhan43@gmail.com>

Addressing review comments

075830b

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Addressing review comments

86de57b

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Merge branch 'kserve:master' into ig_response_code

00d2aa2

yuzisun reviewed Sep 4, 2023

View reviewed changes

cmd/router/utils.go Outdated Show resolved Hide resolved

rachitchauhan43 added 2 commits September 4, 2023 12:10

Addressing review comments

f332df9

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Addressed review comments

912286b

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

rachitchauhan43 requested a review from yuzisun September 6, 2023 13:26

Addressed review comments

28c0fdd

Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Removing parallelism and bumping up timeout to 120 mins to avoid reso…

91c1679

…urce contention issues. Signed-off-by: rachitchauhan43 <rachitchauhan43@gmail.com>

Merge branch 'kserve:master' into ig_response_code

9106bc2

kserve-oss-bot assigned yuzisun Sep 10, 2023

kserve-oss-bot added the lgtm label Sep 10, 2023

yuzisun merged commit a73c949 into kserve:master Sep 10, 2023
57 of 58 checks passed

yuzisun mentioned this pull request Sep 10, 2023

Update docs for inference graph response code kserve/website#289

Open

yuzisun added the approved label Sep 10, 2023

israel-hdez added a commit to israel-hdez/kserve that referenced this pull request Sep 19, 2023

Adapt kserve#3039 for openshift-ci

5e9ee84

Signed-off-by: Edgar Hernández <23639005+israel-hdez@users.noreply.github.com>

yuzisun changed the title ~~IG response code.~~ Inference Graph error response handling Oct 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Inference Graph error response handling #3039

Inference Graph error response handling #3039

rachitchauhan43 commented Jul 19, 2023 •

edited by yuzisun

Loading

kserve-oss-bot commented Jul 19, 2023

rachitchauhan43 commented Jul 25, 2023

kserve-oss-bot commented Jul 25, 2023

kserve-oss-bot commented Jul 25, 2023

yuzisun commented Sep 7, 2023 •

edited

Loading

rachitchauhan43 commented Sep 7, 2023

yuzisun commented Sep 10, 2023

yuzisun commented Sep 10, 2023

yuzisun commented Sep 10, 2023

Inference Graph error response handling #3039

Inference Graph error response handling #3039

Conversation

rachitchauhan43 commented Jul 19, 2023 • edited by yuzisun Loading

kserve-oss-bot commented Jul 19, 2023

rachitchauhan43 commented Jul 25, 2023

kserve-oss-bot commented Jul 25, 2023

kserve-oss-bot commented Jul 25, 2023

yuzisun commented Sep 7, 2023 • edited Loading

rachitchauhan43 commented Sep 7, 2023

yuzisun commented Sep 10, 2023

yuzisun commented Sep 10, 2023

yuzisun commented Sep 10, 2023

rachitchauhan43 commented Jul 19, 2023 •

edited by yuzisun

Loading

yuzisun commented Sep 7, 2023 •

edited

Loading