Add batcher module. #847

zhangrongguo · 2020-05-28T07:06:44Z

Add batcher module.
proposal: https://docs.google.com/document/d/14aa5zRtwK1zqcXdpQc9YClTduhtj8vr9waJcm0vR2GQ

kubeflow-bot · 2020-05-28T07:06:53Z

This change is

k8s-ci-robot · 2020-05-28T07:06:56Z

Hi @zhangrongguo. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

zhangrongguo · 2020-05-28T07:11:11Z

/assign @yuzisun

yuzisun · 2020-05-29T09:32:38Z

/ok-to-test

zhangrongguo · 2020-05-29T10:30:53Z

/retest

pkg/batcher/controllers/default.go

config/default/configmap/inferenceservice.yaml

Makefile

pkg/batcher/controllers/default.go

yuzisun · 2020-05-31T01:23:10Z

@zhangrongguo Glad to see the your progress on batching! I left some comments and questions.

I also did a test with vegeta seems some requests are returning 503 and tail latency is a bit high, any idea why ?

Requests      [total, rate, throughput]  30000, 500.00, 472.28
Duration      [total, attack, wait]      1m0.05819466s, 59.999824032s, 58.370628ms
Latencies     [mean, 50, 95, 99, max]    3.676433197s, 12.482479ms, 23.264452521s, 25.398686534s, 29.171002974s
Bytes In      [total, mean]              2740601, 91.35
Bytes Out     [total, mean]              2460000, 82.00
Success       [ratio]                    94.55%
Status Codes  [code:count]               200:28364  503:1636  
Error Set:
503 Service Unavailable

googlebot · 2020-06-01T05:25:04Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

googlebot · 2020-06-01T05:41:10Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

…vendor/github.com/shiena/ansicolor.

…her.yaml.

docs/samples/batcher/basic/README.md

pkg/apis/serving/v1alpha2/inferenceservice_types.go

cmd/batcher/main.go

zhangrongguo · 2020-06-17T01:24:17Z

/retest

yuzisun · 2020-06-18T11:22:43Z

Good with this for the first cut, suggested follow up changes in #890 and combining logger/batcher will be in proposal #802.

@ellistarn @cliveseldon do you have any comments for batcher?

cmd/batcher/main.go

pkg/batcher/controllers/main_controller.go

pkg/batcher/routers/router.go

pkg/batcher/server.go

pkg/batcher/server_test.go

pkg/webhook/admission/pod/batcher_injector.go

pkg/webhook/admission/pod/batcher_injector_test.go

test/e2e/batcher/test_batcher.py

test/scripts/build-batcher.sh

pkg/apis/serving/v1alpha2/inferenceservice_types.go

ellistarn · 2020-06-19T23:08:32Z

Mostly LGTM. A couple interface nits.

zhangrongguo · 2020-06-20T06:01:40Z

/retest

zhangrongguo · 2020-06-20T11:49:15Z

/retest

pkg/controller/inferenceservice/resources/knative/service.go

yuzisun · 2020-06-20T13:08:18Z

/retest

yuzisun · 2020-06-22T09:39:45Z

Awesome contribution @zhangrongguo! Excited to see this great feature and thanks for addressing all the comments!
/lgtm

yuzisun · 2020-06-22T09:56:25Z

/approve

k8s-ci-robot · 2020-06-22T09:56:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: yuzisun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [yuzisun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* Add batcher module. * Add batcher module. * Add batcher e2e test timeout. * Add batcher e2e test timeout. * Some changes according to PR comments. * Enlarge batcher resource. * Change image name of batcher for CI. * Change batcher resource. * Changes according to comments of PR. * Add vendor/github.com/astaxie/beego vendor/github.com/satori/go.uuid vendor/github.com/shiena/ansicolor. * Add release/triggers/latest/batcher.yaml release/triggers/tagged/batcher.yaml. * Change mode of test/scripts/build-batcher.sh. * Change mode of test/scripts/build-batcher.sh. * Change mode of test/scripts/build-batcher.sh. * Fix import time issue and change the resource of batcher. * Add dependencies. * Add dependencies. * Fix pkg/webhook/admission/pod/batcher_injector_test.go issue. * delete vendor. * Change batcher.Dockerfile. * Change test/scripts/build-batcher.sh prow_config.yaml. * Change test/scripts/build-batcher.sh prow_config.yaml. * Change batcher README.md. * Change batcher README.md. * Change batcher container name to batcher. * Change batcher README.md. * Change batcher README.md. * Change license. * Change the type of maxBatchSize, maxLatency, timeout to int type. * Fix addBatcherAnnotations issue.

zhangrongguo added 3 commits May 28, 2020 14:27

Add batcher module.

403faa9

Merge branch 'master' of https://github.com/kubeflow/kfserving into 727

9fe442c

Add batcher module.

3e8a69c

k8s-ci-robot added the needs-ok-to-test label May 28, 2020

k8s-ci-robot added the size/XXL label May 28, 2020

k8s-ci-robot requested review from jinchihe and Tomcli May 28, 2020 07:07

k8s-ci-robot assigned yuzisun May 28, 2020

k8s-ci-robot added ok-to-test and removed needs-ok-to-test labels May 29, 2020

zhangrongguo added 2 commits May 30, 2020 22:23

Add batcher e2e test timeout.

fcc35a1

Add batcher e2e test timeout.

84ab5fb

yuzisun reviewed May 31, 2020

View changes

Some changes according to PR comments.

c953e47

zhangrongguo force-pushed the 727 branch from a5e8658 to c953e47 Compare June 1, 2020 05:41

zhangrongguo added 7 commits June 1, 2020 13:51

Merge upstream/master.

3995320

Enlarge batcher resource.

b83d0ff

Change image name of batcher for CI.

8f8146b

Change batcher resource.

ccfdf77

Changes according to comments of PR.

05b45e8

Add vendor/github.com/astaxie/beego vendor/github.com/satori/go.uuid …

7b5f0ba

…vendor/github.com/shiena/ansicolor.

Add release/triggers/latest/batcher.yaml release/triggers/tagged/batc…

ba6dcbb

…her.yaml.

yuzisun reviewed Jun 17, 2020

View changes

docs/samples/batcher/basic/README.md Outdated Show resolved Hide resolved

Change batcher README.md.

3319c68

yuzisun reviewed Jun 17, 2020

View changes

pkg/apis/serving/v1alpha2/inferenceservice_types.go Outdated Show resolved Hide resolved

cmd/batcher/main.go Outdated Show resolved Hide resolved

Change batcher README.md.

d3278ac

Merge upstream.

2cf63fb

zhangrongguo requested a review from yuzisun June 17, 2020 08:18

yuzisun reviewed Jun 19, 2020

View changes

Change license.

4975e8b

zhangrongguo requested a review from yuzisun June 19, 2020 00:53

ellistarn reviewed Jun 19, 2020

View changes

pkg/apis/serving/v1alpha2/inferenceservice_types.go Outdated Show resolved Hide resolved

ellistarn reviewed Jun 19, 2020

View changes

pkg/apis/serving/v1alpha2/inferenceservice_types.go Outdated Show resolved Hide resolved

zhangrongguo requested a review from ellistarn June 20, 2020 01:24

Change the type of maxBatchSize, maxLatency, timeout to int type.

a4c5901

yuzisun reviewed Jun 20, 2020

View changes

pkg/controller/inferenceservice/resources/knative/service.go Outdated Show resolved Hide resolved

pkg/controller/inferenceservice/resources/knative/service.go Outdated Show resolved Hide resolved

Fix addBatcherAnnotations issue.

1a8b153

zhangrongguo requested a review from yuzisun June 22, 2020 06:39

k8s-ci-robot added the lgtm label Jun 22, 2020

k8s-ci-robot added the approved label Jun 22, 2020

k8s-ci-robot merged commit 87b45e0 into kserve:master Jun 22, 2020
2 of 3 checks passed

jbottum mentioned this pull request Nov 19, 2020

Kubeflow 1.2 Blog post kubeflow/community#455

Closed

Add batcher module. #847

Add batcher module. #847

zhangrongguo commented May 28, 2020 •

edited by yuzisun

kubeflow-bot commented May 28, 2020

k8s-ci-robot commented May 28, 2020

zhangrongguo commented May 28, 2020

yuzisun commented May 29, 2020

zhangrongguo commented May 29, 2020

yuzisun commented May 31, 2020

googlebot commented Jun 1, 2020

googlebot commented Jun 1, 2020

zhangrongguo commented Jun 17, 2020

yuzisun commented Jun 18, 2020

ellistarn commented Jun 19, 2020

zhangrongguo commented Jun 20, 2020

zhangrongguo commented Jun 20, 2020

yuzisun commented Jun 20, 2020

yuzisun commented Jun 22, 2020

yuzisun commented Jun 22, 2020

k8s-ci-robot commented Jun 22, 2020

Add batcher module. #847

Add batcher module. #847

Conversation

zhangrongguo commented May 28, 2020 • edited by yuzisun

kubeflow-bot commented May 28, 2020

k8s-ci-robot commented May 28, 2020

zhangrongguo commented May 28, 2020

yuzisun commented May 29, 2020

zhangrongguo commented May 29, 2020

yuzisun commented May 31, 2020

googlebot commented Jun 1, 2020

googlebot commented Jun 1, 2020

zhangrongguo commented Jun 17, 2020

yuzisun commented Jun 18, 2020

ellistarn commented Jun 19, 2020

zhangrongguo commented Jun 20, 2020

zhangrongguo commented Jun 20, 2020

yuzisun commented Jun 20, 2020

yuzisun commented Jun 22, 2020

yuzisun commented Jun 22, 2020

k8s-ci-robot commented Jun 22, 2020

zhangrongguo commented May 28, 2020 •

edited by yuzisun