Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use fork for multiprocessing mode #2718

Merged
merged 1 commit into from
Mar 1, 2023
Merged

Use fork for multiprocessing mode #2718

merged 1 commit into from
Mar 1, 2023

Conversation

yuzisun
Copy link
Member

@yuzisun yuzisun commented Feb 26, 2023

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2690
Fixes #2694

Type of changes
Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Feature/Issue validation/testing:

Please describe the tests that you ran to verify your changes and relevent result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

  • Test multi-processing mode with 2 rest workers
python -m model --model_name=custom --worker=2 --enable_rest=true --enable_grpc=true

2023-02-26 19:49:49.687 61502 root INFO [register_model():187] Registering model: custom
2023-02-26 19:49:49.687 61502 root INFO [start():129] Setting max asyncio worker threads as 16
2023-02-26 19:49:49.688 61502 root INFO [serve():139] Starting uvicorn with 2 workers
2023-02-26 19:49:49.760 61502 root INFO [start():62] Starting gRPC server on [::]:8081
2023-02-26 19:49:49.765 61505 uvicorn.error INFO [serve():84] Started server process [61505]
2023-02-26 19:49:49.766 61505 uvicorn.error INFO [startup():45] Waiting for application startup.
2023-02-26 19:49:49 DEBUG [timing_asgi.middleware:40] ASGI scope of type lifespan is not supported yet
2023-02-26 19:49:49.768 61505 uvicorn.error INFO [startup():59] Application startup complete.
2023-02-26 19:49:49.768 61506 uvicorn.error INFO [serve():84] Started server process [61506]
2023-02-26 19:49:49.769 61506 uvicorn.error INFO [startup():45] Waiting for application startup.
2023-02-26 19:49:49 DEBUG [timing_asgi.middleware:40] ASGI scope of type lifespan is not supported yet
2023-02-26 19:49:49.771 61506 uvicorn.error INFO [startup():59] Application startup complete.
^C2023-02-26 19:50:17.428 61502 root INFO [stop():170] Stopping the model server
2023-02-26 19:50:17.428 61502 root INFO [stop():172] Stopping the rest server
2023-02-26 19:50:17.428 61502 root INFO [stop():175] Stopping the grpc server
2023-02-26 19:50:17.428 61502 root INFO [stop():67] Waiting for gRPC server shutdown
2023-02-26 19:50:17.429 61502 root INFO [stop():69] gRPC server shutdown complete
2023-02-26 19:50:17.453 61506 uvicorn.error INFO [shutdown():263] Shutting down
2023-02-26 19:50:17.453 61505 uvicorn.error INFO [shutdown():263] Shutting down
2023-02-26 19:50:17.555 61505 uvicorn.error INFO [shutdown():64] Waiting for application shutdown.
2023-02-26 19:50:17.555 61506 uvicorn.error INFO [shutdown():64] Waiting for application shutdown.
2023-02-26 19:50:17.556 61505 uvicorn.error INFO [shutdown():75] Application shutdown complete.
2023-02-26 19:50:17.556 61506 uvicorn.error INFO [shutdown():75] Application shutdown complete.
2023-02-26 19:50:17.557 61505 uvicorn.error INFO [serve():94] Finished server process [61505]
2023-02-26 19:50:17.557 61506 uvicorn.error INFO [serve():94] Finished server process [61506]
  • Test 1 worker with both REST and gRPC server
python -m model --model_name=custom --worker=1 --enable_rest=true --enable_grpc=true

2023-02-26 19:51:34.557 61522 root INFO [register_model():187] Registering model: custom
2023-02-26 19:51:34.557 61522 root INFO [start():129] Setting max asyncio worker threads as 16
2023-02-26 19:51:34.558 61522 root INFO [serve():139] Starting uvicorn with 1 workers
2023-02-26 19:51:34.595 61522 uvicorn.error INFO [serve():84] Started server process [61522]
2023-02-26 19:51:34.596 61522 uvicorn.error INFO [startup():45] Waiting for application startup.
2023-02-26 19:51:34.599 61522 root INFO [start():62] Starting gRPC server on [::]:8081
2023-02-26 19:51:34 DEBUG [timing_asgi.middleware:40] ASGI scope of type lifespan is not supported yet
2023-02-26 19:51:34.600 61522 uvicorn.error INFO [startup():59] Application startup complete.
^C2023-02-26 19:51:35.955 61522 root INFO [stop():170] Stopping the model server
2023-02-26 19:51:35.955 61522 root INFO [stop():172] Stopping the rest server
2023-02-26 19:51:35.955 61522 root INFO [stop():175] Stopping the grpc server
2023-02-26 19:51:35.955 61522 root INFO [stop():67] Waiting for gRPC server shutdown
2023-02-26 19:51:35.956 61522 root INFO [stop():69] gRPC server shutdown complete
2023-02-26 19:51:36.007 61522 uvicorn.error INFO [shutdown():263] Shutting down
2023-02-26 19:51:36.107 61522 uvicorn.error INFO [shutdown():64] Waiting for application shutdown.
2023-02-26 19:51:36.108 61522 uvicorn.error INFO [shutdown():75] Application shutdown complete.
2023-02-26 19:51:36.108 61522 uvicorn.error INFO [serve():94] Finished server process [61522]
  • Logs

Special notes for your reviewer:

  1. Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Checklist:

  • Have you added unit/e2e tests that prove your fix is effective or that this feature works?
  • Has code been commented, particularly in hard-to-understand areas?
  • Have you made corresponding changes to the documentation?

Release note:


@kserve-oss-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: yuzisun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Dan Sun <dsun20@bloomberg.net>
@rachitchauhan43
Copy link
Contributor

/lgtm

@yuzisun yuzisun merged commit f6deed8 into kserve:master Mar 1, 2023
Suresh-Nakkeran pushed a commit to Suresh-Nakkeran/kserve-1 that referenced this pull request Mar 3, 2023
Signed-off-by: Dan Sun <dsun20@bloomberg.net>
yuzisun added a commit to yuzisun/kserve that referenced this pull request Mar 5, 2023
Signed-off-by: Dan Sun <dsun20@bloomberg.net>
sriumcp pushed a commit to sriumcp/kserve that referenced this pull request Mar 7, 2023
Signed-off-by: Dan Sun <dsun20@bloomberg.net>
Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>
@yuzisun yuzisun self-assigned this Apr 3, 2023
ddelange added a commit to ddelange/kserve that referenced this pull request Apr 3, 2023
…urable-scheduling

* 'master' of https://github.com/kserve/kserve: (49 commits)
  Implement v2/open inference endpoints for OOTB python model servers (kserve#2655)
  Parametrized docker build (kserve#2666)
  Fix formatting and controller tests (kserve#2783)
  support multiple containers in ServingRuntime (kserve#2321)
  model_server.py: fix documentation for enable_latency_logging (kserve#2777)
  Updating Knative Serving and Istio to their latest version (kserve#2697)
  Support Direct VolumeMount for PVC (kserve#2738)
  fix missing ingress config options in helm chart (kserve#2772)
  fix trained model ready status (kserve#2774)
  Fix running out of disk space in e2e (kserve#2765)
  Iter8 canary and performance testing samples (kserve#2752)
  adjust order of types to default to float,int (kserve#2754)
  Make storage dependency as an optional dependency (kserve#2700)
  Creating new manifests for v0.10.1 (kserve#2736) (kserve#2739)
  Upgrdaing cherry pick action to v1.0.9 (kserve#2735)
  Feat: adds huggingface bloom example to torchserve (kserve#2466)
  Adding podaffinity and resourcerequirments to IG Spec (kserve#2711)
  Update OWNERS (kserve#2724)
  Fix knative installation failure in e2e test (kserve#2726)
  Use fork in multiprocessing mode (kserve#2718)
  ...
@lizzzcai lizzzcai mentioned this pull request Apr 17, 2023
9 tasks
Iamlovingit pushed a commit to Iamlovingit/kserve that referenced this pull request Oct 1, 2023
Signed-off-by: Dan Sun <dsun20@bloomberg.net>
Signed-off-by: iamlovingit <freecode666@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Status: Done
Development

Successfully merging this pull request may close these issues.

'AioServer.shutdown' was never awaited for kserve fastapi server Custom model issue on kserve v0.10
3 participants