feat(cli): add CLI command to serve a runner #2920

bojiang · 2022-08-18T03:09:06Z

to arrange our serving CLI/Python APIs like this:

bentoml serve
bentoml serve --production
bentoml serve-grpc
bentoml serve-grpc --production

bentoml start-grpc-server
bentoml start-rest-server
bentoml start-runner-server

The APIs start with start are low-level APIs used in Yatai, would be hidden from the help message for now

run bare servers from container
docker run -it --rm -p 3000:3000 -e BENTOML_SERVE_COMPONENT=rest_server -e BENTOML_SERVE_RUNNER_MAP="asdfasdf=1 py_model.case-1.e2e=tcp://192.168.67.1:4000" general_workflow_service.case-1.e2e:tefgtebeok7iiusu
docker run -it --rm -p 4000:3000 -e BENTOML_SERVE_COMPONENT=runner -e BENTOML_SERVE_RUNNER_NAME=py_model.case-1.e2e general_workflow_service.case-1.e2e:awvtyqreokd4cusu

codecov · 2022-08-18T03:11:19Z

Codecov Report

Merging #2920 (5c5a98b) into main (901256c) will decrease coverage by 0.96%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #2920      +/-   ##
==========================================
- Coverage   69.88%   68.92%   -0.97%     
==========================================
  Files         121      122       +1     
  Lines        9961    10094     +133     
==========================================
- Hits         6961     6957       -4     
- Misses       3000     3137     +137

Impacted Files	Coverage Δ
bentoml/serve.py	`0.00% <0.00%> (ø)`
bentoml/start.py	`0.00% <0.00%> (ø)`
bentoml/_internal/utils/circus/__init__.py	`60.00% <0.00%> (-30.00%)`	⬇️
bentoml/_internal/utils/formparser.py	`18.71% <0.00%> (-1.29%)`	⬇️
bentoml/_internal/service/service.py	`96.70% <0.00%> (-0.11%)`	⬇️
bentoml/__init__.py	`100.00% <0.00%> (ø)`
bentoml/_internal/io_descriptors/json.py	`89.24% <0.00%> (ø)`
bentoml/_internal/log.py	`85.96% <0.00%> (+3.50%)`	⬆️

bentoml_cli/worker/README.md

ssheng · 2022-08-25T03:41:08Z

bentoml/_internal/bento/docker/entrypoint.sh

@@ -11,7 +11,7 @@ _is_sourced() {

 _main() {
 	# if first arg looks like a flag
-	if [ "${1:0:1}" = '-' ]; then
+	if [ -z "$@" ] || [ "${1:0:1}" = '-' ]; then


Is -z "$@" a superset of "${1:0:1}" = '-'?

command:
-z "$@" -> true
[ "${1:0:1}" = '-' ] -> false

Minor style nit, but I think [ $# -eq 1 ] is clearer than [ -z "$@" ].

ssheng · 2022-08-25T03:50:38Z

bentoml/serve.py

+
+
+@inject
+def serve_bare_api_server(


"Serve" isn't technically accurate here. How about just start_api_server and start_runner?

ssheng · 2022-08-25T03:52:54Z

bentoml_cli/serve.py

+        type=click.STRING,
+        default=None,
+        envvar="BENTOML_SERVE_COMPONENT",
+        help="[Experimental] Component (`api-server` or `runner`) to serve, if not specified, will serve all components",


Should it be api_server with a underscore? api_server is how it appears in other references.

bentoml_cli/utils.py

ssheng · 2022-08-25T03:57:35Z

requirements/docs-requirements.txt

 sphinx==4.5.0
+setuptools==60.0.0


What is the reason for the downgrade?

ping @aarnphm

this has to do with the newer version of setuptools and it setup different kind of editables install. It is better to just lock setuptools for now.

sauyon · 2022-08-25T17:03:16Z

I think HTTP is more accurate than REST, since our APIs can pretty easily be used to make a non-RESTful server.

aarnphm · 2022-08-19T08:07:17Z

bentoml_cli/serve.py

+        type=click.STRING,
+        default=None,
+        envvar="BENTOML_SERVE_COMPONENT",
+        help="[Experimental] Component (`api-server` or `runner`) to serve, if not specified, will serve all components",


Suggested change

help="[Experimental] Component (`api-server` or `runner`) to serve, if not specified, will serve all components",

help="[EXPERIMENTAL] Component (`api-server` or `runner`) to serve, if not specified, will serve all components",

aarnphm · 2022-08-19T08:07:28Z

bentoml_cli/serve.py

+        type=click.STRING,
+        default=None,
+        envvar="BENTOML_SERVE_RUNNER_NAME",
+        help="[Experimental] required if `component` is `runner`, specify the runner name to serve",


Suggested change

help="[Experimental] required if `component` is `runner`, specify the runner name to serve",

help="[EXPERIMENTAL] required if `component` is `runner`, specify the runner name to serve",

aarnphm · 2022-08-19T08:07:38Z

bentoml_cli/serve.py

+        type=click.STRING,
+        multiple=True,
+        envvar="BENTOML_SERVE_RUNNER_MAP",
+        help="[Experimental] required if `component` is `api-server' JSON string of runners map",


Suggested change

help="[Experimental] required if `component` is `api-server' JSON string of runners map",

help="[EXPERIMENTAL] required if `component` is `api-server' JSON string of runners map",

aarnphm · 2022-08-25T17:03:12Z

requirements/docs-requirements.txt

 sphinx==4.5.0
+setuptools==60.0.0


this has to do with the newer version of setuptools and it setup different kind of editables install. It is better to just lock setuptools for now.

ssheng · 2022-08-25T17:54:37Z

bentoml/start.py

+
+
+@inject
+def ensure_prometheus_dir(


Are we duplicating this function?

bentoml_cli/utils.py

bojiang requested review from ssheng, parano and a team as code owners August 18, 2022 03:09

bojiang requested review from sauyon and removed request for a team August 18, 2022 03:09

bojiang changed the title ~~fix(cli): add CLI command to serve a runner~~ feat(cli): add CLI command to serve a runner Aug 18, 2022

bojiang force-pushed the cli-runner-server branch from 7b91908 to 9c959b4 Compare August 18, 2022 10:29

aarnphm reviewed Aug 18, 2022

View reviewed changes

bentoml_cli/worker/README.md Outdated Show resolved Hide resolved

bojiang mentioned this pull request Aug 18, 2022

chore(container): add entrypoints for yatai #2897

Closed

bojiang force-pushed the cli-runner-server branch from 0922de9 to db74da2 Compare August 24, 2022 08:01

ssheng added this to the 1.0.4 milestone Aug 25, 2022

ssheng reviewed Aug 25, 2022

View reviewed changes

bojiang added 15 commits August 25, 2022 20:39

chore(container): add entrypoints for yatai

05c6606

fix

102c368

amend

976bb30

fix entrypoint

7bb9a95

fix(cli): add CLI command to serve a runner

791d3ca

fix testing serve utils

523c597

setuptools env

c8d9190

try to fix the env on action

4878d6a

resolve conflicts

829c8a3

lock setuptools

e0e55c3

minor fix

ec46ada

start commands

ce0752d

fix docs

9b9dfef

fix entrypoint

6bea150

log output

9d0bdd8

bojiang force-pushed the cli-runner-server branch from a9d77f8 to 9d0bdd8 Compare August 25, 2022 12:39

aarnphm reviewed Aug 25, 2022

View reviewed changes

ssheng reviewed Aug 25, 2022

View reviewed changes

bentoml_cli/utils.py Outdated Show resolved Hide resolved

ssheng and others added 4 commits August 25, 2022 14:39

Update bentoml_cli/utils.py

3268118

renaming

bdfb918

fix testing suites

98ed2be

patch

5c5a98b

ssheng approved these changes Aug 26, 2022

View reviewed changes

ssheng merged commit 7aab4b6 into bentoml:main Aug 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cli): add CLI command to serve a runner #2920

feat(cli): add CLI command to serve a runner #2920

bojiang commented Aug 18, 2022 •

edited

codecov bot commented Aug 18, 2022 •

edited

ssheng Aug 25, 2022

bojiang Aug 25, 2022

bojiang Aug 25, 2022

sauyon Aug 25, 2022

ssheng Aug 25, 2022

ssheng Aug 25, 2022

ssheng Aug 25, 2022

bojiang Aug 25, 2022

aarnphm Aug 25, 2022

sauyon commented Aug 25, 2022

aarnphm Aug 19, 2022

aarnphm Aug 19, 2022

aarnphm Aug 19, 2022

aarnphm Aug 25, 2022

ssheng Aug 25, 2022

	help="[Experimental] Component (`api-server` or `runner`) to serve, if not specified, will serve all components",
	help="[EXPERIMENTAL] Component (`api-server` or `runner`) to serve, if not specified, will serve all components",

	help="[Experimental] required if `component` is `runner`, specify the runner name to serve",
	help="[EXPERIMENTAL] required if `component` is `runner`, specify the runner name to serve",



		@inject
		def serve_bare_api_server(

		sphinx==4.5.0
		setuptools==60.0.0

		sphinx==4.5.0
		setuptools==60.0.0



		@inject
		def ensure_prometheus_dir(

feat(cli): add CLI command to serve a runner #2920

feat(cli): add CLI command to serve a runner #2920

Conversation

bojiang commented Aug 18, 2022 • edited

codecov bot commented Aug 18, 2022 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sauyon commented Aug 25, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bojiang commented Aug 18, 2022 •

edited

codecov bot commented Aug 18, 2022 •

edited