OpenAI schema #3477

tessapham · 2024-02-26T04:27:28Z

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #3419.

Type of changes
Please delete options that are not relevant.

New feature (non-breaking change which adds functionality)

Feature/Issue validation/testing:

Please describe the tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

Test A
Test B
Logs

Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Checklist:

Have you added unit/e2e tests that prove your fix is effective or that this feature works?
Has code been commented, particularly in hard-to-understand areas?
Have you made corresponding changes to the documentation?

Release note:

python/kserve/kserve/protocol/rest/openai_datamodels.py

python/kserve/kserve/protocol/rest/server.py

python/kserve/kserve/protocol/dataplane.py

docker-compose.yml

terrytangyuan · 2024-04-01T15:11:27Z

python/kserve/kserve/protocol/dataplane.py

+                    body._attributes["content-type"] == "application/cloudevents+json"
+                    or body._attributes["content-type"] == "application/json"


Use JSON_HEADERS?

It'll be harder to read since JSON_HEADERS is just a list, so it'll just be JSON_HEADERS[0], JSON_HEADERS[1].

terrytangyuan · 2024-04-01T15:12:55Z

python/kserve/kserve/protocol/rest/openai/config.py

+    """Retrieve all models in the repository that implement the OpenAI interface"""
+    from .openai_model import OpenAIModel
+
+    return [model for _, model in repository.get_models().items() if isinstance(model, OpenAIModel)]


Will this load ALL models in-memory or only the model that will be served?

pkg/agent/storage/https.go

yuzisun · 2024-04-01T22:13:18Z

python/kserve/kserve/protocol/rest/openai/dataplane.py

+
+        Args:
+            model_name (str): Model name.
+            request (bytes|GenerateRequest|ChatCompletionRequest): Generate Request / ChatCompletion Request body data.


should be CompletionCreateParams

yuzisun · 2024-04-01T22:13:32Z

python/kserve/kserve/protocol/rest/openai/dataplane.py

+
+        Args:
+            model_name (str): Model name.
+            request (bytes|GenerateRequest|ChatCompletionRequest): Generate Request / ChatCompletion Request body data.


should be ChatCompletionCreateParams

python/kserve/kserve/protocol/rest/openai/dataplane.py

yuzisun · 2024-04-01T22:16:55Z

python/kserve/kserve/protocol/rest/openai/openai_model.py

+    async def create_chat_completion(
+        self, params: ChatCompletionCreateParams
+    ) -> Union[ChatCompletion, AsyncIterator[ChatCompletionChunk]]:
+        if params.get("n", 1) != 1:


how about 0 which I guess it is the default if not set?

n = 1 by default as n indicates the number of chat completions to generate for an input message.

python/lgbserver/poetry.lock

pkg/agent/storage/https.go

yuzisun · 2024-04-02T02:02:54Z

python/kserve/kserve/protocol/rest/openai/__init__.py

@@ -0,0 +1,3 @@
+from .openai_model import ChatPrompt, OpenAIChatAdapterModel, OpenAIModel


Still missing the license from this file

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

yuzisun · 2024-04-05T13:06:23Z

Awesome work, thanks @tessapham @cmaddalozzo !

/lgtm
/approve

oss-prow-bot · 2024-04-05T13:06:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tessapham, yuzisun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [yuzisun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

lizzzcai · 2024-04-08T09:46:38Z

python/kserve/kserve/protocol/rest/openai/endpoints.py

+
+def register_openai_endpoints(app: FastAPI, dataplane: OpenAIDataPlane):
+    endpoints = OpenAIEndpoints(dataplane)
+    openai_router = APIRouter(prefix=OPENAI_ROUTE_PREFIX, tags=["OpenAI"])


Maybe a late comment. I think/v1 should be in the OPENAI_ROUTE_PREFIX from openai openapi file and the SDK instead of route, so user can overwrite the base_url/prefix.

any comment? @yuzisun

* OpenAI data models and endpoints from vLLM Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * more components for OpenAI endpoints Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * add OpenAI endpoints to router Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * modify generate() in data plane Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * class OpenAIModel Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * delete and rename files Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * add create_chat_completion() to OpenAIModel Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * update routers and lint Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * Implement streaming Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net> * Add tests for OpenAI data conversion Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net> * Register OpenAI endpoints when appropriate Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net> * Add comments Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net> * Add tests for create_completion and create_chat_completion Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net> * Remove completion types from dataplane methods Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net> * WIP Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net> * fix lint errors Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * update poetry.lock Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * update poetry.lock files Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * add dependency Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * fix test Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * revert poetry.lock files Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * add .itermconfig to .gitignore Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * add docker-compose.yml to .gitignore Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * fix build error Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * fix function descriptions Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * increase limit for model decompression size Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * add license & autoformat Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * make openai dependency mandatory Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * openai dependency back to optional Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * fix openai module import error Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * fix JSON unmarshalling of headers Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * drop formatting changes in unrelated files Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * fix openai_is_available() Signed-off-by: Tessa Pham <hpham111@bloomberg.net> * black reformat Signed-off-by: Tessa Pham <hpham111@bloomberg.net> --------- Signed-off-by: Tessa Pham <hpham111@bloomberg.net> Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net> Co-authored-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net> Signed-off-by: tjandy98 <3953059+tjandy98@users.noreply.github.com>

oss-prow-bot bot requested review from andyi2it and sivanantha321 February 26, 2024 04:27

tessapham marked this pull request as draft February 26, 2024 04:27

oss-prow-bot bot added the do-not-merge/work-in-progress label Feb 26, 2024

tessapham force-pushed the openai branch from 8d0bc85 to 4836bc5 Compare March 4, 2024 16:08

johnugeorge reviewed Mar 6, 2024

View reviewed changes

python/kserve/kserve/protocol/rest/openai_datamodels.py Outdated Show resolved Hide resolved

tessapham force-pushed the openai branch from 4836bc5 to 743808e Compare March 7, 2024 22:37

yuzisun reviewed Mar 10, 2024

View reviewed changes

python/kserve/kserve/protocol/rest/server.py Outdated Show resolved Hide resolved

yuzisun force-pushed the openai branch 3 times, most recently from 3c52af3 to 591060d Compare March 11, 2024 13:12

tessapham force-pushed the openai branch from 591060d to 728ec30 Compare March 13, 2024 14:46

gavrishp reviewed Mar 17, 2024

View reviewed changes

python/kserve/kserve/protocol/dataplane.py Outdated Show resolved Hide resolved

gavrishp reviewed Mar 17, 2024

View reviewed changes

python/kserve/kserve/protocol/dataplane.py Outdated Show resolved Hide resolved

tessapham force-pushed the openai branch from 54d2914 to c2c1a35 Compare April 1, 2024 15:10

terrytangyuan reviewed Apr 1, 2024

View reviewed changes

tessapham force-pushed the openai branch from eee59d9 to 77b8924 Compare April 1, 2024 15:52

yuzisun reviewed Apr 1, 2024

View reviewed changes

pkg/agent/storage/https.go Outdated Show resolved Hide resolved

tessapham force-pushed the openai branch 5 times, most recently from 3ba860b to a6c7510 Compare April 1, 2024 22:08

yuzisun reviewed Apr 1, 2024

View reviewed changes

tessapham force-pushed the openai branch 2 times, most recently from a2bc2d8 to d6f95ef Compare April 2, 2024 00:06

github-advanced-security bot found potential problems Apr 2, 2024

View reviewed changes

pkg/agent/storage/https.go Fixed Show fixed Hide fixed

tessapham force-pushed the openai branch 2 times, most recently from 78e1190 to 270e0ed Compare April 2, 2024 00:47

yuzisun reviewed Apr 2, 2024

View reviewed changes

tessapham added 13 commits April 2, 2024 21:07

revert poetry.lock files

c3c6b97

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

add .itermconfig to .gitignore

ef1fcb0

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

add docker-compose.yml to .gitignore

124cb43

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

fix build error

1016759

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

fix function descriptions

4881a63

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

increase limit for model decompression size

71c1940

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

add license & autoformat

40f0053

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

make openai dependency mandatory

044cc4a

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

openai dependency back to optional

63f7cd7

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

fix openai module import error

2f51c55

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

fix JSON unmarshalling of headers

288b30d

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

drop formatting changes in unrelated files

c476aec

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

fix openai_is_available()

1cbd5b0

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

tessapham force-pushed the openai branch from 9cfecbb to 1cbd5b0 Compare April 3, 2024 01:08

black reformat

fd5cea7

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

yuzisun marked this pull request as ready for review April 3, 2024 07:26

oss-prow-bot bot removed the do-not-merge/work-in-progress label Apr 3, 2024

oss-prow-bot bot requested review from lizzzcai and rachitchauhan43 April 3, 2024 07:26

oss-prow-bot bot assigned yuzisun Apr 5, 2024

oss-prow-bot bot added the lgtm label Apr 5, 2024

oss-prow-bot bot added the approved label Apr 5, 2024

oss-prow-bot bot merged commit 80bf4a0 into kserve:master Apr 5, 2024
56 checks passed

yuzisun mentioned this pull request Apr 7, 2024

Introduce v2 completions endpoint #3504

Closed

9 tasks

lizzzcai reviewed Apr 8, 2024

View reviewed changes

terrytangyuan mentioned this pull request Apr 9, 2024

Add OpenAI API support to Huggingfaceserver #3582

Merged

7 tasks

grandbora mentioned this pull request Apr 9, 2024

Support OpenAIModel in ModelRepository #3590

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenAI schema #3477

OpenAI schema #3477

tessapham commented Feb 26, 2024

terrytangyuan Apr 1, 2024

tessapham Apr 1, 2024

terrytangyuan Apr 1, 2024

yuzisun Apr 1, 2024

yuzisun Apr 1, 2024

yuzisun Apr 1, 2024

tessapham Apr 1, 2024

yuzisun Apr 2, 2024

yuzisun commented Apr 5, 2024

oss-prow-bot bot commented Apr 5, 2024

lizzzcai Apr 8, 2024

		body._attributes["content-type"] == "application/cloudevents+json"
		or body._attributes["content-type"] == "application/json"

		@@ -0,0 +1,3 @@
		from .openai_model import ChatPrompt, OpenAIChatAdapterModel, OpenAIModel

OpenAI schema #3477

OpenAI schema #3477

Conversation

tessapham commented Feb 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuzisun commented Apr 5, 2024

oss-prow-bot bot commented Apr 5, 2024

Choose a reason for hiding this comment