VLLM support for OpenAI Completions in HF server #3589

gavrishp · 2024-04-09T19:26:18Z

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #3419

Type of changes
Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist:

Have you added unit/e2e tests that prove your fix is effective or that this feature works?
Has code been commented, particularly in hard-to-understand areas?
Have you made corresponding changes to the documentation?

 Support OpenAi completion endpoint in vllm.

python/huggingfaceserver/huggingfaceserver/completions_utils.py

python/huggingface_server.Dockerfile

python/huggingfaceserver/huggingfaceserver/completions_utils.py

python/kserve/kserve/protocol/rest/openai/types/openapi.py

cmaddalozzo · 2024-04-12T21:25:30Z

python/kserve/kserve/protocol/rest/openai/types/openapi.py

Should we just add the license to this file so the check passes?

python/huggingfaceserver/huggingfaceserver/model.py

yuzisun · 2024-04-15T00:00:19Z

python/huggingfaceserver/huggingfaceserver/vllm_completions.py

+            return self.create_error_response("suffix is not currently supported")
+
+        model_name = request.model
+        request_id = f"cmpl-{random_uuid()}"


Request id may need to be fetched from the header x-request-id if user sets from the client side, need to think about how we can pass the headers like KServe model does.

Will need to pass through here, right?
https://github.com/kserve/kserve/blob/master/python/kserve/kserve/protocol/rest/openai/openai_model.py#L62

Since CompletionCreateParams is auto-generated thus can't easily have properties added to it maybe we should introduce a new class to wrap it.
i.e.

class CompletionRequest(BaseModel): request_id: str params: CompletionCreateParams context: Dict # headers can go in here

Yes this looks good

Updated with the changes

yuzisun · 2024-04-15T00:04:42Z

python/huggingfaceserver/huggingfaceserver/vllm_completions.py

+                        prompt_token_ids=input_ids,
+                    )
+                )
+        except ValueError as e:


I think we can consider registering openai error response handler like v1/v2 endpoints

kserve/python/kserve/kserve/protocol/rest/server.py

Line 213 in 9fd262f

InvalidInput: invalid_input_handler,

yuzisun · 2024-04-15T00:06:57Z

python/huggingfaceserver/huggingfaceserver/vllm_completions.py

+    def apply_chat_template(self, messages: Iterable[ChatCompletionMessageParam]):
+        return self.tokenizer.apply_chat_template(conversation=messages, tokenize=False)
+
+    def create_error_response(


We can move the openai error responses to openai_errors.py?

kserve/python/kserve/kserve/errors.py

Line 98 in 9fd262f

async def invalid_input_handler(_, exc):

yuzisun · 2024-04-15T00:08:40Z

python/huggingfaceserver/huggingfaceserver/vllm_completions.py

+            response = self.request_output_to_completion_response(
+                final_res_batch, request, request_id, created_time, model_name
+            )
+        except ValueError as e:


same here we can reuse KServe's error handler

yuzisun · 2024-04-15T00:09:13Z

python/huggingfaceserver/huggingfaceserver/vllm_completions.py

+
+        return response
+
+    async def completion_stream_generator(


Can we add a test for this function?

In current state, I will need to install vllm as the Class needs it. Else I need to remove it as class method. What do you suggest?

python/huggingfaceserver/huggingfaceserver/vllm_completions.py

yuzisun · 2024-04-15T00:21:53Z

python/huggingfaceserver/huggingfaceserver/vllm_completions.py

+        return self.create_error_response(
+            message=f"The model `{request.model}` does not exist.",
+            err_type="NotFoundError",
+            status_code=HTTPStatus.NOT_FOUND,


We can raise the ModelNotFound exception

yuzisun · 2024-04-15T00:25:32Z

python/huggingfaceserver/huggingfaceserver/model.py

@@ -137,6 +145,9 @@ def load(self) -> bool:
                self.vllm_engine = AsyncLLMEngine.from_engine_args(
                    self.vllm_engine_args
                )
+                self.openai_serving_completion = OpenAIServingCompletion(
+                    self.vllm_engine, self.model_name


Suggested change

self.vllm_engine, self.model_name

self.vllm_engine, self.name

pbasov · 2024-04-15T22:49:06Z

Quick /openai/v1/chat/completions test with https://huggingface.co/TheBloke/Llama-2-7B-Chat-AWQ (regular /v2/generate works fine):

from openai import OpenAI

client = OpenAI(api_key="nokey")
client.base_url = "https://llama2-predictor.domain.net/openai/v1"
chat_completion = client.chat.completions.create(
    messages=[
        {   
            "role": "user",
            "content": "Hello world!"
        },
    ],
    model="llama2",
)

2024-04-15 22:15:29.670 1 kserve ERROR [generic_exception_handler():111] Exception:
Traceback (most recent call last):
  File "/prod_venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/prod_venv/lib/python3.10/site-packages/timing_asgi/middleware.py", line 70, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/prod_venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/prod_venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/prod_venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/prod_venv/lib/python3.10/site-packages/starlette/routing.py", line 758, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/prod_venv/lib/python3.10/site-packages/starlette/routing.py", line 778, in app
    await route.handle(scope, receive, send)
  File "/prod_venv/lib/python3.10/site-packages/starlette/routing.py", line 299, in handle
    await self.app(scope, receive, send)
  File "/prod_venv/lib/python3.10/site-packages/starlette/routing.py", line 79, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/prod_venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/prod_venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/prod_venv/lib/python3.10/site-packages/starlette/routing.py", line 74, in app
    response = await func(request)
  File "/prod_venv/lib/python3.10/site-packages/fastapi/routing.py", line 299, in app
    raise e
  File "/prod_venv/lib/python3.10/site-packages/fastapi/routing.py", line 294, in app
    raw_response = await run_endpoint_function(
  File "/prod_venv/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/kserve/kserve/protocol/rest/openai/endpoints.py", line 116, in create_chat_completion
    completion = await self.dataplane.create_chat_completion(
  File "/kserve/kserve/protocol/rest/openai/dataplane.py", line 77, in create_chat_completion
    return await model.create_chat_completion(request)
  File "/kserve/kserve/protocol/rest/openai/openai_model.py", line 249, in create_chat_completion
    chat_prompt = self.apply_chat_template(params.messages)
  File "/huggingfaceserver/huggingfaceserver/model.py", line 339, in apply_chat_template
    prompt=self.openai_serving_completion.apply_chat_template(messages)
  File "/huggingfaceserver/huggingfaceserver/vllm_completions.py", line 339, in apply_chat_template
    return self.tokenizer.apply_chat_template(conversation=messages, tokenize=False)
  File "/prod_venv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1783, in apply_chat_template
    rendered = compiled_template.render(
  File "/prod_venv/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/prod_venv/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/prod_venv/lib/python3.10/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
  File "/prod_venv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1828, in raise_exception
    raise TemplateError(message)
jinja2.exceptions.TemplateError: Conversation roles must alternate user/assistant/user/assistant/...
2024-04-15 22:15:29.670 uvicorn.access INFO:     10.0.0.85:0 1 - "POST /openai/v1/chat/completions HTTP/1.1" 500 Internal Server Error

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

gavrishp · 2024-04-16T12:40:32Z

Quick /openai/v1/chat/completions test with https://huggingface.co/TheBloke/Llama-2-7B-Chat-AWQ (regular /v2/generate works fine):

from openai import OpenAI

client = OpenAI(api_key="nokey")
client.base_url = "https://llama2-predictor.domain.net/openai/v1"
chat_completion = client.chat.completions.create(
    messages=[
        {   
            "role": "user",
            "content": "Hello world!"
        },
    ],
    model="llama2",
)

2024-04-15 22:15:29.670 1 kserve ERROR [generic_exception_handler():111] Exception:
Traceback (most recent call last):
  File "/prod_venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/prod_venv/lib/python3.10/site-packages/timing_asgi/middleware.py", line 70, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/prod_venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/prod_venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/prod_venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/prod_venv/lib/python3.10/site-packages/starlette/routing.py", line 758, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/prod_venv/lib/python3.10/site-packages/starlette/routing.py", line 778, in app
    await route.handle(scope, receive, send)
  File "/prod_venv/lib/python3.10/site-packages/starlette/routing.py", line 299, in handle
    await self.app(scope, receive, send)
  File "/prod_venv/lib/python3.10/site-packages/starlette/routing.py", line 79, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/prod_venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/prod_venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/prod_venv/lib/python3.10/site-packages/starlette/routing.py", line 74, in app
    response = await func(request)
  File "/prod_venv/lib/python3.10/site-packages/fastapi/routing.py", line 299, in app
    raise e
  File "/prod_venv/lib/python3.10/site-packages/fastapi/routing.py", line 294, in app
    raw_response = await run_endpoint_function(
  File "/prod_venv/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/kserve/kserve/protocol/rest/openai/endpoints.py", line 116, in create_chat_completion
    completion = await self.dataplane.create_chat_completion(
  File "/kserve/kserve/protocol/rest/openai/dataplane.py", line 77, in create_chat_completion
    return await model.create_chat_completion(request)
  File "/kserve/kserve/protocol/rest/openai/openai_model.py", line 249, in create_chat_completion
    chat_prompt = self.apply_chat_template(params.messages)
  File "/huggingfaceserver/huggingfaceserver/model.py", line 339, in apply_chat_template
    prompt=self.openai_serving_completion.apply_chat_template(messages)
  File "/huggingfaceserver/huggingfaceserver/vllm_completions.py", line 339, in apply_chat_template
    return self.tokenizer.apply_chat_template(conversation=messages, tokenize=False)
  File "/prod_venv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1783, in apply_chat_template
    rendered = compiled_template.render(
  File "/prod_venv/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/prod_venv/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/prod_venv/lib/python3.10/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
  File "/prod_venv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1828, in raise_exception
    raise TemplateError(message)
jinja2.exceptions.TemplateError: Conversation roles must alternate user/assistant/user/assistant/...
2024-04-15 22:15:29.670 uvicorn.access INFO:     10.0.0.85:0 1 - "POST /openai/v1/chat/completions HTTP/1.1" 500 Internal Server Error

Noticed this. This is due to the Union types having root as the parent parameter. Got this updated in the openai generated types file. It should be fixed now. Let me know, if there's anything else.

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

cmaddalozzo · 2024-04-16T17:10:06Z

/lgtm

pbasov · 2024-04-16T17:12:23Z

@gavrishp just built with fresh code, it works, thank you!

yuzisun · 2024-04-17T13:12:26Z

python/kserve/kserve/protocol/rest/openai/errors.py

+
+async def openai_error_handler(_, exc):
+    logger.error("Exception:", exc_info=exc)
+    return JSONResponse(


I think we need to return the openai defined ErrorResponse type, otherwise it will not be compatible with openai client?

Made changes to send contents of type OpenAI error object

For error scenarios the body looks like this

try: completion = openai.completions.create(model="llama2", prompt="") if stream: for chunk in completion: print(chunk.choices[0].text, datetime.datetime.now()) else: print(completion.choices[0].text) except openai.APIStatusError as e: print("Another non-200-range status code was received") print(e.body)

{'code': '500', 'message': 'Either prompt or prompt_ids should be provided.', 'param': '', 'type': 'OpenAIError'}

OpenAI ErrorObject - https://github.com/openai/openai-python/blob/15e7ac157dcc80d952107f596bb9c38cfa06685d/src/openai/types/shared/error_object.py#L10

yuzisun · 2024-04-17T13:15:10Z

python/kserve/kserve/protocol/rest/openai/config.py

@@ -42,9 +32,6 @@ def get_open_ai_models(repository: ModelRepository) -> List[Model]:


 def maybe_register_openai_endpoints(app: FastAPI, model_registry: ModelRepository):
-    # Check if the openai package is available before continuing so we don't run into any import errors
-    if not openai_is_available():


We might still need this check otherwise openai endpoints get registered to other model serving runtimes?

But we can not use the approach checking if openai dependency exists or not now

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

cmaddalozzo · 2024-04-17T15:49:14Z

python/kserve/kserve/protocol/rest/openai/config.py

@@ -42,9 +32,6 @@ def get_open_ai_models(repository: ModelRepository) -> List[Model]:


 def maybe_register_openai_endpoints(app: FastAPI, model_registry: ModelRepository):
-    # Check if the openai package is available before continuing so we don't run into any import errors
-    if not openai_is_available():
-        return
    open_ai_models = get_open_ai_models(model_registry)
    # If no OpenAI models then no need to add the endpoints
    if len(open_ai_models) == 0:


@yuzisun we only register the OpenAI endpoints if one of more of the registered models extends OpenAIModel

yuzisun · 2024-04-19T16:38:12Z

Thanks @gavrishp !
/lgtm
/approve

oss-prow-bot · 2024-04-19T16:38:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cmaddalozzo, gavrishp, yuzisun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [yuzisun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

oss-prow-bot bot requested review from alexagriffith and andyi2it April 9, 2024 19:26

gavrishp mentioned this pull request Apr 9, 2024

Introduce v2 completions endpoint #3504

Closed

9 tasks

gavrishp force-pushed the vllm_completions branch from 538fe1f to 72920dd Compare April 9, 2024 19:39

terrytangyuan reviewed Apr 10, 2024

View reviewed changes

python/huggingfaceserver/huggingfaceserver/completions_utils.py Outdated Show resolved Hide resolved

Datta0 mentioned this pull request Apr 11, 2024

[WIP] Lora module support for HF and VLLM #3507

Draft

9 tasks

yuzisun reviewed Apr 11, 2024

View reviewed changes

python/huggingface_server.Dockerfile Show resolved Hide resolved

yuzisun reviewed Apr 11, 2024

View reviewed changes

python/huggingfaceserver/huggingfaceserver/completions_utils.py Outdated Show resolved Hide resolved

gavrishp commented Apr 12, 2024

View reviewed changes

python/kserve/kserve/protocol/rest/openai/types/openapi.py Show resolved Hide resolved

cmaddalozzo reviewed Apr 12, 2024

View reviewed changes

python/kserve/kserve/protocol/rest/openai/types/openapi.py

Copy link

Contributor

cmaddalozzo Apr 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we just add the license to this file so the check passes?

yuzisun reviewed Apr 14, 2024

View reviewed changes

python/huggingfaceserver/huggingfaceserver/model.py Outdated Show resolved Hide resolved

yuzisun reviewed Apr 15, 2024

View reviewed changes

python/huggingfaceserver/huggingfaceserver/vllm_completions.py Outdated Show resolved Hide resolved

yuzisun reviewed Apr 15, 2024

View reviewed changes

gavrishp added 10 commits April 16, 2024 12:30

VLLM support for OpenAI Completions in HF server

1c60bbc

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

remove unwanted imports

7bbb614

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

minor fixes

2d8748c

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

fix lint

c93695a

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

fix verify license

b3596fc

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

fix verify license

6cabb9e

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

Change base model

bd2969f

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

fix linter

c36145c

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

fix tests

70b4ee4

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

Fix vllm Base and Chat Completion template

765da81

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

gavrishp added 6 commits April 16, 2024 12:30

ignore file from linter and generate

bdd7f21

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

ignore file from linter and generate

0393999

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

add codege license

3a6695f

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

bring in openai errors

4c34c2d

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

fix linting

c40325a

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

Remove openai import and update openai types codegen cmd

ec65ff5

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

gavrishp force-pushed the vllm_completions branch from f82e18c to ec65ff5 Compare April 16, 2024 12:36

gavrishp added 6 commits April 16, 2024 12:41

remove unwanted import

9c98e85

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

fix tests after conflict

dce2e05

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

fix poetry lock

dd3e039

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

fix poetry lock

361362e

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

remove openai from extras

4a2f770

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

fix logprobs

4457e82

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

oss-prow-bot bot assigned cmaddalozzo Apr 16, 2024

oss-prow-bot bot added the lgtm label Apr 16, 2024

cmaddalozzo approved these changes Apr 16, 2024

View reviewed changes

yuzisun reviewed Apr 17, 2024

View reviewed changes

send json repsone content of type openai error

42362f0

Signed-off-by: Gavrish Prabhu <gavrish.prabhu@nutanix.com>

oss-prow-bot bot removed the lgtm label Apr 17, 2024

cmaddalozzo reviewed Apr 17, 2024

View reviewed changes

oss-prow-bot bot assigned yuzisun Apr 19, 2024

oss-prow-bot bot added the lgtm label Apr 19, 2024

oss-prow-bot bot added the approved label Apr 19, 2024

oss-prow-bot bot merged commit 9cc4eed into kserve:master Apr 19, 2024
56 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VLLM support for OpenAI Completions in HF server #3589

VLLM support for OpenAI Completions in HF server #3589

gavrishp commented Apr 9, 2024

cmaddalozzo Apr 12, 2024

yuzisun Apr 15, 2024

gavrishp Apr 15, 2024

cmaddalozzo Apr 15, 2024

gavrishp Apr 16, 2024

gavrishp Apr 16, 2024

yuzisun Apr 15, 2024 •

edited

yuzisun Apr 15, 2024 •

edited

yuzisun Apr 15, 2024

yuzisun Apr 15, 2024

gavrishp Apr 15, 2024

yuzisun Apr 15, 2024

yuzisun Apr 15, 2024

pbasov commented Apr 15, 2024 •

edited

gavrishp commented Apr 16, 2024

cmaddalozzo commented Apr 16, 2024

pbasov commented Apr 16, 2024

yuzisun Apr 17, 2024 •

edited

gavrishp Apr 17, 2024 •

edited

yuzisun Apr 17, 2024

yuzisun Apr 17, 2024

cmaddalozzo Apr 17, 2024

yuzisun commented Apr 19, 2024

oss-prow-bot bot commented Apr 19, 2024

	self.vllm_engine, self.model_name
	self.vllm_engine, self.name

VLLM support for OpenAI Completions in HF server #3589

VLLM support for OpenAI Completions in HF server #3589

Conversation

gavrishp commented Apr 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuzisun Apr 15, 2024 • edited

Choose a reason for hiding this comment

yuzisun Apr 15, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pbasov commented Apr 15, 2024 • edited

gavrishp commented Apr 16, 2024

cmaddalozzo commented Apr 16, 2024

pbasov commented Apr 16, 2024

yuzisun Apr 17, 2024 • edited

Choose a reason for hiding this comment

gavrishp Apr 17, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuzisun commented Apr 19, 2024

oss-prow-bot bot commented Apr 19, 2024

yuzisun Apr 15, 2024 •

edited

yuzisun Apr 15, 2024 •

edited

pbasov commented Apr 15, 2024 •

edited

yuzisun Apr 17, 2024 •

edited

gavrishp Apr 17, 2024 •

edited