Skip to content
This repository has been archived by the owner on May 28, 2024. It is now read-only.

Fix serve config parsing #66

Merged
merged 3 commits into from
Oct 3, 2023
Merged

Fix serve config parsing #66

merged 3 commits into from
Oct 3, 2023

Conversation

gvspraveen
Copy link
Contributor

@gvspraveen gvspraveen commented Oct 3, 2023

This seems to have been accidentally removed in previous PR
serve config.yaml expects args to be dictionary. See existing readme in ray-llm repo

Accidentally remove RouterArgs parsing is causing the following issue

pydantic.error_wrappers.ValidationError: 2 validation errors for VLLMApp
engine_config
  field required (type=value_error.missing)
scaling_config
  field required (type=value_error.missing)

Tested locally with following serve config and verified with fix i am able to start server

applications:
- name: meta-llama--Llama-2-7b-chat-hf
  route_prefix: /
  import_path: aviary.backend:router_application
  args:
    models:
      - "path to model"

Looks like others are also facing this: https://ray-distributed.slack.com/archives/C053J39MJ3A/p1696320039604519

@@ -508,7 +508,7 @@ class AppArgs(BaseModel):


class RouterArgs(BaseModel):
models: Dict[str, Union[str, LLMApp]]
models: Union[str, LLMApp, List[Union[LLMApp, str]]]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new typing doesn't support Dict[str, Union[str, LLMApp]] (the current typing). @Yard1 should we support that type?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't have to anymore, no

Copy link
Contributor

@shrekris-anyscale shrekris-anyscale left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this @gvspraveen!

@Yard1 Yard1 merged commit b9bdaac into master Oct 3, 2023
2 checks passed
@Yard1 Yard1 deleted the pg-fix-router-config branch October 3, 2023 23:03
Yard1 added a commit that referenced this pull request Oct 3, 2023
Yard1 added a commit that referenced this pull request Oct 3, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants