Implement support for vllm as alternative backend

[gavrishp]

What this PR does / why we need it:
Support vllm as alternative backend for text generation use-cases

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Type of changes
Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Feature/Issue validation/testing:

Please describe the tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.
Have not added tests as vllm expects gpu driver as prerequiste

• Logs

Special notes for your reviewer:

1. Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Checklist:

• Have you added unit/e2e tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?

Release note:

[terrytangyuan]

Should this be in draft?

[terrytangyuan]

Missing default here

[gavrishp]

This is on/off flag argument not an option that takes value.

[yuzisun]

Why the default is true? I think we should default to false unless user wants to explicitly turn it off and let it fallback to huggingface API.

[gavrishp]

action='store_true' provide capability to set bool value if the flag is set. In this case if --disable_vllm is set, the value of arg.disable_vllm will be true and in all other cases it is false

https://docs.python.org/3/library/argparse.html#action

The experience is if --disable_vllm is set only then vLLM will not be used explicitly. if this flag is not set then vLLM to be used, with huggingface API as fallback

[terrytangyuan]

We can just assign directly to model_id without checking

[yuzisun]

Is there a reference where these should be default to be disabled?

[gavrishp]

I noticed in latest python versions parser.add_argument(type=bool, ..) for type bool() does not work as expected. It sets values as string.

https://docs.python.org/3/library/argparse.html#type
https://stackoverflow.com/questions/60999816/argparse-not-parsing-boolean-arguments
https://docs.python.org/3/library/argparse.html#action

To support it I added the same functionality by inverting as expectation is to have add_special_tokens as true by default. Hence unless --disable_special_tokens is set add_special_tokens will be True. Similarly for vllm and lower_case

[yuzisun]

Do we need the use_vllm if we can determine the list of models architecture vllm can support?

[gavrishp]

Thought going this route to set whether to use vLLM instead of using the global variable _vllm
https://github.com/kserve/kserve/pull/3415/files#diff-d02c376ad82bad8a11ceaecb4ea815822123e34b961f29fd5050225af82e62b4R80
self.use_vllm = not kwargs.get('disable_vllm', False) if _vllm else False

This is a pre-check if someone explicitly disables vLLM or does not have vLLM package in local.

[terrytangyuan]

HuggingFace out-of-box support has been added in case you'd like to test it together. @gavrishp #3395

[gavrishp]

HuggingFace out-of-box support has been added in case you'd like to test it together. @gavrishp #3395

That's great. I'll give it a try

[yuzisun]

I think we probably we do not want to confuse user that they have to set this config to get Bert to work. I think a better example would be the case that vllm is by default and show an example that it can be disabled.

[gavrishp]

I have added a disable vllm example

[lizzzcai]

I just trying out the new huggingface runtime and notice this. I feel like a bit werid to have vllm as default runtime (as the PR mentioned vllm as alternative backend).

how about --backend=vllm, having huggingface as default and vllm as alternative and future can support other backend like tensorRT

what do you think? @yuzisun

[yuzisun]

Great work on this @gavrishp !

/lgtm
/approve

[oss-prow-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gavrishp, yuzisun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [yuzisun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment