Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support passing model parameters in pipelines #500

Closed
davidmezzetti opened this issue Jul 8, 2023 · 0 comments
Closed

Support passing model parameters in pipelines #500

davidmezzetti opened this issue Jul 8, 2023 · 0 comments
Assignees
Milestone

Comments

@davidmezzetti
Copy link
Member

Currently, all Hugging Face based pipelines have the following constructor.

def __init__(self, path=None, quantize=False, gpu=True, model=None):

This allows setting the model path and target GPU device but prevents many necessary parameters from being passed to the models. For example, many of the new LLMs being released require the trust_remote_code parameter to be set. Running 4 and 8 bit quantization is another example.

The current workaround is to create the models via the auto model classes and pass those to the pipelines. This works in Python but not with application YAML.

This issue will do the following:

  • Add kwargs parameters to all constructors and pass those parameters to the underlying pipeline
  • Resolve arguments from strings as necessary (for example torch_dtype)
@davidmezzetti davidmezzetti added this to the v5.6.0 milestone Jul 8, 2023
@davidmezzetti davidmezzetti self-assigned this Jul 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant