Harrison/add huggingface hub #23
Conversation
| repo_id = values.get("repo_id", DEFAULT_REPO_ID) | ||
| values["client"] = InferenceApi( | ||
| repo_id=repo_id, | ||
| token=os.environ["HUGGINGFACEHUB_API_TOKEN"], |
There was a problem hiding this comment.
These parameters seem like nice ones to include as init args with optional env args as backups.
Also the relationship feels a bit convoluted at first glance
There was a problem hiding this comment.
what in particular seems convoluted?
There was a problem hiding this comment.
I think the main thing is that the client is created in a validation function - but that seems out of scope of this PR given the existing structure
There was a problem hiding this comment.
ah yeah i agree. is a bit weird. may look into post_init
| if "error" in response: | ||
| raise ValueError(f"Error raised by inference API: {response['error']}") | ||
| text = response[0]["generated_text"][len(prompt) :] | ||
| if stop is not None: |
There was a problem hiding this comment.
Could we share this across models?
Yeah it's a bit hacky but I don't see better support via InferenceAPI and ultimately it just ends with a bit of wasted computation
There was a problem hiding this comment.
what do you mean by share across models? its already factored out and used in cohere as well (cohere is a bit different - you can pass stop words but they are included at the end of the prompt)
There was a problem hiding this comment.
Yeah this is sufficient - not enough examples to merit more
| from typing import List | ||
|
|
||
|
|
||
| def enforce_stop_tokens(text: str, stop: List[str]) -> str: |
There was a problem hiding this comment.
Double checking - the prompt isn't returned as part of the generated text, right?
There was a problem hiding this comment.
no it shouldnt be
…lback_docs Update examples to use inheritable callbacks
Add support for huggingface hub
I could not find a good way to enforce stop tokens over the huggingface hub api - that needs to hopefully be cleaned up in the future