-
-
Notifications
You must be signed in to change notification settings - Fork 76
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor backend -> model #176
Changes from all commits
51ca50c
90739e5
3db4ffb
cbd578c
a95040a
f39f1d4
2611542
06c22fd
f78adfc
4b967dd
9e1f406
7b96692
4b4002f
26e3c6c
6cd6101
b068e1d
01c6ad7
05c4ee2
049e8de
70fa932
670ca68
a3dafc4
8b3bab3
b1916ed
1a386fe
8d6ae3f
b9305f4
502697f
45b0a21
5dd2ad4
8ebe1d8
bb194c6
2893606
0cc07ab
011885b
4f18796
5b537fd
e7c556c
f3333f6
81a535e
5bc5993
7ea3ec0
3590ae9
ed88b0f
6c4bc79
912a75e
57bc905
d3f3e74
fee5cef
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,85 @@ | ||
# Migration guides | ||
|
||
<details open> | ||
<summary>0.3.x to 0.4.x</summary> | ||
|
||
## `0.3.x` to `0.4.x` | ||
|
||
`0.4.x` significantly refactors the code to make it more robust and the config more intuitive. 0.4.0 changes the config | ||
paradigm from `backend`- to `model`-centric. This is reflected in the external API in a different config structure. | ||
|
||
Remember that there are three different types of models: the first uses the native REST implementation to communicate | ||
with hosted LLMs, the second builds on HuggingFace's `transformers` model to run models locally and the third leverages | ||
`langchain` to operate on hosted or local models. While the config for all three is rather similar (especially in | ||
0.4.x), there are differences in how these models have to be configured. We show how to migrate your config from 0.3.x | ||
to 0.4.x for each of these model types. | ||
|
||
### All model types | ||
- The registry name has changed - instead of `@llm_backends`, use `@llm_models`. | ||
- The `api` attribute has been removed. | ||
|
||
### Models using REST | ||
|
||
This is the default method to communicate with hosted models. Whenever you don't explicitly use LangChain models | ||
(see section at the bottom) or run models locally, you are using this kind of model. | ||
|
||
In `0.3.x`: | ||
```ini | ||
[components.llm.backend] | ||
@llm_backends = "spacy.REST.v1" | ||
api = "OpenAI" | ||
config = {"model": "gpt-3.5-turbo", "temperature": 0.3} | ||
``` | ||
In `0.4.x`: | ||
```ini | ||
[components.llm.model] | ||
@llm_models = "spacy.gpt-3-5.v1" | ||
name = "gpt-3-5-turbo" | ||
config = {"temperature": 0.3} | ||
``` | ||
Note that the factory function (marked with `@`) refers to the name of the model. Variants of the same model can be | ||
specified with the `name` attribute - for `gpt-3.5` this could be `"gpt-3-5-turbo"` or `"gpt-3-5-turbo-16k"`. | ||
|
||
### Models using HuggingFace | ||
|
||
On top of the changes described in the section above, HF models like `spacy.Dolly.v1` now accept `config_init` and | ||
`config_run` to reflect that differerent arguments can be passed at init or run time. | ||
|
||
In `0.3.x`: | ||
```ini | ||
[components.llm.backend] | ||
@llm_backends = "spacy.Dolly_HF.v1" | ||
model = "databricks/dolly-v2-3b" | ||
config = {} | ||
``` | ||
In `0.4.x`: | ||
```ini | ||
[components.llm.model] | ||
@llm_models = "spacy.Dolly.v1" | ||
name = "dolly-v2-3b" # or databricks/dolly-v2-3b - the prefix is optional | ||
config_init = {} # Arguments passed to HF model at initialization time | ||
config_run = {} # Arguments passed to HF model at inference time | ||
``` | ||
|
||
### Models using LangChain | ||
|
||
LangChain models are now accessible via `langchain.[API].[version]`, e. g. `langchain.OpenAI.v1`. Other than that the | ||
changes from 0.3.x to 0.4.x are identical with REST-based models. | ||
|
||
In `0.3.x`: | ||
```ini | ||
[components.llm.backend] | ||
@llm_backends = "spacy.LangChain.v1" | ||
api = "OpenAI" | ||
config = {"temperature": 0.3} | ||
``` | ||
|
||
In `0.4.x`: | ||
```ini | ||
[components.llm.model] | ||
@llm_models = "langchain.OpenAI.v1" | ||
name = "gpt-3-5-turbo" | ||
config = {"temperature": 0.3} | ||
``` | ||
|
||
</details> | ||
rmitsch marked this conversation as resolved.
Show resolved
Hide resolved
|
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"above" is confusing here, because the section above is about the REST API. I would just remove this subsection, and start with "HF models like ..."