You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
MetaDescription: Find a popular generative AI model by publisher and source. Bring your own model that is hosted with a URL, or select an Ollama model.
5
5
---
6
-
# Models in AI Toolkit
6
+
# Explore models in AI Toolkit
7
7
8
-
AI Toolkit supports a broad range of generative AI models. Both Small Language Models (SLM) and Large Language Models (LLM) are supported.
8
+
AI Toolkit provides comprehensive support for a wide variety of generative AI models, including both Small Language Models (SLMs) and Large Language Models (LLMs).
9
9
10
-
In the model catalog, you can access models from various sources:
10
+
Within the model catalog, you can explore and utilize models from multiple hosting sources:
- Publisher-hosted models (OpenAI ChatGPT models, Anthropic Claude, Google Gemini)
14
-
- Locally downloaded models, for example from HuggingFace
15
-
- Locally running Ollama models
16
-
- Connect to Bring-Your-Own-Models
12
+
- Models hosted on GitHub, such as Llama3, Phi-3, and Mistral.
13
+
- Models provided directly by publishers, including OpenAI's ChatGPT, Anthropic's Claude, and Google's Gemini.
14
+
- Models downloaded locally from repositories like Ollama and ONNX.
15
+
- Custom self-hosted or externally deployed models accessible via Bring-Your-Own-Model (BYOM) integration.
16
+
17
+
17
18
18
19
## Find a model
19
20
20
21
To find a model in the model catalog:
21
22
22
23
1. Select the AI Toolkit view in the Activity Bar
23
-
24
-
1. Select **CATALOG** > **Models** to open the model catalog
25
-
26
-
27
-
28
-
Select a model card in the model catalog to view more details of the selected model.
29
-
24
+
1. Select **MODELS** > **Catalog** to open the model catalog
30
25
1. Use the filters to reduce the list of available models
31
26
32
27
-**Hosted by**: AI Toolkit supports GitHub, ONNX, OpenAI, Anthropic, Google as model hosting sources.
33
-
34
28
-**Publisher**: The publisher for AI models, such as Microsoft, Meta, Google, OpenAI, Anthropic, Mistral AI, and more.
35
-
36
-
-**Tasks**: Currently, only `Text Generation` is supported.
37
-
29
+
-**Feature**: Supported features of the model, such as `Text Attachment`, `Image Attachment`, `Web Search`, `Structured Outputs`, and more.
38
30
-**Model type**: Filter models that can run remotely or locally on CPU, GPU, or NPU. This filter depends on the local availability.
39
-
40
31
-**Fine-tuning Support**: Show models that can be used to run fine-tuning.
32
+
1. Browse the models in different categories, such as:
33
+
-**Popular Models** is a curated list of widely used models across various tasks and domains.
34
+
-**GitHub Models** provide easy access to popular models hosted on GitHub. It's best for fast prototyping and experimentation.
35
+
-**ONNX Models** are optimized for local execution and can run on CPU, GPU, or NPU.
36
+
-**Ollama Models** are popular models that can run locally with Ollama, supporting CPU via GGUF quantization.
37
+
1. Alternatively, use the search box to find a specific model by name or description
41
38
42
-
To reference a self-hosted model or locally-running Ollama model:
39
+
## Add a model from the catalog
40
+
To add a model from the model catalog:
41
+
1. Locate the model you want to add in the model catalog
42
+
1. Select the **Add** on the model card
43
+
1. The flow for adding models will be slightly different based on the providers:
43
44
44
-
1. Select **+ Add model** in the model catalog
45
+
-**GitHub**: AI Toolkit asks for your GitHub credentials to access the model repository. Once authenticated, the model is added directly into AI Toolkit.
46
+
-**ONNX**: The model is downloaded from ONNX and added to AI Toolkit.
47
+
-**Ollama**: The model is downloaded from Ollama and added to AI Toolkit.
45
48
46
-
1. Choose between Ollama or a custom model in the model Quick Pick
49
+
> [!TIP]
50
+
> You can edit the API key later by right clicking the model and selecting **Edit** and view the encrypted value in `${HOME}/.aikt/models/my-models/yml` file.
51
+
> 
47
52
48
-
1. Provide details to add the model
53
+
-**OpenAI**, **Anthropic**, and **Google**: AI Toolkit prompts you to enter the API Key.
54
+
-**Custom models**: Refer to the [Add a custom model](#add-a-custom-model) section for detailed instructions.
49
55
50
-
## License and sign-in
56
+
Once added, the model appears under **MY MODELS** in the tree view, and you can use it in the [**Playground**](/docs/intelligentapps/playground.md) or [**Agent Builder**](/docs/intelligentapps/agentbuilder.md).
51
57
52
-
Some models require a publisher or hosting-service license and account to sign-in. In that case, before you can run the model in the [model playground](/docs/intelligentapps/playground.md), you are prompted to provide this information.
53
-
54
-
## Select a model for testing
55
-
56
-
AI Toolkit enables you to test run a model in the playground for chat completions. You have different options, available through the actions on the model card in the model catalog.
57
-
58
-
-**Try in Playground**: load the selected model for testing in the playground without downloading it
59
-
-**Download**: download the model from a source like Hugging Face
60
-
-**Load in Playground**: load a downloaded model into the playground for chat
58
+
## Add a custom model
59
+
You can also add your own models that are hosted externally or run locally. There are several options available:
60
+
- Add Ollama models from the Ollama library or custom Ollama endpoints.
61
+
- Add custom models that have an OpenAI compatible endpoint, such as a self-hosted model or a model running on a cloud service.
62
+
- Add custom ONNX models, such as those from Hugging Face, using AI Toolkit's [model conversion tool](/docs/intelligentapps/modelconversion.md).
61
63
62
-
## Bring your own models
64
+
There are several entrypoints to add models to AI Toolkit:
65
+
- From **MY MODELS** in the tree view, hover over it and select the `+` icon.
66
+
63
67
64
-
AI Toolkit's playground also supports remote models. If you have a self-hosted or deployed model that is accessible from the internet, you can add it to AI Toolkit and use it in the playground.
68
+
- From the **Model Catalog**, select the **+ Add model** button from the tool bar.
69
+
65
70
66
-
1. Hover over**MY MODELS** in the tree view, and select the `+` icon to add a remote model into AI Toolkit.
67
-
1. Fill in the requested information, such as model name, display name, model hosting URL, and optional auth string.
71
+
- From the**Add Custom Models**section in the model catalog, select **+ Add Your Own Model**.
72
+
68
73
69
-
70
-
71
-
## Add Ollama models
74
+
### Add Ollama models
72
75
73
76
Ollama enables many popular genAI models to run locally with CPU via GGUF quantization. If you have Ollama installed on your local machine with downloaded Ollama models, you can add them to AI Toolkit for use in the model playground.
74
77
75
-
### Prerequisites
78
+
Prerequisites for using Ollama models in AI Toolkit:
76
79
77
80
- AI Toolkit v0.6.2 or newer.
78
81
-[Ollama](https://ollama.com/download) (Tested on Ollama v0.4.1)
79
82
80
-
### Add local Ollama into AI Toolkit
81
-
82
-
1. Hover over **MY MODELS** in the tree view and select the "+" icon to add a model
83
+
To add local Ollama into AI Toolkit
83
84
84
-
Alternatively, select the **+ Add model** button in the model catalog or playground.
85
-
86
-
1. Select **Add an Ollama model**
85
+
1. From one of the entrypoints mentioned above, select **Add Ollama Model**.
87
86
88
87
89
88
@@ -100,3 +99,50 @@ Ollama enables many popular genAI models to run locally with CPU via GGUF quanti
100
99
101
100
> [!NOTE]
102
101
> Attachment is not support yet for Ollama models. Since we connect to Ollama using its [OpenAI compatible endpoint](https://github.com/ollama/ollama/blob/main/docs/openai.md) and it doesn't support attachments yet.
102
+
103
+
### Add a custom model with OpenAI compatible endpoint
104
+
105
+
If you have a self-hosted or deployed model that is accessible from the internet with an OpenAI compatible endpoint, you can add it to AI Toolkit and use it in the playground.
106
+
107
+
1. From one of the entry points above, select **Add Custom Model**.
108
+
1. Enter the OpenAI compatible endpoint URL and the required information.
109
+
110
+
To add a self-hosted or locally running Ollama model:
111
+
112
+
1. Select **+ Add model** in the model catalog.
113
+
1. In the model Quick Pick, choose **Ollama** or **Custom model**.
114
+
1. Enter the required details to add the model.
115
+
116
+
### Add a custom ONNX model
117
+
118
+
To add a custom ONNX model, first convert it to the AI Toolkit model format using the [model conversion tool](/docs/intelligentapps/modelconversion.md). After conversion, add the model to AI Toolkit.
119
+
120
+
## Select a model for testing
121
+
122
+
You can test a model in the playground for chat completions.
123
+
124
+
Use the actions on the model card in the model catalog:
125
+
126
+
-**Try in Playground**: Load the selected model for testing in the [Playground](/docs/intelligentapps/playground.md).
127
+
-**Try in Agent Builder**: Load the selected model in the [Agent Builder](/docs/intelligentapps/agentbuilder.md) to build AI agents.
128
+
129
+
## Manage models
130
+
You can manage your models in the **MY MODELS** section of the AI Toolkit view. Here you can:
131
+
- View the list of models you have added to AI Toolkit.
132
+
- Right-click on a model to access options such as:
133
+
-**Load in Playground**: Load the model in the [Playground](/docs/intelligentapps/playground.md) for testing.
134
+
-**Copy Model Name**: Copy the model name to the clipboard for use in other contexts, such as your code integration.
135
+
-**Refresh**: Refresh the model configuration to ensure you have the latest settings.
136
+
-**Edit**: Modify the model settings, such as the API key or endpoint.
137
+
-**Delete**: Remove the model from AI Toolkit.
138
+
-**About this Model**: View detailed information about the model, including its publisher, source, and supported features.
139
+
140
+
- Right-click on `ONNX` section title to access options such as:
141
+
-**Start Server**: Start the ONNX server to run ONNX models locally.
142
+
-**Stop Server**: Stop the ONNX server if it is running.
143
+
-**Copy Endpoint**: Copy the ONNX server endpoint to the clipboard for use in other contexts, such as your code integration.
144
+
145
+
## License and sign-in
146
+
147
+
Some models require a publisher or hosting-service license and account to sign-in. In that case, before you can run the model in the [model playground](/docs/intelligentapps/playground.md), you are prompted to provide this information.
0 commit comments