Remove modelRouter and add model_providers concept by philipph-askui · Pull Request #224 · askui/python-sdk

philipph-askui · 2026-01-21T07:20:44Z

[Edited] PR Description

To get a quick idea of the new API, I added an example under examples/model_providers.py that you can run
Note: You will need an anthropic API Key for that

This PR:

Removes ModelRouter, ModelRegistry, and model_store
Introduces a provider-based configuration system (AgentSettings)
Renames VisionAgent → ComputerAgent and AndroidVisionAgent → AndroidAgent
Updates docs and examples

Summary

Replaced the ModelRouter/model_store abstraction with three typed provider slots (vlm_provider, image_qa_provider, detection_provider) configured via AgentSettings
Providers own their endpoint, credentials, and model ID — validated lazily on first API call
get() and locate() are now backed by GetTool/LocateTool, which are also available to the LLM during act() — no separate model injection path
Significantly reduced codebase complexity (~4600 lines removed)
Updated all docs and examples to reflect the new API

Key Changes

AgentSettings: single configuration object with provider slots; defaults to AskUI-hosted providers reading credentials from env vars
Built-in providers: AskUIVlmProvider, AskUIImageQAProvider, AskUIDetectionProvider, AnthropicVlmProvider, AnthropicImageQAProvider, GoogleImageQAProvider, OpenAICompatibleProvider
GetTool / LocateTool: wired into the act loop as ToolWithAgentOS — LLM can call them directly during act()
Deleted: entire src/askui/model_store/ directory
Renamed: VisionAgent → ComputerAgent, AndroidVisionAgent → AndroidAgent
Docs: 03_Using-Models-and-BYOM.md fully rewritten; VisionAgent replaced with ComputerAgent across all docs

Further Changes

removes chat-api related code as it was deprecated
removes UI-TARS related code as it was deprecated

Breaking Changes
- VisionAgent is removed — use ComputerAgent
- AndroidVisionAgent is removed — use AndroidAgent
- act_model, get_model, locate_model constructor parameters are removed — use AgentSettings(vlm_provider=..., image_qa_provider=..., detection_provider=...)
- model_store factory functions are removed
- String-based model selection is removed

…del store BREAKING CHANGE: Removed ModelRouter and ModelRegistry classes. Users must now use direct model injection.

…modelrouter

…raises an Error when executing from cache)

…move() methods

…nd AskUiInferenceLocateApi

…ing` group

philipph-askui · 2026-02-16T07:57:21Z

I added a new optional dependency group "tracing", that contains all the OTEL packages.

src/askui/models/openai/messages_api.py

src/askui/models/shared/tools.py

src/askui/model_providers/askui_vlm_provider.py

src/askui/models/google/get_model.py

src/askui/models/openai/act_model.py

programminx-askui · 2026-02-16T15:56:37Z

src/askui/models/openai/messages_api.py

+from askui.models.shared.tools import ToolCollection
+
+
+def _to_openai_messages(messages: list[MessageParam]) -> list[dict[str, Any]]:  # noqa: C901


Should I review this?

I would expect here an integration tests. Which coveres all message types.

src/askui/models/openai/messages_api.py

src/askui/model_providers/anthropic_vlm_provider.py

src/askui/model_providers/askui_image_qa_provider.py

examples/model_providers.py

…tic `provider_options`

docs/00_Overview.md

pyproject.toml

docs/01_Setup.md

programminx-askui · 2026-02-23T08:27:39Z

docs/03_Using-Models-and-BYOM.md

@@ -0,0 +1,886 @@
+# Using Models and BYOM (Bring Your Own Model)


SnakeCase

Rename to "Bring Your Own Model Provider"

docs/06_Observability-Telemetry-Tracing.md

docs/05_Tools.md

programminx-askui · 2026-02-23T08:59:41Z

docs/05_Tools.md

+| Adaptive to UI changes | ❌ | ✅ |
+| Require no AI inference | ✅ | ❌ |
+
+**Example combining both approaches:**


Please add https://app.usepylon.com/kb/2d7bee45-8af7-4b1b-830b-603ce644124d/articles/03e7604a-bceb-4903-a215-f802f1c9f11e

And deprecrate pylon docu

addressed in 389b649

i suggest to wait with deprecating the pylon page until this is merged. Or do you want to deprecate it right now?

… and test reports

programminx-askui

I like the new document structure. much more clearer

docs/08_reporting.md

docs/11_file_support.md

philipph-askui added 3 commits January 21, 2026 06:57

chore: add plan and claude rules

b99056d

refactor: simplify model selection with direct model injection and mo…

7a23ca9

…del store BREAKING CHANGE: Removed ModelRouter and ModelRegistry classes. Users must now use direct model injection.

resolve merge conflict

421d043

philipph-askui changed the title ~~Chore/modelrouter~~ Remove Modelrouter Jan 21, 2026

philipph-askui and others added 12 commits January 21, 2026 08:23

Merge branch 'main' into chore/modelrouter

188ddd2

feat: remove chat related dependencies and components

5c3ade4

Merge remote-tracking branch 'origin/feat/deprecate-chat' into chore/…

dfcda56

…modelrouter

chore: remove dependency on anthropic types

00d359c

fix(agentOS): set default value of name for Display to None (else it …

339717f

…raises an Error when executing from cache)

chore: fix typecheck errors

1d5929b

fix linting errors

021c90b

fix linting errors

2598a46

feat: add model_store

f1460a9

fix linting errors

d8ce5b2

Merge branch 'main' into chore/modelrouter

d4ea0d8

fix linting errors

a0a75b1

philipph-askui changed the title ~~Remove Modelrouter~~ Remove modelRouter and add mode_store Jan 22, 2026

philipph-askui added 11 commits January 22, 2026 15:16

remove UITARS

93ea9ee

remove ModelComposition

b2c79be

update get model interface

f4c3cb2

remove UITars, modelcomposition, and model definition

6222282

Add locate_settings and locate_model parameters to click() and mouse_…

fca6234

…move() methods

Remove AskUiBaseLocateModel class and create LocateApi (Base class) a…

e3dc24a

…nd AskUiInferenceLocateApi

chore: remove GenericActModel

4ac0df4

move HF models and OpenRouter models to model_store

25a3ad8

chore: fix docs

19c8968

fix: make models serializable for telemetry

6217b11

chore: update docs

8f3a9ea

philipph-askui marked this pull request as ready for review January 26, 2026 06:53

philipph-askui requested a review from mlikasam-askui January 26, 2026 06:54

philipph-askui added 4 commits February 16, 2026 07:57

Merge remote-tracking branch 'origin/main' into chore/modelrouter

fb43999

fix: typecheck errors

f2cea46

fix: add otel + instrumentors as optional dependencies in a new `trac…

0a15539

…ing` group

fix: add tracing dependency group to all

aa1b839

programminx-askui requested changes Feb 16, 2026

View reviewed changes

src/askui/models/openai/messages_api.py Outdated Show resolved Hide resolved

src/askui/model_providers/anthropic_vlm_provider.py Show resolved Hide resolved

src/askui/model_providers/askui_image_qa_provider.py Show resolved Hide resolved

programminx-askui reviewed Feb 16, 2026

View reviewed changes

examples/model_providers.py Outdated Show resolved Hide resolved

philipph-askui added 10 commits February 17, 2026 07:38

address review comments of dominik

a8a2649

remove openai integration

210b266

fix: liniting issue

afa8bf9

fix: typos

c23753c

feat: change provider init to lazy init

5e934a2

fix: add google api key to template

7732b15

add inline documentation for settings parameters

01646ba

fix: removes obsolete anthropic beta flags from settings

c4d2f48

fix: replace anthropic-specific betas parameter with provider-agnos…

4b17319

…tic `provider_options`

fix: update docs to explain new provider_options param

b82b61f

programminx-askui reviewed Feb 23, 2026

View reviewed changes

philipph-askui added 3 commits February 23, 2026 10:32

chore: remove pynput dependency

cdb77d8

chore: refactor docs

389b649

chore: update reporting docs to distinguish between execution reports…

e616ee3

… and test reports

philipph-askui requested a review from programminx-askui February 23, 2026 15:18

programminx-askui approved these changes Feb 24, 2026

View reviewed changes

programminx-askui reviewed Feb 24, 2026

View reviewed changes

docs/08_reporting.md Outdated Show resolved Hide resolved

docs/11_file_support.md Outdated Show resolved Hide resolved

chore: address review remarks of dominik

2eba32f

programminx-askui mentioned this pull request Feb 24, 2026

Deprecate chat api #225

Closed

chore: merge file_support information into docs on extracting_data

1251cbe

philipph-askui merged commit 61e42df into main Feb 24, 2026
1 check passed

philipph-askui deleted the chore/modelrouter branch February 24, 2026 08:43

		from askui.models.shared.tools import ToolCollection


		def _to_openai_messages(messages: list[MessageParam]) -> list[dict[str, Any]]: # noqa: C901

		@@ -0,0 +1,886 @@
		# Using Models and BYOM (Bring Your Own Model)

Conversation

philipph-askui commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philipph-askui commented Feb 16, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

programminx-askui Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

programminx-askui Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

programminx-askui Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

philipph-askui Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

philipph-askui Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

programminx-askui left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

philipph-askui commented Jan 21, 2026 •

edited

Loading