Unified overrides for custom router and orchestrator models#820
Unified overrides for custom router and orchestrator models#820
Conversation
adilhafeez
changed the title
adilhafeez
changed the title
salmanap
left a comment
salmanap
left a comment
There was a problem hiding this comment.
Presume you have tested these changes. Minor comments.
cli/planoai/config_generator.py
Outdated
| updated_model_providers.append( | ||
| { | ||
| "name": "plano-orchestrator", | ||
| "provider_interface": "arch", |
There was a problem hiding this comment.
make this plano/plano-orchestrator
crates/common/src/configuration.rs
Outdated
| hermesllm::ProviderId::try_from(self.to_string().as_str()) | ||
| // Plano provider uses the same interface as Arch | ||
| let provider_str = match self { | ||
| LlmProviderType::Plano => "arch", |
There was a problem hiding this comment.
Just make that plano and create a cluster for it in the envoy config file?
docs/source/guides/orchestration.rst
Outdated
| orchestrator_model: plano/katanemo/Plano-Orchestrator-4B | ||
|
|
||
| model_providers: | ||
| - model: plano/katanemo/Plano-Orchestrator-4B |
There was a problem hiding this comment.
just plano/Plano-Orchestrator is fine.
There was a problem hiding this comment.
thats actually name of the model from hugging face
| "name": "plano-orchestrator", | ||
| "provider_interface": "arch", | ||
| "model": "Plano-Orchestrator", | ||
| "model": orchestrator_model_id, |
There was a problem hiding this comment.
we should standardize on the model field. I am not sure how the name is different from the model field.
config/plano_config_schema.yaml
Outdated
| upstream_tls_ca_path: | ||
| type: string | ||
| description: "Path to the trusted CA bundle for upstream TLS verification. Default is '/etc/ssl/certs/ca-certificates.crt'." | ||
| router_model: |
There was a problem hiding this comment.
how about llm_routing_model and agent_orchestration_model
There was a problem hiding this comment.
its more descriptive and better - updating
2 participants
routingandorchestrationconfig sections withoverrides.router_modelandoverrides.orchestrator_modelplanoprovider interface (alias forarch) for self-hosted models