Stencila Cloud is now returning overall quality, cost, and speed scores for each model that it proxies to at https://api.stencila.cloud/v1/models. The scores are all null (except for a few were we have some ad hoc scores so that our routing/fallback algo there could work).
Currently, scoring.csv only provides scores for 9 models.
xh https://api.stencila.cloud/v1/models
[
{
"provider": "stencila",
"identifier": "router",
"name": "Router",
"version": "0.1.0",
"qualityScore": null,
"costScore": null,
"speedScore": null
},
{
"provider": "anthropic",
"identifier": "claude-3-5-sonnet-20240620",
"name": "Claude",
"version": "3-5-sonnet-20240620",
"qualityScore": null,
"costScore": null,
"speedScore": null
},
{
"provider": "anthropic",
"identifier": "claude-3-opus-20240229",
"name": "Claude",
"version": "3-opus-20240229",
"qualityScore": null,
"costScore": null,
"speedScore": null
},
...
Stencila Cloud is now returning overall quality, cost, and speed scores for each model that it proxies to at https://api.stencila.cloud/v1/models. The scores are all
null(except for a few were we have some ad hoc scores so that our routing/fallback algo there could work).Currently,
scoring.csvonly provides scores for 9 models.[ { "provider": "stencila", "identifier": "router", "name": "Router", "version": "0.1.0", "qualityScore": null, "costScore": null, "speedScore": null }, { "provider": "anthropic", "identifier": "claude-3-5-sonnet-20240620", "name": "Claude", "version": "3-5-sonnet-20240620", "qualityScore": null, "costScore": null, "speedScore": null }, { "provider": "anthropic", "identifier": "claude-3-opus-20240229", "name": "Claude", "version": "3-opus-20240229", "qualityScore": null, "costScore": null, "speedScore": null }, ...