A more general function to dispatch to model families

[moodymudskipper]

That would make it easier to program around, something like :

chat_dispatch <- function(model_family, ...) {
  chat <- switch(
    model_family,
    openai = chat_openai(...),
    claude = chat_claude(...),
   # etc
  )
}

[cpsievert]

FWIW, chatlas recently added ChatAuto() posit-dev/chatlas#38

[hadley]

Can you expand on more why this would help? If this is something you need, you could already write your own wrapper like

chat_dispatch <- function(model_family, ...) {
  match.fun(paste0("chat_", model))(...)
}

[moodymudskipper]

Sure, we're working on a an application that does some plots and transformation based on llms, this can work with many llms supported by ellmer so we need a function to generalise the existing functions. Users might pay for one API in particular and want to use this one and we want to make it easy.

I wrote a function similar to what I proposed above and it works fine for me, but I felt this type of use case would be common enough to deserve an out of the box function in ellmer.

To have a nice general function there's a bit more work to do, to fail gracefully, to support or exclude explicitly functions with more inconsistent arguments, maybe guessing model_family from model by default, forwarding more args for a nicer autocomplete... So if it's done well here, users don't need to redo it.

For my case I have a satisfying solution so I'm not blocked :

get_model <- function() {
  getOption("blockr.ai.model", "gpt-4o")
}

get_model_family <- function() {
  # commented out inconsistent apis
  supported <- c(
    # "azure", 
    "bedrock",
    "claude",
    #"cortex",
    #"databricks",
    "gemini",
    "github",
    "groq",
    "ollama",
    "openai",
    "perplexity"
    # "vllm"
  )
  model_family <- getOption("blockr.ai.model_family", "openai")
  rlang::arg_match(model_family, supported)
}

chat_dispatch <- function(system_prompt, ..., turns = NULL, model = get_model(), model_family = get_model_family()) {
  switch(
    model_family,
    bedrock = ellmer::chat_bedrock(system_prompt, turns, model = model, ...),
    claude = ellmer::chat_claude(system_prompt, turns, model = model, ...),
    gemini = ellmer::chat_gemini(system_prompt, turns, model = model, ...),
    github = ellmer::chat_github(system_prompt, turns, model = model, ...),
    groq = ellmer::chat_groq(system_prompt, turns, model = model, ...),
    ollama = ellmer::chat_ollama(system_prompt, turns, model = model, ...),
    openai = ellmer::chat_openai(system_prompt, turns, model = model, ...),
    perplexity = ellmer::chat_perplexity(system_prompt, turns, model = model, ...),
  )
}

[hadley]

We could approach this like inspect, https://inspect.aisi.org.uk/models.html, and adding something like chat_???("anthropic/claude-3-5-sonnet-latest")

And/or some helper that either takes a string or a chat object (or a chat function?), that e.g. btw_client could use.

[hadley]

Proof of concept for discussion:

#' @export
#' @inheritParams chat_openai
chat <- function(name, ..., system_prompt = NULL, params = NULL) {
  check_string(name, allow_empty = FALSE)
  pieces <- strsplit(name, "/", fixed = TRUE)[[1]]
  if (length(pieces) != 2) {
    cli::cli_abort("{.arg name} must be in form {.str provider/model}.")
  }
  provider <- pieces[[1]]
  model <- pieces[[2]]

  provider_name <- paste0("chat_", pieces[[1]])
  chat_fun <- env_get(asNamespace("ellmer"), provider_name, default = NULL)
  if (is.null(chat_fun)) {
    cli::cli_abort("Can't find provider {.code ellmer::{provider_name}()}.")
  }

  chat_fun(model = model, ..., system_prompt = system_prompt, params = params)
}

I'm not sure what the right name is. Maybe chat_()? It feels a bit different from Chatlas's ChatAuto.

[gadenbuie]

It feels a bit different from Chatlas's ChatAuto.

This is quite aligned with ChatAuto, which I see as a sort of experimental first iteration. I think we can easily align chatlas with ellmer's version. Having a single name input is great; and I like chat() more than chat_() or chat_auto(). After using ChatAuto a few times, I also like not prescribing an envvar or global option; I've found that to create more issues than either requiring the name string be provided directly or letting developers decide if an envvar or global option is appropriate.

[moodymudskipper]

I like the "/" separator a lot.

I wouldn't mind name to default to an option so I could change my favourite general purpose provider or model in my .Rprofile only and all my general llm code would just work (since there's no output stability anyway and that we do have output type stability i think such option is ok), and I wouldn't mind pushing the name arg to the back.

The way it's designed, system_prompt must be provided by name, if we're tempted to provide it as the first unnamed arg of ... as would feel natural to many that are used to the openai like variants, it will be ignored silently, so maybe system_prompt could move before the dots, or we should check that all dots are named.

[gadenbuie]

Oh, regarding name, I think it might be okay to relax the length(pieces) != 2 to length(pieces) > 2 and allow something like chat('anthropic'). It's not precise, but it's convenient if you are testing and want to quickly switch between providers using ellmer's default models—usually finding the model name takes a bit of effort.

[simonpcouch]

Agreed that I do like chat() as the function name! The Inspect-like model string seems very reasonable (we don't know of any model names that have slashes in them, right?).

I do think that having a ellmer.name or similar option would be helpful. Otherwise, I don't imagine making much use of this function (without writing my own wrapper), as it feels a bit like the chat_*() functions with less autocomplete/IDE help.

[gadenbuie]

I wouldn't mind name to default to an option so I could change my favourite general purpose provider or model in my .Rprofile only and all my general llm code would just work

This is extremely convenient if you're using chat_*() functions mostly interactively. I've had a variant of this in btw as btw_client() where provider and model are provided in a project-specific btw.md file. It's even more helpful when you're wanting to jump straight into a chat in a dedicated Shiny app, e.g. btw_app().

I think that's convenient enough that I'll walk back my previous comment that chat() shouldn't use a global option or envvar. I think a global option is reasonable. Importantly, the mechanism should be simple; if you're programming around ellmer chats (in a Shiny app widget or an R package with LLM features), you'll likely want to insert your own option in front of ellmer's. I think a single option is good, but if it were more complicated (e.g. looking at an envvar or an option), it gets harder to use or replicate as an extension author).

[hadley]

If we provide an option (easy to set inside of R), I think it's worth providing an env var too (easy to set outside of R). So then the hierarchy is argument >> option >> env var.

Agree that might as well relax the requirement to specify a model if you just want to use the default.

I think autocomplete should be just as good here, since this function should provide all arguments that are common to providers (looks like I forgot about echo).

[gadenbuie]

If we provide an option (easy to set inside of R), I think it's worth providing an env var too (easy to set outside of R). So then the hierarchy is argument >> option >> env var.

The scenario I have in mind are packages like chores and gander. On the one hand, it's helpful that each of these could inherit the user's preference for which ellmer model to use if they have, e.g., ellmer.provider_model or ELLMER_PROVIDER_MODEL set.

On the other hand, it's reasonable that chores and gander would want to have their own options as well, and even that someone might want to pick a different default for chores than gander. The package author then has to re-implement ellmer's option/envvar lookup with their own options in front. This isn't terrible and we could easily provide a snippet in the ellmer docs that authors could use.

One more complication is that the package authors might want to choose a default value, e.g. to use Anthropic's Claude, if none of the options are set. Again not terrible, but requires a bit of code and jumping through hoops, especially if the authors want to use (their options) >> (ellmers options) >> their own default.

This isn't all terribly difficult, but I could see an argument for a simpler interface that just consults an option and leaves some of these choices up to the ellmer extension author.

[hadley]

For now, I think I'll leave it option/env var less. Then it's easy for (e.g.) gander or chores to provide their own option/default.