Extend the structured data vignette to include a best practice example for the usual tibble wrangling process with mutate

[kleinlennart]

Hi, I read through the Structured Data vignette, and I couldn't find a best practice example on how to best implement the ellmer interface for a common research analysis task in a standard dplyr rowwise operations workflow with mutate.

I would really appreciate to learn if my example below follows best practices when using ellmer (which has already been super helpful so far. Thank you for making this!).

I am also unsure how to best implement the new extract_data_parallel function since it would require making a list of prompts first and then passing them to chat, right?

📊 Rowwise analysis example with dplyr mutate

library(tidyverse)
library(ellmer)

# test survey data
df <- tibble::tribble(
  ~ID, ~question, ~assessment,
  "1", "Has your project been achieving the proposed objectives in a timely manner?", "1☐ 2☐ 3☒ 4☐ 5☐",
  "2", "Has your project been delivering the planned outputs?", "1☐ 2☐ 3☒ 4☐ 5☐",
  "3", "Have the preidentified risks been impacting the project delivery?", "1☒ 2☐ 3☐ 4☐ 5☐",
  "4", "If relevant, have your risk mitigation strategies been effectively helping manage and address risks?", "1☐ 2☐ 3☐ 4☐ 5☐"
)

# set up chat
chat <- ellmer::chat_openai(
  model = "gpt-4o",
  system_prompt = "You are a helpful assistant.",
  seed = 42,
  api_args = list(
    temperature = 0
  )
)

# define the type: expecting an integer
# required = FALSE to return NULL when missing
type_score <- type_object(
  score = ellmer::type_integer("The selected score only (values between 1 and 5). Otherwise, return 'NA'.", required = FALSE)
)

# define prompt template
user_prompt_template <- "
Here is a survey answer: '{{assessment_text}}'

Identify and extract the number that has a checked box (☒).
Return only the selected number as an integer between 1 and 5.
If none are selected or any information is missing, respond with 'NA'.
"

# define extraction function
extract_score <- function(assessment_text) {
  # every call gets its own isolated clone of chat object
  chat <- chat$clone(deep = FALSE)

  full_prompt <- ellmer::interpolate(user_prompt_template, assessment_text = assessment_text)
  result <- chat$extract_data(
    full_prompt,
    type = type_score,
    convert = TRUE,
    echo = FALSE
  )$score

  # replace NULL with NA for better readability
  result <- ifelse(is.null(result), NA, result)

  return(result)
}

# apply to all rows
df_out <- df |>
  mutate(
    score = map_int(assessment, extract_score, .progress = TRUE)
  )

df_out

Maybe a similar example could be added to the vignette since I think this is how many people would use it – happy to support if need be :)

[hadley]

Thanks for the suggestion! There's no documentation yet because the tools aren't quite right yet. But I really appreciate your motivating example because it forced me to work through your code and think about how I think it should work. Here's what I came up with:

library(ellmer)

df <- tibble::tribble(
  ~ID, ~question, ~assessment,
  "1", "Has your project been achieving the proposed objectives in a timely manner?", "1☐ 2☐ 3☒ 4☐ 5☐",
  "2", "Has your project been delivering the planned outputs?", "1☐ 2☐ 3☒ 4☐ 5☐",
  "3", "Have the preidentified risks been impacting the project delivery?", "1☒ 2☐ 3☐ 4☐ 5☐",
  "4", "If relevant, have your risk mitigation strategies been effectively helping manage and address risks?", "1☐ 2☐ 3☐ 4☐ 5☐"
)
user_prompt_template <- "
Here is a survey answer: '{{assessment_text}}'

Identify and extract the number that has a checked box (☒).
Return only the selected number as an integer between 1 and 5.
If none are selected or any information is missing, respond with 'NA'.
"
# TODO: remove check from interpolate to make this work
df$prompt <- interpolate(user_prompt_template, assessment_text = df$assessment)

chat <- chat_openai(
  model = "gpt-4o",
  system_prompt = "You are a helpful assistant.",
  params = params(temperature = 0, seed = 42)
)
type_score <- type_object(
  score = type_integer(
    "The selected score only (values between 1 and 5). Otherwise, return 'NA'.",
    required = FALSE
  )
)

# TODO: make interpolate return a list so that you can feed immediately
# to $extract_data_parallel()
result <- chat$extract_data_parallel(
  as.list(df$prompt),
  type = type_score 
)

# TODO: convert should automatically convert NULL to NA for simple types
result <- lapply(result, \(x) lapply(x, \(y) ifelse(is.null(y), NA, y)))

# TODO: extract_data_parallel() should do this for you if convert = TRUE
dplyr::bind_rows(lapply(result, as.data.frame))

Once all the todos are done, then you could compute the prompt and add the results in a single step:

df |> mutate(
  prompt = interpolate(user_prompt_template, assessment_text = assessment),
  chat$extract_data_parallel(df$prompt, type = type_score)
)

(I don't think most people would do that because it relies on somewhat esoteric mutate() behaviour, but that feels pretty close to the optimal tidyverse vision here).

[hadley]

Actually, if you match the interpolated values with the variable names, you can make it even simpler:

df <- tibble::tribble(
  ~ID, ~question, ~assessment,
  "1", "Has your project been achieving the proposed objectives in a timely manner?", "1☐ 2☐ 3☒ 4☐ 5☐",
  "2", "Has your project been delivering the planned outputs?", "1☐ 2☐ 3☒ 4☐ 5☐",
  "3", "Have the preidentified risks been impacting the project delivery?", "1☒ 2☐ 3☐ 4☐ 5☐",
  "4", "If relevant, have your risk mitigation strategies been effectively helping manage and address risks?", "1☐ 2☐ 3☐ 4☐ 5☐"
)
user_prompt_template <- "
Here is a survey answer: '{{assessment}}'

Identify and extract the number that has a checked box (☒).
Return only the selected number as an integer between 1 and 5.
If none are selected or any information is missing, respond with 'NA'.
"

df |> dplyr::mutate(prompt = interpolate(user_prompt_template))

[hadley]

(BTW you might want to rethink the "Otherwise, return 'NA'" in your prompt. NA is not an integer in the JSON schema, so I think it would be safer to say "return nothing").

[hadley]

I'll also make it so df$prompt <- interpolate(prompt, !!!df) works if you're not inside of mutate() or friends.