A hard dollar limit per background job with RubyLLM (openai_api_base + with_headers)

[domondi1]

A Rails job that summarizes a ticket, drafts a reply and runs a tool or two can make several model calls, and a retry or a long thread can quietly multiply that. Token counts get recorded afterwards, but nothing stops one job from overspending. Here's one way to give each job a hard dollar budget using two things RubyLLM already has: openai_api_base and with_headers.

Run a small OpenAI-compatible gateway next to the app (it uses your provider key):

pip install inferrail
export OPENAI_API_KEY=sk-...
inferrail serve --quickstart --app-mode   # http://127.0.0.1:8000/v1

# config/initializers/ruby_llm.rb
RubyLLM.configure do |config|
  config.openai_api_key  = "unused"                    # the gateway holds the real key
  config.openai_api_base = "http://127.0.0.1:8000/v1"
end

class DraftReplyJob < ApplicationJob
  def perform(ticket_id)
    ticket = Ticket.find(ticket_id)
    chat = RubyLLM.chat(model: "gpt-4o-mini")
                  .with_params(max_tokens: 800)            # the reservation is based on it
                  .with_headers("X-Inferrail-Attribute-Work-Id" => "draft-reply-#{job_id}",
                                "X-Inferrail-Budget-Usd" => "0.20")
    chat.ask("Summarize this ticket and draft a reply:\n\n#{ticket.body}")
  end
end

Every call in the job sends the same id, so they share one budget. Each call reserves its worst-case cost before it's sent, and a call that doesn't fit is refused with HTTP 402 before it reaches OpenAI, which surfaces as an error in the job. inferrail work draft-reply-<job id> shows what the job cost. Using job_id means an ActiveJob retry of the same job draws from the same budget instead of getting a fresh one.

Caveats: chat completions only (embeddings should stay on their normal endpoint), the budget store is a SQLite file on the gateway host, so run one gateway for all workers, and the model needs a price in Inferrail (gpt-4o-mini, gpt-4.1-mini, gpt-4.1 are built in). I've tested the gateway with the Python and TypeScript OpenAI clients, not RubyLLM end to end, so reports are welcome.

Setup details: guide. I maintain Inferrail (open source, Apache-2.0), so take the suggestion with that in mind.