Embed Ferrum in a Rust application: a complete text-generation example (v0.8.8)

[sizzlecar]

Maintainer here. For a Rust application that wants to own model lifetime and consume generated text directly, Ferrum exposes the same engine abstraction used by its CLI and server. The application calls the engine directly.

This example covers in-process text generation; vector embeddings and reranking are separate model tasks.

The public builder selects a real tokenizer and executor when a model path is supplied. This example selects the CPU Llama executor. Use a locally provisioned, unquantized LlamaForCausalLM SafeTensors checkpoint with its config.json, tokenizer files and referenced weight shards. Allow enough RAM for the selected model.

Start with the released source:

git clone --depth 1 --branch v0.8.8 https://github.com/sizzlecar/ferrum-infer-rs source
mkdir -p example/src

Put this in example/Cargo.toml:

[package]
name = "ferrum-inprocess-example"
version = "0.1.0"
edition = "2021"

[dependencies]
ferrum-engine = { path = "../source/crates/ferrum-engine" }
ferrum-types = { path = "../source/crates/ferrum-types" }
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
futures = "0.3"
anyhow = "1"

And example/src/main.rs:

use anyhow::{Context, Result};
use ferrum_engine::{create_engine, EngineConfig, InferenceRequest};
use ferrum_types::{Device, ModelId, SamplingParams};
use futures::StreamExt;
use std::io::{self, Write};

#[tokio::main(worker_threads = 2)]
async fn main() -> Result<()> {
    let mut args = std::env::args().skip(1);
    let model_dir = args.next().context("Pass a local model directory")?;
    let prompt_file = args.next().context("Pass a UTF-8 prompt file")?;
    let prompt = std::fs::read_to_string(prompt_file)?;
    let model_id = ModelId::new("local-llama");
    let mut config = EngineConfig::default();
    config.model.model_id = model_id.clone();
    config.runtime.model_path = Some(model_dir);
    config.backend.device = Device::CPU;
    config.runtime.max_model_len = Some(2048);
    config.scheduler.max_running_requests = 1;
    let engine = create_engine(config).await?;
    let request = || {
        let mut sampling = SamplingParams::default();
        sampling.max_tokens = 64;
        sampling.temperature = 0.0;
        InferenceRequest::new(prompt.clone(), model_id.clone())
            .with_sampling_params(sampling)
    };
    let output: Result<()> = async {
        println!("{}", engine.infer(request()).await?.text);
        let mut stream = engine.infer_stream(request().with_stream(true)).await?;
        while let Some(chunk) = stream.next().await {
            print!("{}", chunk?.text);
            io::stdout().flush()?;
        }
        println!();
        Ok(())
    }.await;
    let stopped = engine.shutdown().await;
    output?;
    stopped?;
    Ok(())
}

Run it with your model directory and a UTF-8 prompt file:

cargo run --manifest-path example/Cargo.toml -- /absolute/llama-directory /absolute/prompt.txt

The program performs two generations: one collected response, then one stream. Supply a raw completion prompt, or apply your instruction model's chat template before calling the engine. The generation interface accepts InferenceRequest, not OpenAI messages.

Your application owns the Tokio runtime, local model files, prompt construction, request limits and shutdown ordering. Here it admits one active request, sets a 2,048-token context limit and requests at most 64 new tokens. Ferrum manages scheduling and execution resources within that configuration. Both successful and failed generation paths await shutdown.

Choose ferrum serve when multiple clients need a separate process and OpenAI-compatible HTTP endpoints; choose the library when direct Rust calls and host-managed lifecycle fit your application.

Verification: this complete example passed CPU cargo check against the immutable v0.8.8 source using cached dependencies. No model was loaded or inference measured for this tutorial. Written with AI assistance and reviewed against release source.