Skip to content

Releases: floneum/kalosm

kalosm-0.4

09 Feb 23:49
@ealmloff ealmloff
kalosm-0.4.0
dbcc4bb

Choose a tag to compare

View all tags
kalosm-0.4 Latest
Latest

Introducing Kalosm 0.4: A Local-First AI Meta-Framework for Rust

Kalosm is a simple, local first interface for pre-trained language, audio, and image models written in Rust. It is built on top of the candle library for ML inference. Kalosm makes it easy to run quantized models on your local machine accelerated by cuda or metal. Whether you’re building a chatbot, a transcription tool, or an image processing application, Kalosm 0.4 brings major improvements to help you get started quickly and efficiently.

In this release, we’re excited to announce:

  • A Simplified API for language models, tasks, and chat sessions
  • Extended support for remote models via OpenAI and Anthropic adapters with structured generation
  • Single-file model loading that consolidates model weights, tokenizer, and chat template metadata into one GGUF file
  • Improved whisper transcription with word-level timestamps using dynamic time warping

Simplified LLM API

One of the biggest improvements in Kalosm 0.4 is the streamlined API for interacting with local large language models. Developers can now configure chat sessions and tasks with a series of builder methods until the first message is added. This flexibility simplifies the code and makes your applications easier to maintain.

For example, here’s how you can set up a chat session with a system prompt:

use kalosm::language::*;

// Create a new chat-capable Llama model
let model = Llama::new_chat().await?;

// Configure a chat session with a system prompt
let mut chat = model
    .chat()
    .with_system_prompt("The assistant will act like a pirate");

loop {
    // Get user input and stream the model’s response to stdout
    chat(&prompt_input("\n> ")?)
        .to_std_out()
        .await?;
}

Similarly, the task API now supports automatic constraint inference. By deriving parsing and schema traits on your data types, you can easily generate concrete types from your language model:

use kalosm::language::*;

/// A fictional account holder
#[derive(Schema, Parse, Clone, Debug)]
struct Account {
    /// A brief summary of the account holder
    #[parse(pattern = r"[a-zA-Z,.?!\d ]{1,80}")]
    summary: String,
    /// The account holder’s name (full name or pseudonym)
    #[parse(pattern = "[a-zA-Z ]{1,20}")]
    name: String,
    /// The account holder’s age
    #[parse(range = 1..=100)]
    age: u8,
}

// Create a small reasoning-focused model
let llm = Llama::phi_3().await?;

// Create a task for generating accounts, with automatic type-based constraint inference
let create_account = llm
    .task("You generate accounts based on a description of the account holder")
    .typed();

// Generate an account from a natural language prompt
let account: Account = create_account(
    "Candice is the CEO of a fortune 500 company. She is 30 years old."
).await?;

println!("Generated Account: {account:?}");

OpenAI and Anthropic Support with Structured Generation

Kalosm 0.4 extends its support for remote models by integrating with the OpenAI and Anthropic chat APIs. Now you can use Kalosm’s structured generation even when working with remote chat models. Unfortunately Anthropic currently doesn't support structured generation, so support for tasks with Anthropic models is currently limited.

To work with OpenAI models, enable the openai feature and set your API key via the OPENAI_API_KEY environment variable. Then create a model adapter:

// Create a chat model adapter for OpenAI-compatible models
let llm = OpenAICompatibleChatModel::builder()
    .with_gpt_4o_mini()
    .build();

// Use the adapter just like any other chat model
let generate_character = llm
    .task("You generate accounts based on a description of the account holder")
    .typed();

let account: Account = generate_character(
    "Candice is the CEO of a fortune 500 company. She is 30 years old."
).await?;
println!("Generated Account: {account:?}");

For Anthropic models, enable the anthropic feature and set your ANTHROPIC_API_KEY:

// Create a chat model adapter for Anthropic-compatible models
let llm = AnthropicCompatibleChatModel::builder()
    .with_claude_3_5_haiku()
    .build();

// Start a chat session with a custom system prompt
let mut chat = llm
    .chat()
    .with_system_prompt("The assistant will act like a pirate");

loop {
    chat(&prompt_input("\n> ")?)
        .to_std_out()
        .await?;
}

These integrations allow you to combine local inference with remote model capabilities and structured generation features, making it easier to build hybrid AI applications where local models may not be enough.

Single-File LLM Loading

Previously, loading a custom model required managing separate files for the tokenizer, chat template, and model weights. With version 0.4, Kalosm supports loading all model metadata from a single GGUF file. This improvement simplifies model distribution and deployment.

To load a custom model, you can now just point to a local or huggingface GGUF file:

let model = Llama::builder()
    // Specify a custom model source using a GGUF file
    .with_source(LlamaSource::new(
        FileSource::HuggingFace {
            model_id: "QuantFactory/SmolLM-1.7B-Instruct-GGUF".to_string(),
            revision: "main".to_string(),
            file: "SmolLM-1.7B-Instruct.Q4_K_M.gguf".to_string(),
        },
    ))
    .build()
    .await?;

let mut chat = model
    .chat()
    .with_system_prompt("The assistant will act like a pirate");

loop {
    chat(&prompt_input("\n> ")?)
        .to_std_out()
        .await?;
}

This unified approach to model loading makes it much easier to switch between different models and update to new models over time.

Timestamped Transcription with Whisper

Kalosm 0.4 also brings improvements to audio transcription. The new Whisper API supports word-level timestamps, which can be critical for applications that require precise alignment between audio and text.

Below is an example that demonstrates how to transcribe audio and print each segment along with its corresponding timestamps:

// Build a new Whisper model using a quantized variant
let model = WhisperBuilder::default()
    .with_source(WhisperSource::QuantizedLargeV3Turbo)
    .build()
    .await?;

// Open and decode an audio file
let file = BufReader::new(File::open("./samples_jfk.wav")?);
let audio = Decoder::new(file)?;

// Transcribe the audio with timestamping enabled
let mut text = model.transcribe(audio).timestamped();

// Print each transcribed segment with its start and end times
while let Some(segment) = text.next().await {
    for chunk in segment.chunks() {
        let timestamp = chunk.timestamp().unwrap();
        println!("{:0.2}..{:0.2}: {}", timestamp.start, timestamp.end, chunk);
    }
}

This level of granularity in transcription is particularly useful for applications like video captioning, meeting transcription, or any context where knowing the exact timing of spoken words matters.

Community and Support

Kalosm is built by a passionate community dedicated to making local AI inference accessible and efficient. For help, discussion, or to share feedback, join the Kalosm Discord community. Whether you’re troubleshooting, looking to contribute, or simply curious about the latest updates, our Discord is the best place to connect with fellow developers and the Kalosm team.

Conclusion

Kalosm 0.4 brings a range of enhancements—from simplified APIs and unified model loading to extended remote support and precise transcription capabilities—that we hope will make your development process smoother and more enjoyable. We're truly excited to see the creative applications and projects you build with these tools.

Happy coding!

Assets 2
Loading

kalosm-0.3.0

21 Aug 22:00
@ealmloff ealmloff
kalosm-0.3.0
af7d08e

Choose a tag to compare

View all tags
kalosm-0.3.0

Kalosm 0.3

Kalosm 0.3 makes it significantly easier to use structured generation, improves transcription, and makes it possible to track model download progress. It also includes performance improvements for text generation and transcription models along with parser improvements

Performance Improvements

The new version of Kalosm includes significant performance improvements for llama, mistral, and phi models. We have also developed sampler aware structured generation which lets us skip parsing most tokens in loose structures. Performance should be between 2-4x as fast depending on your usecase:

Demo Kalosm 0.2 Kalosm 0.3
Text generation
kalosm-0.2.mp4
kalosm-0.3.mp4
Structured Generation
kalosm-0.2-structured.mp4
kalosm-0.3-structured.mp4

Structured Generation Improvements

Structured generation is both easier and faster in 0.3. Many structured generation tasks can use json. If you just need a json parser, kalosm 0.3 lets you derive the parser from your data:

use kalosm::language::*;

/// A fictional character
#[derive(Parse, Schema, Clone, Debug)]
struct Character {
    /// The name of the character
    #[parse(pattern = "[A-Z][a-z]{2,10} [A-Z][a-z]{2,10}")]
    name: String,
    /// The age of the character
    #[parse(range = 1..=100)]
    age: u8,
    /// A description of the character
    #[parse(pattern = "[A-Za-z ]{40,200}")]
    description: String,
}

Then you can build a task that generates the character:

#[tokio::main]
async fn main() {
    // First create a model. Chat models tend to work best with structured generation
    let model = Llama::new_chat().await.unwrap();
    // Then create a task with the parser as constraints
    let task = Task::builder_for::<Character>("You generate realistic JSON placeholders for characters")
        .build();
    // Finally, run the task
    let mut stream = task.run("Create a random character", &model);
    stream.to_std_out().await.unwrap();
    let character = stream.await.unwrap();
    println!("{character:?}");
}

Along with the parser, you can also derive a json schema that matches the parser which is useful for function calling models.

You can read more about how structured generation works in kalosm in our last blog post.

Streaming Voice Transcription

Kalosm 0.3 adds support for transcribing audio streams like microphone in chunks based on voice activity. You can now read the audio stream directly from the microphone and transcribe it as voices are detected:

// Create a new whisper model.
let model = Whisper::new().await.unwrap();

// Stream audio from the microphone
let mic = MicInput::default();
let stream = mic.stream().unwrap();

// Transcribe the audio into text in chunks based on voice activity.
let mut text_stream = stream.transcribe(model);

// Finally, print the text to the console
text_stream.to_std_out().await.unwrap();

Model Progress

Loading models is now async with a callback for loading progress:

let model = Bert::builder()
    // build with loading handler lets you track the progress of the model loading
    .build_with_loading_handler(|loading| match loading {
        ModelLoadingProgress::Downloading {
            source,
            start_time,
            progress,
        } => {
            let elapsed = start_time.elapsed();
            println!("Downloading model from {source}...{progress}% (elapsed {elapsed:?})");
        }
        ModelLoadingProgress::Loading { progress } => {
            println!("Loading model into memory...{progress}%");
        }
    })
    .await
    .unwrap();

Whisper transcriptions and wuerstchen generations are also async with progress info thanks to @newfla:

// Create a new whisper model
let model = WhisperBuilder::default()
    .with_source(WhisperSource::QuantizedDistilLargeV3)
    .build()
    .await.unwrap();

let mic = MicInput::default(); 
let audio = mic.stream().unwrap();

// Transcribe the source audio into text
let mut text = audio.transcribe(model);

// As the model transcribes the audio, print the text to the console
while let Some(chunk) = text.next().await {
    let text = chunk.as_ref();
    println!("{text}");
    println!(
        "estimated time left to decode chunk: {}s",
        chunk.remaining_time().as_secs()
    );
}

Documentation improvements

The inline documentation has been significantly improved in 0.3. Common items now include inline guides to help you get started like the language page and concept explanations like embeddings

New models!

Along with the new release, kalosm supports a few new models:

  • Quantized whisper models are now supported with presets for distilled versions of whisper to run even faster
  • The Phi-3 series of models is supported by kalosm-llama. The Phi series performs above its weight for structured json generation tasks

Full changelog

Read more

Contributors

haoxins, Yevgnen, and 6 other contributors
Loading

0.2.0

12 Sep 00:48
@ealmloff ealmloff
v0.2.0
c314e5d

Choose a tag to compare

View all tags
0.2.0 Pre-release
Pre-release

Floneum 0.2 is here! Here are some major features in the new release:

You can read the full release notes in the 0.2 release post

Loading

Nightly-7-23-2023

24 Jul 03:32
@ealmloff ealmloff
nightly-7-23-2023
cd83ac7
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Nightly-7-23-2023 Pre-release
Pre-release 
Merge pull request #32 from floneum/logging

Add tracing support
Loading

0.1.0

12 Jul 15:21
@ealmloff ealmloff
v0.1.0
8d43e60

Choose a tag to compare

View all tags
0.1.0 Pre-release
Pre-release

The first pre-release of Floneum. This is a pre release, there will be bugs. If you run into any bugs, you can report them on the discord or open a github issue

Loading