Release v0.1.6

FlexLLama v0.1.6 Release Notes

Overview

This release improves FlexLLama's performance and usability with enhanced timeout configuration, physical batch size support, comprehensive documentation restructuring, and critical installation fixes.

🎉 What's New

Enhanced Timeout Configuration

Flexible Request Management: Added configurable timeouts for both regular and streaming requests with separate timeout controls for optimal performance across different use cases.

Physical Batch Size Support

Improved Performance Control: Added u_batch (physical batch size) parameter for fine-tuned prompt processing performance, giving users more control over memory usage and processing speed.

Documentation Restructuring

Comprehensive Configuration Guide: Moved detailed configuration documentation from README to dedicated CONFIGURATION.md file, making the main README cleaner while providing thorough configuration reference.

Example

Enhanced timeout and batch configuration:

{
    "request_timeout_seconds": 1800,
    "streaming_timeout_seconds": 0,
    "models": [
        {
            "n_batch": 512,
            "u_batch": 128,
            "runner": "runner1"
        }
    ]
}

Ready to experience improved performance control and reliable installation? Get started with FlexLLama v0.1.6!