Skip to content

v0.2.0 - Evaluation Framework and Critical Fixes

Choose a tag to compare

View all tags
@assareh assareh released this 22 Nov 17:18
· 111 commits to main since this release
v0.2.0
9bc59da

🎉 Major Release: Evaluation Framework

This release adds a comprehensive evaluation framework for testing LLM applications and includes critical stability/security fixes from roadmap analysis.

✨ New Features

Evaluation Framework

  • TestCase class for defining tests with flexible validation criteria
  • Evaluator class for running tests against API endpoints
  • HTML report generator with beautiful visual pass/fail results
  • JSON report generator for CI/CD integration
  • Console reporter with color-coded output
  • Custom validators for domain-specific validation logic
  • Performance metrics including response time and success rates
  • Complete documentation and working examples

Backend Improvements

  • Backend health checks on startup (HEALTH_CHECK_ON_STARTUP)
  • Configurable backend timeouts (BACKEND_CONNECT_TIMEOUT, BACKEND_READ_TIMEOUT)
  • Configurable host binding (DEFAULT_HOST)

🔧 Critical Fixes

  • Fixed WebUI subprocess pipe handling (prevents buffer-filling hangs)
  • Improved exception handling - replaced bare except: with specific exception types
  • Added JSON request validation with proper 400 Bad Request responses
  • Better error handling for backend connection failures
  • Proper timeout handling prevents infinite hangs

🔒 Security Improvements

  • Breaking Change: Default host binding changed from 0.0.0.0 to 127.0.0.1
    • This prevents unintended network exposure
    • To bind to all interfaces, set HOST=0.0.0.0 or <PREFIX>_HOST=0.0.0.0
    • Security warning displayed when binding to 0.0.0.0

📚 Documentation

🚀 Quick Start: Evaluation Framework

```python
from llm_api_server.eval import Evaluator, TestCase, HTMLReporter

Define test cases

tests = [
TestCase(
question="What is 2+2?",
description="Basic arithmetic test",
expected_keywords=["4", "four"],
min_response_length=10
)
]

Run evaluation

evaluator = Evaluator(api_url="http://localhost:8000")
results = evaluator.run_tests(tests)

Generate HTML report

HTMLReporter().generate(results, "report.html")
```

📦 Installation

```bash

Using uv (recommended)

uv sync

Or with pip

pip install llm-api-server
```

⚠️ Breaking Changes

  • Default host binding is now 127.0.0.1 (localhost only) instead of 0.0.0.0
  • Users needing network access must explicitly set HOST=0.0.0.0

🙏 Credits

Based on roadmap analysis from the Ivan project.

See CHANGELOG.md for complete details.

Assets 2
Loading