Skip to content

Promptheus v0.3.0 - Telemetry & MCP Improvements

Choose a tag to compare

View all tags
@abhichandra21 abhichandra21 released this 05 Dec 23:35
· 18 commits to main since this release
v0.3.0
0c76b9d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

What's New

This release introduces a comprehensive telemetry system for tracking usage and performance metrics, along with MCP server improvements and critical bug fixes.

✨ New Features

Telemetry System

  • Anonymous usage and performance metrics tracking
  • Token usage tracking across all providers
  • Performance latency metrics for API calls
  • Task type classification metrics (analysis vs generation)
  • Local JSONL storage with configurable sampling
  • Summary reporting via promptheus telemetry summary
  • Privacy-preserving design (no prompts or API keys stored)
  • Telemetry data visualization and analytics

MCP Server Enhancements

  • Extended question handling in refine_prompt tool
  • Improved clarification workflow with better answer mapping
  • Enhanced error handling and response formatting

Shell Completions

  • Added mcp and telemetry subcommands to bash/zsh completions

Web API Token Tracking

  • Prompt refinement endpoint now returns token usage statistics

Test Coverage

  • Comprehensive telemetry test suite with 280+ passing tests
  • End-to-end telemetry tests
  • Extended unit tests
  • Summary generation tests

🔧 Changes

  • Provider Token Tracking: All providers now report input/output/total token usage
    • Added last_input_tokens, last_output_tokens, last_total_tokens attributes
    • Best-effort token tracking with graceful fallbacks
  • History Management: Added telemetry directory configuration support
    • PROMPTHEUS_HISTORY_DIR environment variable for custom location
  • Documentation: Updated with telemetry architecture and usage guides

🐛 Fixes

  • Token counting for providers with inconsistent response formats
  • Provider error handling for missing token usage metadata
  • Critical bug: Fixed indentation issue in telemetry summary aggregation (src/promptheus/telemetry_summary.py:228)
    • Interface and provider metrics were not being collected due to incorrect indentation
    • Updated test assertions to match actual formatted output

🔐 Environment Variables

  • PROMPTHEUS_TELEMETRY_ENABLED: Enable/disable telemetry (default: 1)
  • PROMPTHEUS_TELEMETRY_FILE: Custom telemetry file path (optional)
  • PROMPTHEUS_TELEMETRY_SAMPLE_RATE: Sampling rate 0.0-1.0 (default: 1.0)
  • PROMPTHEUS_HISTORY_DIR: Custom history directory location (optional)

📚 Technical Details

  • Telemetry events stored in platform-specific directories
  • JSONL format for efficient append-only logging
  • Atomic file writes with file locking
  • Comprehensive event schema with timestamps and metadata
  • Provider-agnostic token tracking implementation

Full Changelog: v0.2.4...v0.3.0

Installation

pip install --upgrade promptheus

Documentation

See our documentation for usage examples and guides.

Assets 2
Loading