Optimize category system prompts using prompt engineering to improve LLM Accuracy

[Xunzhuo]

Is your feature request related to a problem? Please describe.

Currently, the semantic router supports category-specific system prompts, but these prompts are not optimized using advanced prompt engineering techniques. This leads to suboptimal performance because:

• System prompts lack structured prompt engineering patterns that have been proven to improve LLM performance
• MoE (Mixture-of-Experts) models are not receiving optimally crafted prompts to activate the most relevant expert networks
• Current prompts don't leverage techniques like Chain-of-Thought, role-based prompting, or domain-specific instruction patterns
• There's no systematic approach to measure and improve prompt effectiveness across different categories
• Prompts don't include explicit instructions for expert network activation in MoE architectures

Describe the solution you'd like

Implement advanced prompt engineering optimization for category-specific system prompts:

1. Advanced Prompt Engineering Techniques:

   • Chain-of-Thought (CoT): Add step-by-step reasoning instructions for complex domains
   • Role-based Prompting: Enhanced professional persona definitions with specific expertise areas
   • Few-shot Examples: Include domain-specific examples in prompts where beneficial
   • Structured Output: Define clear output formats and quality standards
   • Meta-prompting: Instructions that help the model understand its own capabilities

2. MoE-Specific Optimization:

   • Expert Activation Keywords: Include domain-specific terminology that triggers relevant expert networks
   • Capability Mapping: Explicit instructions about what the model should excel at in each domain
   • Context Priming: Structured context that helps MoE routing decisions
   • Performance Indicators: Clear success criteria for each category

3. Category-Specific Enhancements:

   • Mathematics: Include step-by-step reasoning, formula explanation, and verification steps
   • Computer Science: Code quality standards, best practices, and debugging approaches
   • Business: Strategic thinking frameworks, stakeholder analysis, and ROI considerations
   • Science: Scientific method, evidence-based reasoning, and peer-review standards
   • Creative Writing: Style guides, narrative techniques, and audience considerations

4. Dynamic Prompt Optimization:

   • A/B testing framework for different prompt versions
   • Performance metrics collection (accuracy, user satisfaction, expert activation rates)
   • Automated prompt refinement based on feedback
   • Version control for prompt iterations

5. Implementation Features:

   • Prompt template system with configurable components
   • Validation tools to ensure prompt quality and consistency
   • Documentation and examples for each optimized prompt
   • Integration with existing reasoning mode and classification systems

Example Optimized Prompt Structure:

categories:
  - name: "math"
    system_prompt: |
      You are a mathematics expert with deep knowledge in algebra, calculus, statistics, and applied mathematics.
      
      EXPERTISE ACTIVATION: Focus on mathematical reasoning, formula derivation, and step-by-step problem solving.
      
      APPROACH:
      1. Understand the mathematical problem completely
      2. Identify relevant mathematical concepts and formulas
      3. Show your work step-by-step with clear explanations
      4. Verify your solution using alternative methods when possible
      5. Explain the mathematical reasoning behind each step
      
      OUTPUT STANDARDS:
      - Always show intermediate steps
      - Use proper mathematical notation
      - Explain why each step is valid
      - Provide context for the solution's practical meaning
      
      QUALITY INDICATORS: Accuracy, clarity of explanation, proper notation, step-by-step reasoning.

Additional context

Expected Performance Improvements:

• 20-40% improvement in domain-specific accuracy through better expert activation
• Enhanced consistency in response quality across categories
• Better alignment with user expectations for domain expertise
• Improved reasoning quality in complex problem-solving scenarios

Technical Implementation:

• Extend existing system_prompt configuration with template support
• Add prompt validation and testing utilities
• Implement metrics collection for prompt effectiveness
• Create prompt optimization workflows and documentation

Research Foundation:

• Based on recent advances in prompt engineering research
• Incorporates best practices from MoE model optimization studies
• Leverages domain-specific instruction tuning techniques
• Applies cognitive load theory to prompt design

Integration Points:

• Works seamlessly with existing category classification
• Enhances current reasoning mode functionality
• Maintains compatibility with all supported model types
• Supports gradual rollout and A/B testing

This optimization will significantly improve the semantic router's ability to extract maximum performance from MoE architecture models by providing expertly crafted, domain-specific prompts that activate the most relevant expert networks.

[Xunzhuo]

/area core research user-experience
/milestone v0.1
/priority P1

[Xunzhuo]

after #256