v0.235.003

(v0.235.003)

New Features

• Approval Workflow System

  • Comprehensive approval process for sensitive Control Center operations requiring review and approval before execution.
  • Protected Operations: Take ownership, transfer ownership, delete documents, and delete group operations now require approval.
  • Approval Features: Documented justification, review process by group owners/admins, complete audit trail, auto-expiration after 3 days, notification integration.
  • Database: New approvals container with TTL-based expiration.
  • (Ref: route_backend_control_center.py, route_frontend_control_center.py, control_center.html, approval workflow UI)

• Agent Streaming Support

  • Real-time streaming support for Semantic Kernel agents with incremental response display.
  • Features: Agent responses stream word-by-word, plugin citation capture during streaming, async generator pattern for efficient streaming, proper async/await handling.
  • User Experience: Matches existing chat streaming experience, see agent thinking in real-time, immediate visual feedback.
  • (Ref: route_backend_chats.py, agent streaming implementation, Semantic Kernel integration)

• Control Center

  • Comprehensive administrative interface for data and workspace management.
  • User Management: View all users with search/filtering, grant/deny access with time-based restrictions, manage file upload permissions, monitor user engagement and storage.
  • Activity Trends: Visual analytics with Chart.js showing daily activity metrics (chats, uploads, logins, document actions) across 7/30/90-day periods.
  • Group Management: Approval workflow integration, group status management, member activity monitoring.
  • Dashboard: Real-time statistics, key alerts, activity insights.
  • (Ref: route_frontend_control_center.py, route_backend_control_center.py, control_center.html)

• Message Threading System

  • Linked-list threading system establishing proper message relationships throughout conversations.
  • Thread Fields: thread_id (unique identifier), previous_thread_id (links to previous message), active_thread (thread active status), thread_attempt (retry tracking).
  • Benefits: Proper message ordering, file upload tracking, image generation association, legacy message support.
  • Message Flow: Links user messages to AI responses, system augmentations, file uploads, and image generations.
  • (Ref: route_backend_chats.py, message schema updates, thread chain implementation)

• User Profile Dashboard

  • Complete redesign into modern dashboard with personalized analytics and visualizations.
  • Metrics Display: Login statistics, chat activity, document usage, storage consumption, token tracking.
  • Visualizations: Chart.js-powered activity trends, 30-day time-series data, interactive charts.
  • Features: Cached metrics for performance, real-time data aggregation, responsive design.
  • (Ref: route_frontend_profile.py, profile.html, Chart.js integration)

• Speech-to-Text Chat Input

  • Voice recording up to 90 seconds directly in chat interface with Azure Speech Service transcription.
  • Features: Visual waveform display during recording, 90-second countdown timer, review before send, cancel anytime, responsive design.
  • Browser Support: Chrome 49+, Edge 79+, Firefox 25+, Safari 14.1+.
  • Integration: Uses existing Azure Speech Service configuration, MediaRecorder API, Web Audio API.
  • (Ref: route_backend_settings.py, chat-speech-to-text.js, Speech Service integration)

• Text-to-Speech AI Responses

  • AI messages read aloud using Azure Speech Service with high-quality DragonHD voices.
  • Features: 27 DragonHD Latest Neural Voices across languages, voice preview in profile, customizable speech speed (0.5x-2.0x), play/pause/stop controls.
  • Playback: Inline "Listen" button per message, visual feedback during playback, auto-play mode option, prevents multiple simultaneous playbacks.
  • Integration: Automatically disables streaming when auto-play enabled, per-user profile settings.
  • (Ref: route_backend_tts.py, chat-tts.js, Azure Speech Service)

• Message Edit Functionality

  • Comprehensive message editing system allowing users to modify their sent messages and regenerate AI responses.
  • Features: Modal interface for editing message text, preserves conversation context and settings, automatically regenerates AI response with edited content, maintains message metadata and threading.
  • User Experience: Edit button on user messages, inline editing workflow, real-time validation, preserves agent/model selection.
  • Integration: Works with /api/message/<id>/edit endpoint, updates conversation history, maintains thread relationships.
  • (Ref: chat-edit.js, route_backend_chats.py, message edit modal)

• Message Delete Capability

  • One-click message deletion with proper conversation thread cleanup and metadata updates.
  • Features: Delete button on user messages, ownership validation (author-only), updates message threading chains, removes associated metadata.
  • Safety: Confirmation prompt, author verification, cascading thread updates, preserves conversation integrity.
  • Integration: API endpoint for message deletion, updates conversation message count, maintains thread consistency.
  • (Ref: chat-messages.js, message deletion handlers, thread management)

• Message Retry/Regenerate System

  • Powerful message regeneration system allowing users to retry AI responses with different models, agents, or settings.
  • Features: Modal interface with agent/model selection, adjustable reasoning effort for o-series models, preserves original user message, generates new AI response with selected configuration.
  • Configuration Options: Switch between agents, change model deployments, adjust reasoning effort (low/medium/high), modify generation parameters.
  • User Experience: Retry button on AI messages, dropdown selection for agents/models, real-time configuration updates.
  • (Ref: chat-retry.js, retry modal interface, agent/model switching)

• Message Masking System

  • Privacy-focused message masking capability for hiding sensitive information with visual overlays and PII protection.
  • Features: Visual mask overlay on message content, masked_ranges metadata tracking character positions, mask/unmask toggle buttons, preserves original content while displaying masked state.
  • Privacy Protection: Masks sensitive data in UI, tracks masked regions in database, supports partial message masking, reversible masking for authorized users.
  • Integration: /api/message/<id>/mask endpoint, masked and masked_ranges metadata fields, visual indicators (bi-front/bi-back icons).
  • User Experience: Mask button on messages, visual overlay showing masked content, toggle between masked and unmasked states.
  • (Ref: chat-messages.js, route_backend_chats.py, masked content handling, applyMaskedState() function)

• Conversation Pinning

  • Pin important conversations to the top of the conversation list for quick access and improved organization.
  • Features: Single conversation pinning, bulk pin operations for multiple conversations, persistent pin state in database, visual pin indicators in sidebar.
  • Operations: Pin/unpin toggle, bulk selection interface, priority sorting (pinned conversations appear first), is_pinned metadata field.
  • API Endpoints: /api/conversations/<id>/pin (POST), /api/conversations/bulk-pin (POST).
  • User Experience: Pin icon in conversation list, bulk selection checkboxes, immediate visual feedback.
  • (Ref: chat-conversations.js, toggleConversationPin(), bulkPinConversations(), conversation state management)

• Conversation Hiding

  • Hide conversations from the main list to declutter the sidebar without permanent deletion.
  • Features: Single conversation hiding, bulk hide operations, toggle visibility without data loss, is_hidden metadata field for state persistence.
  • Benefits: Declutter conversation list, temporary archiving without deletion, reversible operation, maintains conversation data.
  • API Endpoints: /api/conversations/<id>/hide (POST), /api/conversations/bulk-hide (POST).
  • User Experience: Hide button in conversation list, bulk selection interface, show hidden conversations toggle.
  • (Ref: chat-conversations.js, toggleConversationHide(), bulkHideConversations(), visibility management)

• Quick Search for Conversations

  • Real-time client-side conversation filtering for instant search results without server roundtrips.
  • Features: Real-time text filtering, searches conversation titles, client-side performance, keyboard shortcut support (Ctrl+K).
  • Search Scope: Filters visible conversations in current workspace, highlights matching conversations, instant results as you type.
  • User Experience: Search input in sidebar header, keyboard shortcut, clear button, responsive filtering.
  • (Ref: chat-conversations.js, toggleQuickSearch(), client-side filtering)

• Advanced Search Modal

  • Comprehensive search functionality with filters, pagination, and search history for finding conversations across all workspaces.
  • Features: Full-text search across conversation content, classification filters, date range selection, pagination support, search history tracking.
  • Search Capabilities: Search conversation titles and content, filter by workspace scope, filter by date range, view search history, export results.
  • User Experience: Modal interface with filter controls, results pagination, search history dropdown, results summary display.
  • Integration: Server-side search API, search history persistence, results caching.
  • (Ref: chat-search-modal.js, openAdvancedSearchModal(), performAdvancedSearch(), search history management)

• Automated Retention Policy System

  • Scheduled automatic deletion of aged conversations and documents based on configurable retention policies.
  • Features: User-configurable retention periods, separate policies for conversations and documents, scheduled execution via daemon thread, exemption support for protected conversations/documents.
  • Configuration Options: Retention periods by workspace scope (personal/group/public), auto-deletion scheduling (daily execution), user opt-in/opt-out controls, admin override capabilities.
  • Scopes: Personal workspace retention, group workspace retention, public workspace retention, per-user policy settings.
  • Safety Features: User exemption lists, dry-run mode for testing, deletion audit logging, grace period before deletion.
  • Integration: Background daemon thread, admin configuration interface, user profile settings, Cosmos DB TTL-based cleanup.
  • (Ref: functions_retention_policy.py, execute_retention_policy(), scheduled execution, user settings integration)

• Embedding Token Tracking

  • Comprehensive token tracking for document embedding generation in personal workspaces.
  • Tracking: Captures token usage per document chunk, accumulates total tokens, stores embedding tokens and model deployment name in document metadata.
  • Benefits: Embedding cost tracking, usage pattern analysis, document-level token metrics.
  • (Ref: functions_content.py, functions_documents.py, embedding token capture)

• Search Result Caching

  • Ensures consistent search results across identical queries with Cosmos DB-based distributed caching.
  • Features: Document set fingerprinting for cache invalidation, score normalization across indexes, 5-minute TTL, multi-instance deployment support.
  • Architecture: Cosmos DB search_cache container, SHA256 cache keys, automatic expiration, cache sharing across instances.
  • Benefits: Consistent user experience, reduced Azure AI Search costs, improved performance.
  • (Ref: functions_search.py, search_cache container, fingerprint-based invalidation)

• Activity Trends Visualization

  • Interactive Chart.js visualization of daily activity metrics in Control Center.
  • Categories: Chats, uploads, logins, document actions tracked separately.
  • Time Periods: 7-day, 30-day, and 90-day views.
  • Data Sources: Real data from Cosmos DB containers with sample data fallback.
  • (Ref: route_backend_control_center.py, control_center.html, Chart.js implementation)

• Group Activity Timeline

  • Comprehensive real-time view of all group workspace activities.
  • Activity Types: Document creation/deletion/updates, member additions/removals, status changes, conversations.
  • Features: Icon-based activity display, timestamp tracking, member attribution, detailed metadata.
  • Benefits: Group usage monitoring, audit trail, compliance tracking.
  • (Ref: route_frontend_groups.py, activity timeline UI, activity logs integration)

• Dynamic OpenAPI Schema Generation

  • Dynamic schema generation reducing hardcoded OpenAPI definitions.
  • Features: Analyzes Flask routes to generate schemas, maps routes to appropriate references, minimal required schemas for common patterns.
  • Benefits: Reduced maintenance overhead, automatic schema updates, comprehensive API documentation.
  • (Ref: route_external_openapi_spec.py, dynamic schema functions)

• Enhanced User Management

  • Comprehensive user activity metrics and analytics in Control Center.
  • Profile Features: Profile image display with Base64 support, chat metrics (conversations, messages, 3-month activity), document metrics (count, storage, AI search size).
  • Analytics: Last chat activity timestamps, storage estimations, feature status indicators.
  • (Ref: route_backend_control_center.py, enhanced user metrics)

• Group Status Management

  • Fine-grained control over group workspace operations through status-based access controls.
  • Status Types: Active (full functionality), Locked (read-only), Upload Disabled (no new uploads), Inactive (disabled).
  • Features: Full audit trail logging, operation-level restrictions, compliance support.
  • Use Cases: Legal holds, storage management, project lifecycle, risk mitigation.
  • (Ref: functions_groups.py, route_backend_groups.py, status enforcement)

• Workflow System

  • Document processing workflows including PII analysis and approval workflows.
  • Features: PDF document display in modals, workflow summary generation, approval routing, activity logging.
  • (Ref: route_frontend_workflow.py, workflow templates, CSP configuration)

• Full Width Chat Support

  • Option to expand chat interface to full browser width for better screen utilization.
  • (Ref: chats.html, responsive layout updates)

• Enhanced Document Metrics

  • Comprehensive document metadata tracking with enhanced analytics.
  • (Ref: document metrics implementation across containers)

• Group Member Activity Logging

  • Detailed logging when group members are added or removed.
  • (Ref: activity logging system, group member operations)

• Enable Group Creation Setting

  • Admin toggle to control whether users can create new groups.
  • (Ref: admin settings, group creation permissions)

• YAML OpenAPI Specification Support

  • Support for YAML format OpenAPI specifications alongside JSON.
  • (Ref: OpenAPI plugin system, YAML parsing)

• Inline OpenAPI Schema Generation

  • Generate OpenAPI schemas inline during plugin configuration.
  • (Ref: plugin configuration UI, schema generation)

• Microphone Permission Management

  • Improved handling of browser microphone permissions for speech-to-text.
  • (Ref: speech-to-text implementation, browser permissions)

Bug Fixes

• Agent Streaming Plugin Execution Fix

  • Fixed agent streaming failure when agents execute plugins during streaming.
  • Root Cause: Event loop conflicts from loop.run_until_complete(async_gen.__anext__()) pattern breaking async generator protocol.
  • Solution: Proper async/await pattern with asyncio.run() for complete async context.
  • Impact: Plugins like SmartHttpPlugin now work correctly in streaming mode.
  • (Ref: route_backend_chats.py, async generator handling, plugin execution)

• Search Cache Cosmos DB Migration

  • Migrated search caching from in-memory to Cosmos DB for multi-instance deployment support.
  • Problem: In-memory cache didn't share across App Service instances causing inconsistent results.
  • Solution: Cosmos DB search_cache container with 5-minute TTL, partition key on user_id.
  • Benefits: Cache sharing across instances, consistent user experience, distributed invalidation.
  • (Ref: functions_search.py, search_cache container, TTL configuration)

• Vision Model Parameter Fix

  • Fixed GPT-5 and o-series model failures in vision analysis with "Unsupported parameter: 'max_tokens'" error.
  • Root Cause: GPT-5 and o-series models require max_completion_tokens instead of max_tokens.
  • Solution: Dynamic parameter selection based on model type.
  • Impact: Vision analysis now works with all model families.
  • (Ref: route_backend_settings.py, functions_documents.py, model-aware parameters)

• Group Plugin Global Merge Fix

  • Fixed group workspaces unable to see globally managed actions when merge setting enabled.
  • Root Cause: /api/group/plugins endpoint didn't append global actions.
  • Solution: Merge global actions into group plugins response with read-only badges.
  • Impact: Groups can now select and use global actions while protecting them from modification.
  • (Ref: route_backend_groups.py, global plugin merging)

• Workflow Summary Generation O1 API Fix

  • Fixed o1 model failures in workflow summary generation with "Unsupported parameter: 'temperature'" error.
  • Root Cause: Unconditional application of temperature parameter to all models.
  • Solution: Conditional parameter logic excluding temperature for o1 models.
  • (Ref: route_frontend_workflow.py, model-aware parameter handling)

• Validation Utilities Consolidation

  • Consolidated duplicate validation functions across multiple files into centralized module.
  • Duplicated Functions: validateGuid() in 4 locations, validateEmail() in 2 locations.
  • Solution: Created validation-utils.js module with ValidationUtils namespace.
  • Benefits: Single source of truth, easier maintenance, consistency.
  • (Ref: validation-utils.js, code refactoring across control center and workspace files)

• Public Workspace Storage Calculation Fix

  • Fixed public workspaces showing 0 bytes storage despite having documents.
  • Root Cause: Incorrect folder prefix (public/{workspace_id}/ instead of {workspace_id}/).
  • Solution: Fixed folder prefix, enhanced fallback logic, improved error handling.
  • (Ref: route_backend_control_center.py, storage calculation logic)

• Public Workspace Metrics Caching Consistency Fix

  • Improved consistency in public workspace metrics caching across Control Center views.
  • (Ref: metrics caching implementation)

• Activity Timeline All Logs Fix

  • Fixed activity timeline to properly display all log types.
  • (Ref: activity log filtering)

• Activity Trends Field Mapping Fix

  • Corrected field mappings for activity trends data display.
  • (Ref: activity trends API, field mapping)

• All File Types Embedding Token Tracking Fix

  • Extended embedding token tracking to all file types beyond just text.
  • (Ref: functions_documents.py, comprehensive token tracking)

• PDF Embedding Token Tracking Fix

  • Fixed token tracking specifically for PDF document embeddings.
  • (Ref: PDF processing, token capture)

• Create Group Button Visibility Fix

  • Fixed group creation button visibility based on admin settings.
  • (Ref: UI conditional rendering, permission checks)

• File Message Metadata Loading Fix

  • Fixed metadata loading for file-related messages in conversations.
  • (Ref: message metadata display, file associations)

• Group Agent Metadata Fix

  • Corrected agent metadata display and management in group contexts.
  • (Ref: agent configuration, group agent handling)

• Group Document Metrics Date Format Fix

  • Fixed date formatting for group document metrics display.
  • (Ref: document metrics, date formatting)

• Group Notification Context Enhancement

  • Enhanced notification context for group-related activities.
  • (Ref: notification system, group context)

• Group Status UI Visibility Fix

  • Fixed UI visibility of group status indicators and controls.
  • (Ref: group status display, conditional UI rendering)

• Group Table Auto-Refresh Fix

  • Fixed automatic refresh of group tables after operations.
  • (Ref: table refresh logic, UI updates)

• Groups Tab Refresh Fix

  • Fixed refresh behavior on groups management tab.
  • (Ref: tab state management, data refresh)

• Hidden Conversations Sidebar Click Fix

  • Fixed sidebar click handling for hidden conversations.
  • (Ref: sidebar navigation, conversation visibility)

• Sidebar Group Badge Fix

  • Fixed group badge display in conversation sidebar.
  • (Ref: sidebar UI, badge rendering)

• Top Nav Sidebar Overlap Fix

  • Fixed overlapping issues between top navigation and sidebar in certain layouts.
  • (Ref: CSS layout, navigation positioning)

• Vision Analysis Debug Logging

  • Added comprehensive debug logging for vision analysis operations.
  • (Ref: functions_documents.py, debug logging)

• Workflow PDF Iframe CSP Fix

  • Fixed Content Security Policy for PDF display in workflow iframes.
  • (Ref: CSP headers, iframe configuration)

• Workflow PDF Viewer Height Fix

  • Fixed height issues in workflow PDF viewer modals.
  • (Ref: modal styling, PDF viewer layout)

• Workspace Activity Modal Fix

  • Fixed workspace activity modal display and interaction issues.
  • (Ref: modal functionality, workspace activity display)

• Search Cache Sharing Fix

  • Improved search cache sharing across user contexts.
  • (Ref: cache key generation, sharing logic)