Enhance speaker recognition service integration and error handling #245
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Updated `docker-compose-test.yml` to enable speaker recognition in the test environment and added a new `speaker-service-test` service for testing purposes. - Refactored `run-test.sh` to improve the execution of Robot Framework tests from the repository root. - Enhanced error handling in `speaker_recognition_client.py` to return detailed error messages for connection issues. - Improved error logging in `speaker_jobs.py` to handle and report errors from the speaker recognition service more effectively. - Updated `Dockerfile` to copy the full source code after dependencies are cached, ensuring all necessary files are included in the image.
Contributor
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
0xrushi
approved these changes
Jan 2, 2026
…EN verification - Deleted the `integration-tests.yml` workflow file to streamline CI processes. - Updated `robot-tests.yml` to include verification for the new `HF_TOKEN` secret, ensuring all required secrets are checked before running tests.
|
| Metric | Count |
|---|---|
| ✅ Passed | 90 |
| ❌ Failed | 4 |
| 📊 Total | 94 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
|
| Metric | Count |
|---|---|
| ✅ Passed | 92 |
| ❌ Failed | 2 |
| 📊 Total | 94 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
…ervices - Removed the creation of the test environment file from the GitHub Actions workflow to streamline setup. - Updated the Robot Framework tests to utilize a unified test script for improved consistency. - Enhanced error messages in the MemoryService class to provide more context on connection failures for LLM and vector store providers. - Added critical checks for API key presence in the OpenAIProvider class to ensure valid credentials are provided before proceeding. - Adjusted various test setup scripts to use a centralized BACKEND_DIR variable for better maintainability and clarity.
|
| Metric | Count |
|---|---|
| ✅ Passed | 90 |
| ❌ Failed | 4 |
| 📊 Total | 94 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
- Updated the script to dynamically construct container names from docker-compose services, improving maintainability and reducing hardcoded values. - Enhanced the cleanup process for stuck test containers by utilizing the COMPOSE_PROJECT_NAME variable.
AnkushMalaker
merged commit 4a4105a
into
fix/cleanup-pre-release-v0.1.0
1 of 2 checks passed
|
| Metric | Count |
|---|---|
| ✅ Passed | 91 |
| ❌ Failed | 3 |
| 📊 Total | 94 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
AnkushMalaker
added a commit
that referenced
this pull request
Jan 2, 2026
* Enhance configuration management and add new setup scripts - Updated .gitignore to include config.yml and its template. - Added config.yml.template for default configuration settings. - Introduced restart.sh script for service management. - Enhanced services.py to load config.yml and check for Obsidian/Neo4j integration. - Updated wizard.py to prompt for Obsidian/Neo4j configuration during setup and create config.yml from template if it doesn't exist. * Refactor transcription providers and enhance configuration management - Updated Docker Compose files to include the new Neo4j service configuration. - Added support for Obsidian/Neo4j integration in the setup process. - Refactored transcription providers to utilize a registry-driven approach for Deepgram and Parakeet. - Enhanced error handling and logging in transcription processes. - Improved environment variable management in test scripts to prioritize command-line overrides. - Removed deprecated Parakeet provider implementation and streamlined audio stream workers. * Update configuration management and enhance file structure, add test-matrix (#237) * Update configuration management and enhance file structure - Refactored configuration file paths to use a dedicated `config/` directory, including updates to `config.yml` and its template. - Modified service scripts to load the new configuration path for `config.yml`. - Enhanced `.gitignore` to include the new configuration files and templates. - Updated documentation to reflect changes in configuration file locations and usage. - Improved setup scripts to ensure proper creation and management of configuration files. - Added new test configurations for various provider combinations to streamline testing processes. * Add test requirements and clean up imports in wizard.py - Introduced a new `test-requirements.txt` file to manage testing dependencies. - Removed redundant import of `shutil` in `wizard.py` to improve code clarity. * Add ConfigManager for unified configuration management - Introduced a new `config_manager.py` module to handle reading and writing configurations from `config.yml` and `.env` files, ensuring backward compatibility. - Refactored `ChronicleSetup` in `backends/advanced/init.py` to utilize `ConfigManager` for loading and updating configurations, simplifying the setup process. - Removed redundant methods for loading and saving `config.yml` directly in `ChronicleSetup`, as these are now managed by `ConfigManager`. - Enhanced user feedback during configuration updates, including success messages for changes made to configuration files. * Refactor transcription provider configuration and enhance setup process - Updated `.env.template` to clarify speech-to-text configuration and removed deprecated options for Mistral. - Modified `docker-compose.yml` to streamline environment variable management by removing unused Mistral keys. - Enhanced `ChronicleSetup` in `init.py` to provide clearer user feedback and updated the transcription provider selection process to rely on `config.yml`. - Improved error handling in the websocket controller to determine the transcription provider from the model registry instead of environment variables. - Updated health check routes to reflect the new method of retrieving the transcription provider from `config.yml`. - Adjusted `config.yml.template` to include comments on transcription provider options for better user guidance. * Enhance ConfigManager with deep merge functionality - Updated the `update_memory_config` method to perform a deep merge of updates into the memory configuration, ensuring nested dictionaries are merged correctly. - Added a new `_deep_merge` method to handle recursive merging of dictionaries, improving configuration management capabilities. * Refactor run-test.sh and enhance memory extraction tests - Removed deprecated environment variable handling for TRANSCRIPTION_PROVIDER in `run-test.sh`, streamlining the configuration process. - Introduced a new `run-custom.sh` script for executing Robot tests with custom configurations, improving test flexibility. - Enhanced memory extraction tests in `audio_keywords.robot` and `memory_keywords.robot` to include detailed assertions and result handling. - Updated `queue_keywords.robot` to fail fast if a job is in a 'failed' state when expecting 'completed', improving error handling. - Refactored `test_env.py` to load environment variables with correct precedence, ensuring better configuration management. * unify tests to robot test, add some more clean up * Update health check configuration in docker-compose-test.yml (#241) - Increased the number of retries from 5 to 10 for improved resilience during service readiness checks. - Extended the start period from 30s to 60s to allow more time for services to initialize before health checks commence. * Add step to create test configuration file in robot-tests.yml - Introduced a new step in the GitHub Actions workflow to copy the test configuration file from tests/configs/deepgram-openai.yml to a new config/config.yml. - Added logging to confirm the creation of the test config file, improving visibility during the test setup process. * remove cache step since not required * coderabbit comments * Refactor ConfigManager error handling for configuration file loading - Updated the ConfigManager to raise RuntimeError exceptions when the configuration file is not found or is invalid, improving error visibility and user guidance. - Removed fallback behavior that previously returned the current directory, ensuring users are explicitly informed about missing or invalid configuration files. * Refactor _find_repo_root method in ConfigManager - Updated the _find_repo_root method to locate the repository root using the __file__ location instead of searching for config/config.yml, simplifying the logic and improving reliability. - Removed the previous error handling that raised a RuntimeError if the configuration file was not found, as the new approach assumes config_manager.py is always at the repo root. * Enhance speaker recognition service integration and error handling (#245) * Enhance speaker recognition service integration and error handling - Updated `docker-compose-test.yml` to enable speaker recognition in the test environment and added a new `speaker-service-test` service for testing purposes. - Refactored `run-test.sh` to improve the execution of Robot Framework tests from the repository root. - Enhanced error handling in `speaker_recognition_client.py` to return detailed error messages for connection issues. - Improved error logging in `speaker_jobs.py` to handle and report errors from the speaker recognition service more effectively. - Updated `Dockerfile` to copy the full source code after dependencies are cached, ensuring all necessary files are included in the image. * Remove integration tests workflow and enhance robot tests with HF_TOKEN verification - Deleted the `integration-tests.yml` workflow file to streamline CI processes. - Updated `robot-tests.yml` to include verification for the new `HF_TOKEN` secret, ensuring all required secrets are checked before running tests. * Fix key access in system admin tests to use string indexing for speakers data * Refactor Robot Framework tests and enhance error handling in memory services - Removed the creation of the test environment file from the GitHub Actions workflow to streamline setup. - Updated the Robot Framework tests to utilize a unified test script for improved consistency. - Enhanced error messages in the MemoryService class to provide more context on connection failures for LLM and vector store providers. - Added critical checks for API key presence in the OpenAIProvider class to ensure valid credentials are provided before proceeding. - Adjusted various test setup scripts to use a centralized BACKEND_DIR variable for better maintainability and clarity. * Refactor test container cleanup in run-robot-tests.sh - Updated the script to dynamically construct container names from docker-compose services, improving maintainability and reducing hardcoded values. - Enhanced the cleanup process for stuck test containers by utilizing the COMPOSE_PROJECT_NAME variable. * Enhance run-robot-tests.sh for improved logging and cleanup - Set absolute paths for consistent directory references to simplify navigation. - Capture container logs, status, and resource usage for better debugging. - Refactor cleanup process to utilize dynamic backend directory references, improving maintainability. - Ensure proper navigation back to the tests directory after operations. * Add speaker recognition configuration and update test script defaults - Introduced speaker recognition settings in config.yml.template, allowing for easy enable/disable and service URL configuration. - Updated run-robot-tests.sh to use a test-specific configuration file that disables speaker recognition for improved CI performance. - Modified deepgram-openai.yml to disable speaker recognition during CI tests to enhance execution speed. * Refactor speaker recognition configuration management - Updated docker-compose-test.yml to clarify speaker recognition settings, now controlled via config.yml for improved CI performance. - Enhanced model_registry.py to include a dedicated speaker_recognition field for better configuration handling. - Modified speaker_recognition_client.py to load configuration from config.yml, allowing for dynamic enabling/disabling of the speaker recognition service based on the configuration. * Add minimum worker count verification to infrastructure tests - Introduced a new keyword to verify that the minimum number of workers are registered, enhancing the robustness of health checks. - Updated the worker count validation test to include a wait mechanism for worker registration, improving test reliability. - Clarified comments regarding expected worker counts to reflect the distinction between RQ and audio stream workers. * Update configuration management and enhance model handling - Added OBSIDIAN_ENABLED configuration to ChronicleSetup for improved feature toggling. - Introduced speaker_recognition configuration handling in model_registry.py to streamline model loading. - Refactored imports in deepgram.py to improve clarity and reduce redundancy.
AnkushMalaker
added a commit
that referenced
this pull request
Jan 3, 2026
* audio upload extension with gdrive credentials * FIX: API parameters * UPDATE: tmp files cleanup n code refactored as per review * REFACTOR: minor refactor as per review * REFACTOR: minor update as per review * UPDATE: gdrive sync logic * REFACTOR: code update as per gdrive and update credential client * REFACTOR: validation updated - as per review from CR * UPDATE: code has been refactore for UUID for diffrent audio upload sources * REFACTOR: updated code as per review * Update documentation and configuration to reflect the transition from 'friend-backend' to 'chronicle-backend' across various files, including setup instructions, Docker configurations, and service logs. * Update test script to use docker-compose-test.yml for all test-related operations * Added standard MIT license * Fix/cleanup model (#219) * refactor memory * add config * docstring * more cleanup * code quality * code quality * unused return * DOTTED GET * Refactor Docker and CI configurations - Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process. - Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services. - Added new dependencies for Google API clients in `uv.lock` to support upcoming features. * Update configuration files for model providers and Docker setup - Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram. - Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving. - Updated memory configuration endpoint to accept plain text for YAML input. * Update transcription job handling to format speaker IDs - Changed variable name from `speaker_name` to `speaker_id` for clarity. - Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling. * Remove loading of backend .env file in test environment setup - Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests. * Enhance configuration management and setup wizard - Updated README to reflect the new setup wizard process. - Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers. - Improved user feedback during configuration updates, including success messages for configuration file updates. - Enabled backup of existing `config.yml` before saving changes. * Enhance HTTPS configuration in setup wizard - Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates. - Improved user prompts for server IP/domain input during HTTPS setup. - Updated default behavior to use existing IP or localhost based on user input. - Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access. * Add source parameter to audio file writing in websocket controller - Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking. --------- Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com> * fix/broken-tests (#230) * refactor memory * add config * docstring * more cleanup * code quality * code quality * unused return * DOTTED GET * Refactor Docker and CI configurations - Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process. - Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services. - Added new dependencies for Google API clients in `uv.lock` to support upcoming features. * Update configuration files for model providers and Docker setup - Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram. - Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving. - Updated memory configuration endpoint to accept plain text for YAML input. * Update transcription job handling to format speaker IDs - Changed variable name from `speaker_name` to `speaker_id` for clarity. - Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling. * Remove loading of backend .env file in test environment setup - Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests. * Enhance configuration management and setup wizard - Updated README to reflect the new setup wizard process. - Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers. - Improved user feedback during configuration updates, including success messages for configuration file updates. - Enabled backup of existing `config.yml` before saving changes. * Enhance HTTPS configuration in setup wizard - Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates. - Improved user prompts for server IP/domain input during HTTPS setup. - Updated default behavior to use existing IP or localhost based on user input. - Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access. * Add source parameter to audio file writing in websocket controller - Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking. * Refactor error handling in system controller and update memory config routes - Replaced ValueError with HTTPException for better error handling in `save_diarization_settings` and `validate_memory_config` functions. - Introduced a new Pydantic model, `MemoryConfigRequest`, for validating memory configuration requests in the system routes. - Updated the `validate_memory_config` endpoint to accept the new request model, improving input handling and validation. --------- Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com> * Feat/add obsidian 3 (#233) * obsidian support * neo4j comment * cleanup code * unused line * unused line * Fix MemoryEntry object usage in chat service * comment * feat(obsidian): add obsidian memory search integration to chat * unit test * use rq * neo4j service * typefix * test fix * cleanup * cleanup * version changes * profile * remove unused imports * Refactor memory configuration validation endpoints - Removed the deprecated `validate_memory_config_raw` endpoint and replaced it with a new endpoint that accepts plain text for validation. - Updated the existing `validate_memory_config` endpoint to clarify that it now accepts JSON input. - Adjusted the API call in the frontend to point to the new validation endpoint. * Refactor health check model configuration loading - Updated the health check function to load model configuration from the models registry instead of the root config. - Improved error handling by logging warnings when model configuration loading fails. --------- Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com> * Update .gitignore to exclude all files in app/ios and app/android directories (#238) * fix: Copy full source code in speaker-recognition Dockerfile (#243) Adds COPY src/ src/ step after dependency installation to ensure all source files are available in the Docker image. This improves build caching while ensuring complete source code is present. * Enhance configuration management and add new setup scripts (#235) * Enhance configuration management and add new setup scripts - Updated .gitignore to include config.yml and its template. - Added config.yml.template for default configuration settings. - Introduced restart.sh script for service management. - Enhanced services.py to load config.yml and check for Obsidian/Neo4j integration. - Updated wizard.py to prompt for Obsidian/Neo4j configuration during setup and create config.yml from template if it doesn't exist. * Refactor transcription providers and enhance configuration management - Updated Docker Compose files to include the new Neo4j service configuration. - Added support for Obsidian/Neo4j integration in the setup process. - Refactored transcription providers to utilize a registry-driven approach for Deepgram and Parakeet. - Enhanced error handling and logging in transcription processes. - Improved environment variable management in test scripts to prioritize command-line overrides. - Removed deprecated Parakeet provider implementation and streamlined audio stream workers. * Update configuration management and enhance file structure, add test-matrix (#237) * Update configuration management and enhance file structure - Refactored configuration file paths to use a dedicated `config/` directory, including updates to `config.yml` and its template. - Modified service scripts to load the new configuration path for `config.yml`. - Enhanced `.gitignore` to include the new configuration files and templates. - Updated documentation to reflect changes in configuration file locations and usage. - Improved setup scripts to ensure proper creation and management of configuration files. - Added new test configurations for various provider combinations to streamline testing processes. * Add test requirements and clean up imports in wizard.py - Introduced a new `test-requirements.txt` file to manage testing dependencies. - Removed redundant import of `shutil` in `wizard.py` to improve code clarity. * Add ConfigManager for unified configuration management - Introduced a new `config_manager.py` module to handle reading and writing configurations from `config.yml` and `.env` files, ensuring backward compatibility. - Refactored `ChronicleSetup` in `backends/advanced/init.py` to utilize `ConfigManager` for loading and updating configurations, simplifying the setup process. - Removed redundant methods for loading and saving `config.yml` directly in `ChronicleSetup`, as these are now managed by `ConfigManager`. - Enhanced user feedback during configuration updates, including success messages for changes made to configuration files. * Refactor transcription provider configuration and enhance setup process - Updated `.env.template` to clarify speech-to-text configuration and removed deprecated options for Mistral. - Modified `docker-compose.yml` to streamline environment variable management by removing unused Mistral keys. - Enhanced `ChronicleSetup` in `init.py` to provide clearer user feedback and updated the transcription provider selection process to rely on `config.yml`. - Improved error handling in the websocket controller to determine the transcription provider from the model registry instead of environment variables. - Updated health check routes to reflect the new method of retrieving the transcription provider from `config.yml`. - Adjusted `config.yml.template` to include comments on transcription provider options for better user guidance. * Enhance ConfigManager with deep merge functionality - Updated the `update_memory_config` method to perform a deep merge of updates into the memory configuration, ensuring nested dictionaries are merged correctly. - Added a new `_deep_merge` method to handle recursive merging of dictionaries, improving configuration management capabilities. * Refactor run-test.sh and enhance memory extraction tests - Removed deprecated environment variable handling for TRANSCRIPTION_PROVIDER in `run-test.sh`, streamlining the configuration process. - Introduced a new `run-custom.sh` script for executing Robot tests with custom configurations, improving test flexibility. - Enhanced memory extraction tests in `audio_keywords.robot` and `memory_keywords.robot` to include detailed assertions and result handling. - Updated `queue_keywords.robot` to fail fast if a job is in a 'failed' state when expecting 'completed', improving error handling. - Refactored `test_env.py` to load environment variables with correct precedence, ensuring better configuration management. * unify tests to robot test, add some more clean up * Update health check configuration in docker-compose-test.yml (#241) - Increased the number of retries from 5 to 10 for improved resilience during service readiness checks. - Extended the start period from 30s to 60s to allow more time for services to initialize before health checks commence. * Add step to create test configuration file in robot-tests.yml - Introduced a new step in the GitHub Actions workflow to copy the test configuration file from tests/configs/deepgram-openai.yml to a new config/config.yml. - Added logging to confirm the creation of the test config file, improving visibility during the test setup process. * remove cache step since not required * coderabbit comments * Refactor ConfigManager error handling for configuration file loading - Updated the ConfigManager to raise RuntimeError exceptions when the configuration file is not found or is invalid, improving error visibility and user guidance. - Removed fallback behavior that previously returned the current directory, ensuring users are explicitly informed about missing or invalid configuration files. * Refactor _find_repo_root method in ConfigManager - Updated the _find_repo_root method to locate the repository root using the __file__ location instead of searching for config/config.yml, simplifying the logic and improving reliability. - Removed the previous error handling that raised a RuntimeError if the configuration file was not found, as the new approach assumes config_manager.py is always at the repo root. * Enhance speaker recognition service integration and error handling (#245) * Enhance speaker recognition service integration and error handling - Updated `docker-compose-test.yml` to enable speaker recognition in the test environment and added a new `speaker-service-test` service for testing purposes. - Refactored `run-test.sh` to improve the execution of Robot Framework tests from the repository root. - Enhanced error handling in `speaker_recognition_client.py` to return detailed error messages for connection issues. - Improved error logging in `speaker_jobs.py` to handle and report errors from the speaker recognition service more effectively. - Updated `Dockerfile` to copy the full source code after dependencies are cached, ensuring all necessary files are included in the image. * Remove integration tests workflow and enhance robot tests with HF_TOKEN verification - Deleted the `integration-tests.yml` workflow file to streamline CI processes. - Updated `robot-tests.yml` to include verification for the new `HF_TOKEN` secret, ensuring all required secrets are checked before running tests. * Fix key access in system admin tests to use string indexing for speakers data * Refactor Robot Framework tests and enhance error handling in memory services - Removed the creation of the test environment file from the GitHub Actions workflow to streamline setup. - Updated the Robot Framework tests to utilize a unified test script for improved consistency. - Enhanced error messages in the MemoryService class to provide more context on connection failures for LLM and vector store providers. - Added critical checks for API key presence in the OpenAIProvider class to ensure valid credentials are provided before proceeding. - Adjusted various test setup scripts to use a centralized BACKEND_DIR variable for better maintainability and clarity. * Refactor test container cleanup in run-robot-tests.sh - Updated the script to dynamically construct container names from docker-compose services, improving maintainability and reducing hardcoded values. - Enhanced the cleanup process for stuck test containers by utilizing the COMPOSE_PROJECT_NAME variable. * Enhance run-robot-tests.sh for improved logging and cleanup - Set absolute paths for consistent directory references to simplify navigation. - Capture container logs, status, and resource usage for better debugging. - Refactor cleanup process to utilize dynamic backend directory references, improving maintainability. - Ensure proper navigation back to the tests directory after operations. * Add speaker recognition configuration and update test script defaults - Introduced speaker recognition settings in config.yml.template, allowing for easy enable/disable and service URL configuration. - Updated run-robot-tests.sh to use a test-specific configuration file that disables speaker recognition for improved CI performance. - Modified deepgram-openai.yml to disable speaker recognition during CI tests to enhance execution speed. * Refactor speaker recognition configuration management - Updated docker-compose-test.yml to clarify speaker recognition settings, now controlled via config.yml for improved CI performance. - Enhanced model_registry.py to include a dedicated speaker_recognition field for better configuration handling. - Modified speaker_recognition_client.py to load configuration from config.yml, allowing for dynamic enabling/disabling of the speaker recognition service based on the configuration. * Add minimum worker count verification to infrastructure tests - Introduced a new keyword to verify that the minimum number of workers are registered, enhancing the robustness of health checks. - Updated the worker count validation test to include a wait mechanism for worker registration, improving test reliability. - Clarified comments regarding expected worker counts to reflect the distinction between RQ and audio stream workers. * Update configuration management and enhance model handling - Added OBSIDIAN_ENABLED configuration to ChronicleSetup for improved feature toggling. - Introduced speaker_recognition configuration handling in model_registry.py to streamline model loading. - Refactored imports in deepgram.py to improve clarity and reduce redundancy. * Refactor configuration management in wizard and ChronicleSetup (#246) * Refactor configuration management in wizard and ChronicleSetup - Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling. - Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth. - Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers. * Fix string formatting for error message in ChronicleSetup --------- Co-authored-by: 01PrathamS <pratham21btai35@karnavatiuniversity.edu.in> Co-authored-by: Stu Alexandere <thestumonkey@gmail.com> Co-authored-by: Stuart Alexander <stu@theawesome.co.uk> Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
AnkushMalaker
added a commit
that referenced
this pull request
Jan 6, 2026
* audio upload extension with gdrive credentials * FIX: API parameters * UPDATE: tmp files cleanup n code refactored as per review * REFACTOR: minor refactor as per review * REFACTOR: minor update as per review * UPDATE: gdrive sync logic * REFACTOR: code update as per gdrive and update credential client * REFACTOR: validation updated - as per review from CR * UPDATE: code has been refactore for UUID for diffrent audio upload sources * REFACTOR: updated code as per review * Update documentation and configuration to reflect the transition from 'friend-backend' to 'chronicle-backend' across various files, including setup instructions, Docker configurations, and service logs. * Update test script to use docker-compose-test.yml for all test-related operations * Added standard MIT license * Fix/cleanup model (#219) * refactor memory * add config * docstring * more cleanup * code quality * code quality * unused return * DOTTED GET * Refactor Docker and CI configurations - Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process. - Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services. - Added new dependencies for Google API clients in `uv.lock` to support upcoming features. * Update configuration files for model providers and Docker setup - Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram. - Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving. - Updated memory configuration endpoint to accept plain text for YAML input. * Update transcription job handling to format speaker IDs - Changed variable name from `speaker_name` to `speaker_id` for clarity. - Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling. * Remove loading of backend .env file in test environment setup - Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests. * Enhance configuration management and setup wizard - Updated README to reflect the new setup wizard process. - Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers. - Improved user feedback during configuration updates, including success messages for configuration file updates. - Enabled backup of existing `config.yml` before saving changes. * Enhance HTTPS configuration in setup wizard - Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates. - Improved user prompts for server IP/domain input during HTTPS setup. - Updated default behavior to use existing IP or localhost based on user input. - Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access. * Add source parameter to audio file writing in websocket controller - Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking. --------- Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com> * fix/broken-tests (#230) * refactor memory * add config * docstring * more cleanup * code quality * code quality * unused return * DOTTED GET * Refactor Docker and CI configurations - Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process. - Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services. - Added new dependencies for Google API clients in `uv.lock` to support upcoming features. * Update configuration files for model providers and Docker setup - Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram. - Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving. - Updated memory configuration endpoint to accept plain text for YAML input. * Update transcription job handling to format speaker IDs - Changed variable name from `speaker_name` to `speaker_id` for clarity. - Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling. * Remove loading of backend .env file in test environment setup - Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests. * Enhance configuration management and setup wizard - Updated README to reflect the new setup wizard process. - Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers. - Improved user feedback during configuration updates, including success messages for configuration file updates. - Enabled backup of existing `config.yml` before saving changes. * Enhance HTTPS configuration in setup wizard - Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates. - Improved user prompts for server IP/domain input during HTTPS setup. - Updated default behavior to use existing IP or localhost based on user input. - Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access. * Add source parameter to audio file writing in websocket controller - Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking. * Refactor error handling in system controller and update memory config routes - Replaced ValueError with HTTPException for better error handling in `save_diarization_settings` and `validate_memory_config` functions. - Introduced a new Pydantic model, `MemoryConfigRequest`, for validating memory configuration requests in the system routes. - Updated the `validate_memory_config` endpoint to accept the new request model, improving input handling and validation. --------- Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com> * Feat/add obsidian 3 (#233) * obsidian support * neo4j comment * cleanup code * unused line * unused line * Fix MemoryEntry object usage in chat service * comment * feat(obsidian): add obsidian memory search integration to chat * unit test * use rq * neo4j service * typefix * test fix * cleanup * cleanup * version changes * profile * remove unused imports * Refactor memory configuration validation endpoints - Removed the deprecated `validate_memory_config_raw` endpoint and replaced it with a new endpoint that accepts plain text for validation. - Updated the existing `validate_memory_config` endpoint to clarify that it now accepts JSON input. - Adjusted the API call in the frontend to point to the new validation endpoint. * Refactor health check model configuration loading - Updated the health check function to load model configuration from the models registry instead of the root config. - Improved error handling by logging warnings when model configuration loading fails. --------- Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com> * Update .gitignore to exclude all files in app/ios and app/android directories (#238) * fix: Copy full source code in speaker-recognition Dockerfile (#243) Adds COPY src/ src/ step after dependency installation to ensure all source files are available in the Docker image. This improves build caching while ensuring complete source code is present. * Enhance configuration management and add new setup scripts (#235) * Enhance configuration management and add new setup scripts - Updated .gitignore to include config.yml and its template. - Added config.yml.template for default configuration settings. - Introduced restart.sh script for service management. - Enhanced services.py to load config.yml and check for Obsidian/Neo4j integration. - Updated wizard.py to prompt for Obsidian/Neo4j configuration during setup and create config.yml from template if it doesn't exist. * Refactor transcription providers and enhance configuration management - Updated Docker Compose files to include the new Neo4j service configuration. - Added support for Obsidian/Neo4j integration in the setup process. - Refactored transcription providers to utilize a registry-driven approach for Deepgram and Parakeet. - Enhanced error handling and logging in transcription processes. - Improved environment variable management in test scripts to prioritize command-line overrides. - Removed deprecated Parakeet provider implementation and streamlined audio stream workers. * Update configuration management and enhance file structure, add test-matrix (#237) * Update configuration management and enhance file structure - Refactored configuration file paths to use a dedicated `config/` directory, including updates to `config.yml` and its template. - Modified service scripts to load the new configuration path for `config.yml`. - Enhanced `.gitignore` to include the new configuration files and templates. - Updated documentation to reflect changes in configuration file locations and usage. - Improved setup scripts to ensure proper creation and management of configuration files. - Added new test configurations for various provider combinations to streamline testing processes. * Add test requirements and clean up imports in wizard.py - Introduced a new `test-requirements.txt` file to manage testing dependencies. - Removed redundant import of `shutil` in `wizard.py` to improve code clarity. * Add ConfigManager for unified configuration management - Introduced a new `config_manager.py` module to handle reading and writing configurations from `config.yml` and `.env` files, ensuring backward compatibility. - Refactored `ChronicleSetup` in `backends/advanced/init.py` to utilize `ConfigManager` for loading and updating configurations, simplifying the setup process. - Removed redundant methods for loading and saving `config.yml` directly in `ChronicleSetup`, as these are now managed by `ConfigManager`. - Enhanced user feedback during configuration updates, including success messages for changes made to configuration files. * Refactor transcription provider configuration and enhance setup process - Updated `.env.template` to clarify speech-to-text configuration and removed deprecated options for Mistral. - Modified `docker-compose.yml` to streamline environment variable management by removing unused Mistral keys. - Enhanced `ChronicleSetup` in `init.py` to provide clearer user feedback and updated the transcription provider selection process to rely on `config.yml`. - Improved error handling in the websocket controller to determine the transcription provider from the model registry instead of environment variables. - Updated health check routes to reflect the new method of retrieving the transcription provider from `config.yml`. - Adjusted `config.yml.template` to include comments on transcription provider options for better user guidance. * Enhance ConfigManager with deep merge functionality - Updated the `update_memory_config` method to perform a deep merge of updates into the memory configuration, ensuring nested dictionaries are merged correctly. - Added a new `_deep_merge` method to handle recursive merging of dictionaries, improving configuration management capabilities. * Refactor run-test.sh and enhance memory extraction tests - Removed deprecated environment variable handling for TRANSCRIPTION_PROVIDER in `run-test.sh`, streamlining the configuration process. - Introduced a new `run-custom.sh` script for executing Robot tests with custom configurations, improving test flexibility. - Enhanced memory extraction tests in `audio_keywords.robot` and `memory_keywords.robot` to include detailed assertions and result handling. - Updated `queue_keywords.robot` to fail fast if a job is in a 'failed' state when expecting 'completed', improving error handling. - Refactored `test_env.py` to load environment variables with correct precedence, ensuring better configuration management. * unify tests to robot test, add some more clean up * Update health check configuration in docker-compose-test.yml (#241) - Increased the number of retries from 5 to 10 for improved resilience during service readiness checks. - Extended the start period from 30s to 60s to allow more time for services to initialize before health checks commence. * Add step to create test configuration file in robot-tests.yml - Introduced a new step in the GitHub Actions workflow to copy the test configuration file from tests/configs/deepgram-openai.yml to a new config/config.yml. - Added logging to confirm the creation of the test config file, improving visibility during the test setup process. * remove cache step since not required * coderabbit comments * Refactor ConfigManager error handling for configuration file loading - Updated the ConfigManager to raise RuntimeError exceptions when the configuration file is not found or is invalid, improving error visibility and user guidance. - Removed fallback behavior that previously returned the current directory, ensuring users are explicitly informed about missing or invalid configuration files. * Refactor _find_repo_root method in ConfigManager - Updated the _find_repo_root method to locate the repository root using the __file__ location instead of searching for config/config.yml, simplifying the logic and improving reliability. - Removed the previous error handling that raised a RuntimeError if the configuration file was not found, as the new approach assumes config_manager.py is always at the repo root. * Enhance speaker recognition service integration and error handling (#245) * Enhance speaker recognition service integration and error handling - Updated `docker-compose-test.yml` to enable speaker recognition in the test environment and added a new `speaker-service-test` service for testing purposes. - Refactored `run-test.sh` to improve the execution of Robot Framework tests from the repository root. - Enhanced error handling in `speaker_recognition_client.py` to return detailed error messages for connection issues. - Improved error logging in `speaker_jobs.py` to handle and report errors from the speaker recognition service more effectively. - Updated `Dockerfile` to copy the full source code after dependencies are cached, ensuring all necessary files are included in the image. * Remove integration tests workflow and enhance robot tests with HF_TOKEN verification - Deleted the `integration-tests.yml` workflow file to streamline CI processes. - Updated `robot-tests.yml` to include verification for the new `HF_TOKEN` secret, ensuring all required secrets are checked before running tests. * Fix key access in system admin tests to use string indexing for speakers data * Refactor Robot Framework tests and enhance error handling in memory services - Removed the creation of the test environment file from the GitHub Actions workflow to streamline setup. - Updated the Robot Framework tests to utilize a unified test script for improved consistency. - Enhanced error messages in the MemoryService class to provide more context on connection failures for LLM and vector store providers. - Added critical checks for API key presence in the OpenAIProvider class to ensure valid credentials are provided before proceeding. - Adjusted various test setup scripts to use a centralized BACKEND_DIR variable for better maintainability and clarity. * Refactor test container cleanup in run-robot-tests.sh - Updated the script to dynamically construct container names from docker-compose services, improving maintainability and reducing hardcoded values. - Enhanced the cleanup process for stuck test containers by utilizing the COMPOSE_PROJECT_NAME variable. * Enhance run-robot-tests.sh for improved logging and cleanup - Set absolute paths for consistent directory references to simplify navigation. - Capture container logs, status, and resource usage for better debugging. - Refactor cleanup process to utilize dynamic backend directory references, improving maintainability. - Ensure proper navigation back to the tests directory after operations. * Add speaker recognition configuration and update test script defaults - Introduced speaker recognition settings in config.yml.template, allowing for easy enable/disable and service URL configuration. - Updated run-robot-tests.sh to use a test-specific configuration file that disables speaker recognition for improved CI performance. - Modified deepgram-openai.yml to disable speaker recognition during CI tests to enhance execution speed. * Refactor speaker recognition configuration management - Updated docker-compose-test.yml to clarify speaker recognition settings, now controlled via config.yml for improved CI performance. - Enhanced model_registry.py to include a dedicated speaker_recognition field for better configuration handling. - Modified speaker_recognition_client.py to load configuration from config.yml, allowing for dynamic enabling/disabling of the speaker recognition service based on the configuration. * Add minimum worker count verification to infrastructure tests - Introduced a new keyword to verify that the minimum number of workers are registered, enhancing the robustness of health checks. - Updated the worker count validation test to include a wait mechanism for worker registration, improving test reliability. - Clarified comments regarding expected worker counts to reflect the distinction between RQ and audio stream workers. * Update configuration management and enhance model handling - Added OBSIDIAN_ENABLED configuration to ChronicleSetup for improved feature toggling. - Introduced speaker_recognition configuration handling in model_registry.py to streamline model loading. - Refactored imports in deepgram.py to improve clarity and reduce redundancy. * Refactor configuration management in wizard and ChronicleSetup (#246) * Refactor configuration management in wizard and ChronicleSetup - Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling. - Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth. - Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers. * Fix string formatting for error message in ChronicleSetup * added JWT issuers for audience auth for service interop and shared us… (#250) * added JWT issuers for audience auth for service interop and shared user accounts * amended default value in line wioth code --------- Co-authored-by: 01PrathamS <pratham21btai35@karnavatiuniversity.edu.in> Co-authored-by: Stu Alexandere <thestumonkey@gmail.com> Co-authored-by: Stuart Alexander <stu@theawesome.co.uk> Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
docker-compose-test.ymlto enable speaker recognition in the test environment and added a newspeaker-service-testservice for testing purposes.run-test.shto improve the execution of Robot Framework tests from the repository root.speaker_recognition_client.pyto return detailed error messages for connection issues.speaker_jobs.pyto handle and report errors from the speaker recognition service more effectively.Dockerfileto copy the full source code after dependencies are cached, ensuring all necessary files are included in the image.