Refactor logging

[mrT23]

User description

Goals

• divide the log levels into a simple info/debug levels

• log every relevant detail from a run, to ensure each run can be properly debugged, in an easy and accessible way:

  • relevant configuration
  • input patches
  • AI prompt
  • AI text response
  • AI response extra details
  • prev and new labels
  • actual answer sent back to github
  • more

• avoid generic log messages without actual data, to reduce clutter

---

Type

enhancement, documentation

---

Description

• Enhanced logging across various modules for better readability and debugging.
• Introduced dynamic log level setting based on environment variable.
• Adjusted logging levels and added debug logs for relevant configurations and outputs.
• Improved handling and logging for user descriptions in Git provider.

---

Changes walkthrough

	Relevant files
Enhancement	13 files litellm_ai_handler.py Enhance AI Handler Logging                                                              pr_agent/algo/ai_handlers/litellm_ai_handler.py Added debug logging for prompts and full AI response. Removed logging of usage from AI response. +5/-3      pr_processing.py Improve PR Processing Logging                                                        pr_agent/algo/pr_processing.py Added info logging for filtered files, PR main language, tokens count, and pruning details. Enhanced debug logging for the final diff after pruning. +29/-7    utils.py Adjust Logging Level for User Labels                                          pr_agent/algo/utils.py Changed logging level for keeping user labels from info to debug. +1/-1      git_provider.py Enhance Git Provider Logging and Handling                                pr_agent/git_providers/git_provider.py Added handling for user_description attribute. Changed logging level for existing description to debug. +7/-2      utils.py Improve Logging for Repo Settings Application                        pr_agent/git_providers/utils.py Improved logging for applying repo settings. +1/-1      __init__.py Refine Logging Setup                                                                          pr_agent/log/init.py Adjusted logging setup for JSON format and console output. +3/-2      github_app.py Enhance Logging in GitHub App Server                                          pr_agent/servers/github_app.py Enhanced logging for request body in various event handling. +3/-3      pr_code_suggestions.py Improve Logging in PR Code Suggestions Tool                            pr_agent/tools/pr_code_suggestions.py Added debug logging for relevant configs and PR output. Removed verbose info logging. +39/-47  pr_description.py Improve Logging in PR Description Tool                                      pr_agent/tools/pr_description.py Added debug logging for relevant configs and PR output. Adjusted logging for error handling and configurations. +22/-40  pr_help_message.py Improve Logging in PR Help Message Tool                                    pr_agent/tools/pr_help_message.py Added debug logging for relevant configs and PR output. +4/-0      pr_questions.py Improve Logging in PR Questions Tool                                          pr_agent/tools/pr_questions.py Added debug logging for relevant configs and PR output. +13/-10  pr_reviewer.py Improve Logging in PR Reviewer Tool                                            pr_agent/tools/pr_reviewer.py Added debug logging for relevant configs and PR output. Enhanced AI response logging. +12/-26  pr_update_changelog.py Improve Logging in PR Update Changelog Tool                            pr_agent/tools/pr_update_changelog.py Added debug logging for relevant configs and PR output. +14/-15
Configuration changes	1 files cli.py Dynamic Log Level Configuration                                                    pr_agent/cli.py Introduced dynamic log level setting based on environment variable. +2/-1

---

✨ PR-Agent usage:
Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

[codiumai-pr-agent-pro]

PR Description updated to latest commit (b40894b)

• Copy walkthrough table to "Files Changed" Tab

[codiumai-pr-agent-pro]

PR Review

PR feedback
⏱️ Estimated effort to review [1-5]	2, because the changes are mostly related to logging enhancements and simplifications across various files. The modifications include adding debug logging, simplifying logging conditions, and improving error handling. These changes are straightforward and mostly involve the logging mechanism, which is not likely to introduce complex issues. However, a thorough review is still necessary to ensure that logging levels are appropriately used and that sensitive information is not being logged.
🧪 Relevant tests	No
🔍 Possible issues	Logging sensitive information: The PR introduces extensive logging, including debug logs that might inadvertently log sensitive information if not carefully implemented. Performance impact: Extensive logging, especially at debug levels, could impact the performance if not properly managed or if the log level is set to a very verbose level in a production environment.
🔒 Security concerns	No

---

✨ Review tool usage guide:

---

Overview:
The review tool scans the PR code changes, and generates a PR review. The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on any PR.
When commenting, to edit configurations related to the review tool (pr_reviewer section), use the following template:

/review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=...

With a configuration file, use the following template:

[pr_reviewer]
some_config1=...
some_config2=...

Utilizing extra instructions The review tool can be configured with extra instructions, which can be used to guide the model to a feedback tailored to the needs of your project. Be specific, clear, and concise in the instructions. With extra instructions, you are the prompter. Specify the relevant sub-tool, and the relevant aspects of the PR that you want to emphasize. Examples for extra instructions: [pr_reviewer] # /review # extra_instructions=""" In the 'possible issues' section, emphasize the following: - Does the code logic cover relevant edge cases? - Is the code logic clear and easy to understand? - Is the code logic efficient? ... """ Use triple quotes to write multi-line instructions. Use bullet points to make the instructions more readable.

How to enable\disable automation When you first install PR-Agent app, the default mode for the review tool is: pr_commands = ["/review", ...] meaning the review tool will run automatically on every PR, with the default configuration. Edit this field to enable/disable the tool, or to change the used configurations

Auto-labels The review tool can auto-generate two specific types of labels for a PR: a possible security issue label, that detects possible security issues (enable_review_labels_security flag) a Review effort [1-5]: x label, where x is the estimated effort to review the PR (enable_review_labels_effort flag)

Extra sub-tools The review tool provides a collection of possible feedbacks about a PR. It is recommended to review the possible options, and choose the ones relevant for your use case. Some of the feature that are disabled by default are quite useful, and should be considered for enabling. For example: require_score_review, require_soc2_ticket, and more.

Auto-approve PRs By invoking: /review auto_approve The tool will automatically approve the PR, and add a comment with the approval. To ensure safety, the auto-approval feature is disabled by default. To enable auto-approval, you need to actively set in a pre-defined configuration file the following: [pr_reviewer] enable_auto_approval = true (this specific flag cannot be set with a command line argument, only in the configuration file, committed to the repository) You can also enable auto-approval only if the PR meets certain requirements, such as that the estimated_review_effort is equal or below a certain threshold, by adjusting the flag: [pr_reviewer] maximal_review_effort = 5

More PR-Agent commands To invoke the PR-Agent, add a comment using one of the following commands: /review: Request a review of your Pull Request. /describe: Update the PR title and description based on the contents of the PR. /improve [--extended]: Suggest code improvements. Extended mode provides a higher quality feedback. /ask <QUESTION>: Ask a question about the PR. /update_changelog: Update the changelog based on the PR's contents. /add_docs 💎: Generate docstring for new components introduced in the PR. /generate_labels 💎: Generate labels for the PR based on the PR's contents. /analyze 💎: Automatically analyzes the PR, and presents changes walkthrough for each component. See the tools guide for more details. To list the possible configuration parameters, add a /config comment.

See the review usage page for a comprehensive guide on using this tool.

[mrT23]

PR Description updated to latest commit (7dcbdff)

[mrT23]

PR Description updated to latest commit (877796b)

[mrT23]

PR Code Suggestions

	Suggestions
bug	Correct the string formatting in the debug log message to include variables.  The debug log message for prompts is missing the formatting for the variables system and user. Use the correct string formatting method to include these variables in the log message. pr_agent/algo/ai_handlers/litellm_ai_handler.py [117] -get_logger().debug(f"Prompts", system=system, user=user) +get_logger().debug(f"Prompts system={system}, user={user}")
enhancement	Remove unnecessary newline characters from the debug log message.             The debug log message for the AI response is using a newline character (\n) which might not be necessary or correctly interpreted in structured logging systems. Consider removing it or ensuring it's compatible with your logging system. pr_agent/algo/ai_handlers/litellm_ai_handler.py [134] -get_logger().debug(f"\nAI response:\n{resp}", full_response=response) +get_logger().debug(f"AI response: {resp}", full_response=response) Add a log message to indicate fallback behavior when an error occurs in getting user labels. The exception handling for getting user labels logs the exception but then returns current_labels without indicating an error occurred. Consider adding a log message to indicate that the default or fallback labels are being used due to an error. pr_agent/algo/utils.py [438-439] get_logger().exception(f"Failed to get user labels: {e}") +get_logger().info("Using default or fallback labels due to error.") return current_labels Convert to structured logging for improved log analysis.                      Use structured logging for better log parsing and analysis. pr_agent/tools/pr_code_suggestions.py [77] -get_logger().debug("Relevant configs", configs=relevant_configs) +get_logger().debug("Relevant configs: %s", relevant_configs) Add exception handling around comment publishing to ensure robustness.        Handle potential exceptions when publishing comments to avoid breaking the execution flow. pr_agent/tools/pr_code_suggestions.py [83] -self.git_provider.publish_comment("Preparing suggestions...", is_temporary=True) +try: + self.git_provider.publish_comment("Preparing suggestions...", is_temporary=True) +except Exception as e: + get_logger().error("Failed to publish temporary comment: %s", e) Add error handling for robust help message generation.                        Include error handling for potential failures in generating the help message. pr_agent/tools/pr_help_message.py [19] -pr_comment += "\n\nHere is a list of tools you can use to interact with the PR Agent:\n" +try: + pr_comment += "\n\nHere is a list of tools you can use to interact with the PR Agent:\n" +except Exception as e: + get_logger().error("Failed to generate PR help message: %s", e) Improve log readability with structured logging.                              Use structured logging for better log management and readability. pr_agent/tools/pr_reviewer.py [120] -get_logger().debug(f"PR output", review=pr_review) +get_logger().debug("PR output", review=pr_review)
best practice	Improve exception handling by re-raising the exception after logging.         The exception handling block in the filtering of diff files logs an error but does not re-raise the exception or handle it, which could lead to silent failures. Consider re-raising the exception or handling it appropriately. pr_agent/algo/pr_processing.py [66] except Exception as e: - pass + raise e Use structured logging instead of string formatting.                          Replace string formatting with structured logging for better log management. pr_agent/tools/pr_questions.py [60] -get_logger().debug(f"PR output", answer=pr_comment) +get_logger().debug("PR output", answer=pr_comment) Use structured logging for variable data.                                     Ensure that logging statements that include variable data use structured logging. pr_agent/tools/pr_questions.py [75] -get_logger().debug(f"PR diff", diff=self.patches_diff) +get_logger().debug("PR diff", diff=self.patches_diff) Enhance log management with structured logging.                               Utilize structured logging for enhanced log management. pr_agent/tools/pr_update_changelog.py [62] -get_logger().debug(f"PR output", changlog=answer) +get_logger().debug("PR output", changlog=answer) Improve log readability with structured logging.                              Use structured logging to improve log readability and management. pr_agent/tools/pr_update_changelog.py [74] -get_logger().debug(f"PR diff", diff=self.patches_diff) +get_logger().debug("PR diff", diff=self.patches_diff)
maintainability	Rename the environment variable for clarity in logging configuration.         The check for the environment variable LOG_SANE to determine the logging format is not intuitive. Consider renaming the environment variable to something more descriptive, like ENABLE_JSON_LOGGING, to improve code readability and configuration clarity. pr_agent/log/init.py [24] -if fmt == LoggingFormat.JSON and os.getenv("LOG_SANE", "0").lower() == "0": +if fmt == LoggingFormat.JSON and os.getenv("ENABLE_JSON_LOGGING", "0").lower() == "0": Temporarily disable semantic file types without altering global settings.     Ensure that disabling semantic file types is reversible in case of subsequent executions. pr_agent/tools/pr_description.py [40] -get_settings().pr_description.enable_semantic_files_types = False +# Temporarily disable semantic file types for this execution only +self.temp_disable_semantic_files_types = True Improve variable naming for clarity.                                          Use a more descriptive variable name for data to enhance code readability. pr_agent/tools/pr_description.py [267] -self.data = data = None +self.description_data = data = None

---

✨ Improve tool usage guide:

---

Overview:
The improve tool scans the PR code changes, and automatically generates suggestions for improving the PR code. The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on a PR.
When commenting, to edit configurations related to the improve tool (pr_code_suggestions section), use the following template:

/improve --pr_code_suggestions.some_config1=... --pr_code_suggestions.some_config2=...

With a configuration file, use the following template:

[pr_code_suggestions]
some_config1=...
some_config2=...

Enabling\disabling automation When you first install the app, the default mode for the improve tool is: pr_commands = ["/improve --pr_code_suggestions.summarize=true", ...] meaning the improve tool will run automatically on every PR, with summarization enabled. Delete this line to disable the tool from running automatically.

Utilizing extra instructions Extra instructions are very important for the improve tool, since they enable to guide the model to suggestions that are more relevant to the specific needs of the project. Be specific, clear, and concise in the instructions. With extra instructions, you are the prompter. Specify relevant aspects that you want the model to focus on. Examples for extra instructions: [pr_code_suggestions] # /improve # extra_instructions=""" Emphasize the following aspects: - Does the code logic cover relevant edge cases? - Is the code logic clear and easy to understand? - Is the code logic efficient? ... """ Use triple quotes to write multi-line instructions. Use bullet points to make the instructions more readable.

A note on code suggestions quality While the current AI for code is getting better and better (GPT-4), it's not flawless. Not all the suggestions will be perfect, and a user should not accept all of them automatically. Suggestions are not meant to be simplistic. Instead, they aim to give deep feedback and raise questions, ideas and thoughts to the user, who can then use his judgment, experience, and understanding of the code base. Recommended to use the 'extra_instructions' field to guide the model to suggestions that are more relevant to the specific needs of the project, or use the custom suggestions 💎 tool With large PRs, best quality will be obtained by using 'improve --extended' mode.

More PR-Agent commands To invoke the PR-Agent, add a comment using one of the following commands: /review: Request a review of your Pull Request. /describe: Update the PR title and description based on the contents of the PR. /improve [--extended]: Suggest code improvements. Extended mode provides a higher quality feedback. /ask <QUESTION>: Ask a question about the PR. /update_changelog: Update the changelog based on the PR's contents. /add_docs 💎: Generate docstring for new components introduced in the PR. /generate_labels 💎: Generate labels for the PR based on the PR's contents. /analyze 💎: Automatically analyzes the PR, and presents changes walkthrough for each component. See the tools guide for more details. To list the possible configuration parameters, add a /config comment.

See the improve usage page for a more comprehensive guide on using this tool.