Skip to content

Add guardrail tracing #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 2, 2025
Merged

Add guardrail tracing #9

merged 1 commit into from
May 2, 2025

Conversation

KyMidd
Copy link
Owner

@KyMidd KyMidd commented May 2, 2025

Human notes

  • Add guardrail tracing option, surfaces why guardrail blocked things to the user in the slack response
  • Fix system prompt injection

Robot notes beep boop

This pull request introduces several updates to the python/devopsbot.py file to enhance functionality, improve error handling, and refine AI interaction. Additionally, there are changes to the Terraform configuration in lambda_receiver.tf to standardize IAM role naming. Below is a categorized summary of the most important changes.

Enhancements to AI Interaction and Guardrails

  • Introduced guardrailTracing to enable tracing options for guardrails (enabled, enabled_full, disabled) and included it in the AI request payload (def ai_request) to enhance security and debugging capabilities. [1] [2]
  • Replaced model_guidance with system_prompt and updated the initialization process to use initial_model_system_prompt for better clarity and maintainability. [1] [2]
  • Enhanced Slack responses to include detailed guardrail blocking information, such as type, confidence, and filter strength, when content is flagged by Veradigm's content filter.

Error Handling Improvements

  • Added exception handling in ask_bedrock_llm_with_knowledge_base to catch and log errors, including specific handling for Aurora auto-pausing scenarios, with user-friendly Slack messages prompting retries. [1] [2]
  • Added debug logging for raw and structured knowledge base responses when VERA_DEBUG is enabled.

Refactoring for Consistency

  • Renamed url fields to source in knowledge base responses to better represent diverse data sources (e.g., Confluence, S3). Updated all related references in rerank_text and handle_message_event. [1] [2] [3]
  • Updated AI request conversation structure to append assistant responses instead of user responses for knowledge base citations. [1] [2]

Terraform Configuration Updates

  • Standardized IAM role naming in lambda_receiver.tf by renaming Ue1TiDevOpsBotReceiverRole to DevOpsBotReceiverRole across all related resources for improved clarity and consistency. [1] [2] [3] [4]

These changes collectively improve the bot's robustness, maintainability, and user experience while aligning infrastructure naming conventions.

@KyMidd KyMidd merged commit d55ff87 into master May 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant