Team_CNCA project repository for Case Study 15 - CSV - Computer System Validation or Revalidation .
https://github.com/AI-Agent-Incubator-Month/Team_CNCA/tree/master
Team_CNCA
This project was developed by a dedicated team of contributors:
- Hon Lee (hon.lee@capgemini.com): Solution Architect / Lead Developer
- Malay Dhang (malay.dhang@capgemini.com): Main Developer
- Stew Wareing (stew.wareing@capgemini.com): Developer
- Alec Yates (alec-robert.yates@capgemini.com): Developer
- George Sambrook (george.sambrook@capgemini.com): Developer
Welcome to our project submission for Team_CNCA Aspire AI Computer System Validation Chat! This repository contains the code and documentation for our innovative solution, this is on repo https://github.com/AI-Agent-Incubator-Month/Team_CNCA/tree/master. Our project aims to:
-
Reduce Computer System Validation Planning time and scope by 25%+ using AI. This means that less rigorous testing and documentation might be required for systems that pose minimal risk to product quality or patient safety.
-
Develop a comprehensive semi automated AI solution to assist in the planning process, particularly in validating planning applications for CSV, but still requires human input and review. This approach leverages AI and machine learning to identify potential errors or omissions in applications, freeing up planning officers to focus on more complex cases and strategic decision-making.
-
Provide GAMP 5 compliance and ISO 13485 establishing a systematic approach to ensure medical devices are consistently produced to meet defined specifications and safety requirements within the validation planning document.
-
Conduct using risk assessment and stakeholder alignment inputs using multiagents to highlight any key pointers that the document is missing on or needs updating.
-
Provide comprehensive repoting to provide multiagant feedback in the form of PDF reports to help analyse the thought process between the AI agents.
-
DEMO EXPLANATION VIDEO - https://www.loom.com/share/4624e8adf59142acb3ea9df968c4d6b6
Our project leverages Azure Aspire to create a AI multiagent powered Computer System Validation planner tool and chat system. The core functionality includes:
- Validation Planning Document Analysis: [This does analyisis and infers the document intelligence service to give context on what areas the design plan has good coverage for a medical design validation plan=.]
- Validation Planning Document Feedback and Reporting: [This generates a reporting document giving logging and feedback from each of the multiagents called]
- Documentation upload to ingest data for further AI Knowledge: [This allows future documentation to be inferred in the web app either as manual upload or background service for further Semantic Kernel AI knowledge.]
- General Summarisation of documentation: [The web application allows a human friendly chat system interface for searching and summarising information with uploaded documents].
The primary intent of our project is to allow semi automated stepped approach within a multiagent process feedback loop for CSV validation planning. We aim to have a creation of a Validation Plan Document for medical equipment using OpenAI ‘gpt-4o’ model that involves leveraging AI-driven natural language generation and document structuring capabilities to develop a compliant, structured, and traceable document that outlines the strategy for verifying that medical equipment meets its intended use, user needs, and regulatory requirements. Application uses a multi-agent AI framework to intelligently manage and streamline the validation planning process using Azure AI Services.
The application leverages advanced AI capabilities to simplify and automate the validation planning process for medical equipment in accordance with Good Manufacturing Practice (GMP) and regulatory standards. Bing Search plugin id used to access external knowledge beyond the LLM's native training data. It is a hybrid intelligence approach that combines internal reasoning with real-time external research.
Assigned Industry Use Case: [Computer System Validation and Revalidation - Good Manufacturing Practices (GMP) Compliance] (https://ai-idea.azurewebsites.net/subdomain-ideas/Life%20Sciences/Manufacturing%20and%20Supply%20Chain)
Our solution is designed to be used by Computer System Validation engineers for a manuafacturing system. It can be applied in scenarios such as:
- Scenario 1 - Validation Planning Document Analysis: [This does analyisis and infers the document intelligence service to give context on what areas the design plan has good coverage for a medical design validation plan and highlights areas that need to improve for coverage such as traceability or versioning sections.]
- Scenario 2 - Validation Planning Document Feedback and Reporting: [This generates a reporting document giving logging and feedback from each of the multiagents called]
- Scenario 3 - Documentation upload to ingest data for further AI Knowledge: [This allows future documentation to be inferred in the web app either as manual upload or background service for further Semantic Kernel AI knowledge for past medical design plan documents or complaince documents checks.]
- Scenario 4 - General Summarisation of documentation: [The web application allows a human friendly chat system interface for searching and summarising information with uploaded documents related to the design plans or any related part of the system to CSV].
![Screenshot 1]!
![Screenshot 2]!
![Screenshot 3]!
![Screenshot 4]!
![Screenshot 5]!
![Screenshot 6]!
![Screenshot 7]!
![Screenshot 8]!
![Screenshot 9]!
![Screenshot 10]!
![Screenshot 11]!
![Screenshot 12]!
![Screenshot 13]!
![Screenshot 14]!
![Screenshot 15]!
![Screenshot 16]
![Screenshot 17]!
![Screenshot 18]!
See the document LLM document comparison 1.docx for more detailed information with the evaluation results on how to write the prompt commands into the chat system to call the multiagents, with detailed screenshots. Presentation ![Screenshot 1](path/to/screenshotreenshot2.png
Before you run the application from the project AspireAIAgentsCSVSChat.AppHost, you need to configure within this project the user secrets in Manage User Secrets by right clicking and typing inside the JSON file: -
{ "ConnectionStrings:openai": "Endpoint=YOUR ENDPOINT/;Key=YOUR KEY", "ConnectionStrings:docintelligence": "Endpoint=YOURENDPOINT", "ConnectionStrings:bing": "YOURBINGKING" }
Here is a detailed explanation of how the project was implemented, including any technical details, architecture, and design decisions. The computer system validation system that uses a pdf document input for medical design plan (pdf) which goes to a chat system as ingested document upload data, this can call a validation planner agent, risk assessment agent, a stakeholder alignment agent and finally goes to an ongoing review agent, it uses semantic kernel utilising azure open ai (chatgpt 4o). This outputs a report pdf and llm output into the chat. As storage, it uses local vectorised json files for storage for data ingestion that is fed into as knowledge base into the semantic kenrel for all the multiagents. It uses azure document intelligence service for grounding for the initial document analysis for the design plan document noting section and confidence levels. The overall framework is using Azure Aspire as error reporting and Open Telemetry report logging is much easier using this framework and allows distributed system to be developed much quicker from ground up especially with API management, web service / microservice and databases included. The UI framework used is Blazor for a user friendly front end with plugins that include Bing Search to refine the multiagents when searching additional infromation and Azure Document Intelligence for grounding.
Here is infromation about each of the multiagents within the @process plugin function for the Medical Device Design Plan.
Validation Planner Agent ensures that the medical design plan adheres to regulatory standards and validation protocols. 📌 Receives PDF medical design plan → Extracts validation criteria → Plans validation steps → Sends structured validation reports to the chat system.
Risk Assessment Agent identifies, categorizes, and mitigates potential hazards that could compromise patient safety or regulatory compliance. 📌 Extracts risk factors from design plan → Identifies gaps → Performs risk classification → Sends risk mitigation strategies to the chat system for validation.
Stakeholder Alignment Agent ensures that validation processes incorporate input from regulatory teams, engineers, quality assurance specialists, and medical professionals. 📌 Aggregates stakeholder requirements → Aligns validation tasks → Ensures traceability across approvals → Sends stakeholder consensus reports to the chat system.
Ongoing Review Agent continuously monitors post-market performance, defect tracking, and compliance updates for the medical device. 📌 Monitors post-validation updates → Identifies non-conformities → Automates defect correction strategies → Sends compliance revision summaries to the chat system.
We believe the project has significant potential however with the limited time and various other constraints we have developed "streamlining medical equipment validation using AI" or "enhancing compliance with real-time regulatory checks". Future plans was to use Azure AI Foundry when given more time, to do some model switch and context switching that is saved and archvied on the cloud against the multiagents and to use evaluation tools for the grounding. Effectively the team had built their own localised Azure AI Foundry as it stands. Also to use the new services with AI proccess documenting like the new Azure AI Content Understanding API (https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/). It can use for Azure CosmosDB for larger scaling process and document storage.
Please feel free to review the presentation, and We would be happy to discuss it further if you have any questions or suggestions for future plans or further development.