feat: Add JSON-aware mermaid diagram validation

[buger]

Summary

Fixes the issue where mermaid validation couldn't detect diagrams embedded in JSON string values. Previously, the regex pattern only matched literal markdown code blocks, missing diagrams with escaped newlines (\n) in JSON responses.

Problem

The mermaid validation used a pattern /```mermaid([^\n]*)\n([\s\S]*?)```/gi that only matched literal code blocks, not escaped ones inside JSON strings:

Before (not detected):

{
  "text": "```mermaid\ngraph TD\n  A --> B\n```"
}

After (detected and validated):

{
  "text": "```mermaid\ngraph TD\n  A --> B\n```"
}

Changes

Core Functionality

• extractMermaidFromJson() - New function to detect mermaid diagrams in JSON string values

  • Parses JSON and recursively searches all string properties
  • Handles both escaped (\n) and literal newlines
  • Tracks JSON path for each diagram (e.g., data.visualization)

• extractMermaidFromMarkdown() - Enhanced to auto-detect JSON responses

  • Falls back to JSON-aware extraction when JSON is detected
  • Maintains backward compatibility with markdown extraction

• replaceMermaidDiagramsInJson() - Properly replaces diagrams in JSON

  • Navigates JSON structure using paths
  • Re-escapes content for valid JSON output
  • Preserves code block wrappers if present

• replaceMermaidDiagramsInMarkdown() - Auto-detects format

  • Routes to JSON or markdown replacement as needed

Test Coverage

Added comprehensive tests in mermaidValidation.test.js:

• ✅ Extract mermaid from JSON string values
• ✅ Extract from JSON in code blocks
• ✅ Extract multiple diagrams from JSON
• ✅ Extract from nested JSON objects and arrays
• ✅ Handle escaped newlines correctly
• ✅ Replace diagrams in JSON while preserving structure
• ✅ Automatic JSON vs markdown detection

Updated validationFlow.test.js to reflect new behavior where diagrams in JSON are now correctly detected.

Testing

All tests pass:

• 323 mermaid/schema tests ✅
• 1,118 total tests ✅
• No regressions

Backward Compatibility

✅ Fully backward compatible - existing markdown mermaid validation continues to work as before. This enhancement adds JSON support without changing existing behavior.

🤖 Generated with Claude Code

[probelabs]

🔍 Code Analysis Results

PR Overview: JSON-aware Mermaid Diagram Validation

Summary

This PR introduces JSON-aware mermaid diagram validation to fix a critical gap where mermaid diagrams embedded in JSON string values were not being detected or validated. The previous regex-based approach only matched literal markdown code blocks, missing diagrams with escaped newlines ( ) in JSON responses.

Files Changed Analysis

3 files changed with 428 additions, 11 deletions:

• npm/src/agent/schemaUtils.js: +170/-8 (core functionality)
• npm/tests/unit/mermaidValidation.test.js: +253/-1 (comprehensive test coverage)
• npm/tests/integration/validationFlow.test.js: +5/-2 (integration test updates)

Architecture & Impact Assessment

What This PR Accomplishes

• Fixes validation gap: Mermaid diagrams in JSON strings are now properly detected and validated
• Maintains backward compatibility: Existing markdown mermaid validation continues unchanged
• Enhances auto-fixing: JSON-embedded diagrams can be automatically corrected using existing maid + AI pipeline

Key Technical Changes

1. extractMermaidFromJson() - New core function:

   • Parses JSON and recursively searches all string properties
   • Handles both escaped ( ) and literal newlines in mermaid patterns
   • Tracks JSON paths for precise diagram location (e.g., data.visualization)
   • Returns structured data with diagrams, paths, and parsed JSON

2. Enhanced extractMermaidFromMarkdown():

   • Auto-detects JSON responses and routes to JSON-aware extraction
   • Falls back to traditional markdown extraction for non-JSON content
   • Maintains full backward compatibility

3. replaceMermaidDiagramsInJson() - Smart replacement:

   • Navigates JSON structure using recorded paths
   • Properly re-escapes content for valid JSON output
   • Preserves code block wrappers when present
   • Handles both plain JSON and code-block-wrapped JSON

4. Enhanced replaceMermaidDiagramsInMarkdown():

   • Auto-detects format and routes to appropriate replacement logic
   • Seamless integration with existing validation pipeline

System Components Affected

graph TD
    A[AI Response] --> B{Format Detection}
    B -->|JSON| C[extractMermaidFromJson]
    B -->|Markdown| D[extractMermaidFromMarkdown]
    C --> E[validateMermaidResponse]
    D --> E
    E --> F{Valid?}
    F -->|No| G[maid auto-fix]
    G --> H{Fixed?}
    H -->|No| I[MermaidFixingAgent]
    H -->|Yes| J[replaceMermaidDiagramsInJson]
    I --> K[replaceMermaidDiagramsInJson]
    J --> L[Fixed Response]
    K --> L
    F -->|Yes| L

Loading

Integration Points:

• validateAndFixMermaidResponse() - Uses enhanced extraction
• ProbeAgent.answer() - Schema processing pipeline unchanged
• MermaidFixingAgent - Works with JSON-extracted diagrams
• Telemetry system - Tracks JSON-aware validation events

Scope Discovery & Context Expansion

Immediate Impact

• Schema validation: JSON responses with mermaid diagrams now get proper validation
• Auto-correction: Both maid auto-fix and AI fixing work with JSON-embedded diagrams
• Error reporting: Enhanced error messages include JSON path context

Broader System Integration

• MCP Server: Inherits JSON-aware validation through shared schemaUtils
• CLI tools: Automatic mermaid validation now covers JSON responses
• Telemetry: New events track JSON-specific validation metrics
• Testing: Comprehensive test suite ensures reliability

Related Components to Consider

• JSON Schema validation: Works seamlessly with enhanced mermaid validation
• Response cleaning: Preserves JSON structure during mermaid extraction
• Error handling: Graceful fallback when JSON parsing fails
• Performance: Minimal overhead - JSON parsing only when needed

Testing Coverage

New test scenarios in mermaidValidation.test.js:

• ✅ Extract mermaid from JSON string values with escaped newlines
• ✅ Extract from JSON in markdown code blocks
• ✅ Handle multiple diagrams in single JSON response
• ✅ Navigate nested JSON objects and arrays
• ✅ Proper newline unescaping in diagram content
• ✅ Replace diagrams while preserving JSON structure
• ✅ Auto-detection between JSON vs markdown formats

Integration test updates:

• Updated validationFlow.test.js to reflect new behavior where JSON-embedded diagrams are now correctly detected and validated

Backward Compatibility

✅ Fully backward compatible - All existing functionality preserved:

• Traditional markdown mermaid validation unchanged
• Existing API contracts maintained
• No breaking changes to function signatures
• Enhanced features are additive only

Performance & Reliability

• Efficient detection: JSON parsing only triggered when response appears to be JSON
• Robust error handling: Graceful degradation when JSON parsing fails
• Memory conscious: Streamlined processing with minimal object creation
• Test coverage: 253 new test lines ensure reliability

This enhancement significantly improves the robustness of mermaid diagram validation while maintaining the existing architecture and performance characteristics.

Metadata

• Review Effort: 3 / 5
• Primary Label: feature

🐛 Debug Information

Provider: anthropic
Model: glm-4.6
API Key Source: ANTHROPIC_API_KEY
Processing Time: 204590ms
Timestamp: 2025-11-16T17:31:16.093Z
Prompt Length: 48878 characters
Response Length: 10794 characters
JSON Parse Success: ✅

Debug Details

⚠️ Debug information is too large for GitHub comments.
📁 Full debug information saved to artifact: visor-debug-2025-11-16T17-31-18-664Z.md

🔗 Download Link: visor-debug-595
💡 Go to the GitHub Action run above and download the debug artifact to view complete prompts and responses.

---

Powered by Visor from Probelabs

Last updated: 2025-11-16T17:31:18.839Z | Triggered by: opened | Commit: 5766baa

💡 TIP: You can chat with Visor using /visor ask <your question>

[probelabs]

🔍 Code Analysis Results

Security Issues (5)

Severity	Location	Issue
🟠 Error	npm/src/agent/schemaUtils.js:855	JSON.parse() called without input validation or size limits, vulnerable to memory exhaustion and prototype pollution attacks 💡 Suggestion Add input validation, size limits, and prototype pollution protection before JSON.parse() 🔧 Suggested Fix // Add size limit check (e.g., 10MB) if (jsonContent.length > 10 * 1024 * 1024) { return { diagrams: [], jsonPaths: [], parsedJson: null }; } // Sanitize JSON to prevent prototype pollution const sanitizedJson = jsonContent.replace(/"proto"|"constructor"|"prototype"/g, '"_$1"'); let parsedJson; try { parsedJson = JSON.parse(sanitizedJson); } catch (e) { return { diagrams: [], jsonPaths: [], parsedJson: null }; }
🟠 Error	npm/src/agent/schemaUtils.js:874	Complex regex pattern vulnerable to ReDoS attacks with crafted input containing nested backticks or mermaid blocks 💡 Suggestion Add timeout protection and simplify regex pattern to prevent catastrophic backtracking 🔧 Suggested Fix // Add timeout protection for regex execution const REGEX_TIMEOUT = 5000; // 5 seconds const startTime = Date.now(); const mermaidPattern = /```mermaid([^ `]*?)(?: |\ )([^`]*?)```/gi; let match; while ((match = mermaidPattern.exec(obj)) !== null) { if (Date.now() - startTime > REGEX_TIMEOUT) { console.warn('Regex execution timeout, stopping mermaid extraction'); break; }
🟠 Error	npm/src/agent/schemaUtils.js:975	JSON path navigation vulnerable to prototype pollution through malicious jsonPath values like '__proto__' or 'constructor.prototype' 💡 Suggestion Validate and sanitize jsonPath before navigation to prevent prototype pollution 🔧 Suggested Fix // Validate and sanitize path parts const pathParts = diagram.jsonPath.split('.').filter(p => p && !p.match(/^__(proto__|constructor)$/)); if (pathParts.length === 0) { continue; } let current = modifiedJson; for (let i = 0; i < pathParts.length - 1; i++) { const part = pathParts[i]; // Additional validation for array indices if (part.startsWith('[') && part.endsWith(']')) { const index = parseInt(part.slice(1, -1), 10); if (isNaN(index) || index < 0 || !Array.isArray(current) || index >= current.length) { continue; // Skip invalid array access } current = current[index]; } else { // Prevent prototype pollution if (part === 'proto' || part === 'constructor' || part === 'prototype') { continue; } if (!current || typeof current !== 'object') { continue; } current = current[part]; } }
🟡 Warning	npm/src/agent/schemaUtils.js:981	Array index access without bounds checking could throw exceptions or access unintended memory 💡 Suggestion Add bounds checking for array index access 🔧 Suggested Fix if (part.startsWith('[') && part.endsWith(']')) { const index = parseInt(part.slice(1, -1), 10); if (!Array.isArray(current) || index < 0 || index >= current.length) { continue; // Skip invalid array access } current = current[index]; }
🟡 Warning	npm/src/agent/schemaUtils.js:1002	Array index replacement without bounds checking in the final replacement step 💡 Suggestion Add bounds checking for array index replacement 🔧 Suggested Fix if (lastPart.startsWith('[') && lastPart.endsWith(']')) { const index = parseInt(lastPart.slice(1, -1), 10); if (!Array.isArray(current) || index < 0 || index >= current.length) { continue; // Skip invalid array access } const originalString = current[index]; current[index] = originalString.replace(diagram.fullMatch, newCodeBlock); }

Architecture Issues (1)

Severity	Location	Issue
🟢 Info	AI_RESPONSE:1	{ "issues": [ { "file": "npm/src/agent/schemaUtils.js", "line": 979, "endLine": 987, "ruleId": "architecture/error-handling", "message": "JSON path navigation lacks bounds checking and error handling for invalid paths", "severity": "error", "category": "architecture", "suggestion": "Add proper error handling for array index bounds and property existence checks during JSON path navigation", "replacement": "for (let i = 0; i < pathParts.length - 1; i++) { const part = pathParts[i]; if (part.startsWith('[') && part.endsWith(']')) { const index = parseInt(part.slice(1, -1), 10); if (!Array.isArray(current) || index < 0 || index >= current.length) { console.warn(`Invalid array index ${index} at path ${diagram.jsonPath}`); continue; // Skip this diagram } current = current[index]; } else { if (!current || typeof current !== 'object' || !(part in current)) { console.warn(`Invalid property '${part}' at path ${diagram.jsonPath}`); continue; // Skip this diagram } current = current[part]; } }" }, { "file": "npm/src/agent/schemaUtils.js", "line": 994, "endLine": 1003, "ruleId": "architecture/error-handling", "message": "Final path segment replacement lacks bounds checking and could throw runtime errors", "severity": "error", "category": "architecture", "suggestion": "Add validation for the final path segment before attempting replacement", "replacement": "if (lastPart.startsWith('[') &amp;&amp; lastPart.endsWith(']')) { const index = parseInt(lastPart.slice(1, -1), 10); if (!Array.isArray(current) || index &lt; 0 || index &gt;= current.length) { console.warn(`Invalid array index ${index} for final path segment ${diagram.jsonPath}`); continue; } const originalString = current[index]; if (typeof originalString !== 'string') { console.warn(`Target at ${diagram.jsonPath} is not a string`); continue; } current[index] = originalString.replace(diagram.fullMatch, newCodeBlock); } else { if (!current || typeof current !== 'object' || !(lastPart in current)) { console.warn(`Invalid property '${lastPart}' for final path segment ${diagram.jsonPath}`); continue; } const originalString = current[lastPart]; if (typeof originalString !== 'string') { console.warn(`Target at ${diagram.jsonPath} is not a string`); continue; } current[lastPart] = originalString.replace(diagram.fullMatch, newCodeBlock); }" }, { "file": "npm/src/agent/schemaUtils.js", "line": 868, "ruleId": "architecture/regex-complexity", "message": "Complex regex pattern for mermaid extraction could be simplified and made more maintainable", "severity": "warning", "category": "architecture", "suggestion": "Break down the complex regex into simpler, more focused patterns for better maintainability", "replacement": "// Handle both escaped (\ ) and literal newlines const mermaidPattern = /```mermaid ([^\ `]*?)(?:\ |\\\ )([\\s\\S]*?)```/gi; // Consider extracting this to a constant for reuse and testing" The content you provided is not a Mermaid diagram. It appears to be JSON data or code showing a linting rule violation in a JavaScript file (npm/src/agent/schemaUtils.js) with details about an architecture issue and suggested code replacement. To fix a Mermaid diagram, please provide the actual Mermaid diagram code that starts with one of these valid diagram types: graph or flowchart pie sequenceDiagram classDiagram stateDiagram-v2 gantt erDiagram journey gitgraph requirementDiagram C4Context Please share the actual Mermaid diagram code that needs syntax correction. }&#34; } ] }</div></td> </tr> </tbody> </table> ### Performance Issues (5) <table> <thead> <tr> <th>Severity</th> <th>Location</th> <th>Issue</th> </tr> </thead> <tbody> <tr> <td>🟡 Warning</td> <td><code>npm/src/agent/schemaUtils.js:858</code></td> <td><div>The regex pattern /```mermaid([^ `]*?)(?: |\ )([\s\S]*?)```/gi has potential for catastrophic backtracking with complex nested structures <details><summary>💡 <strong>Suggestion</strong></summary>Consider using a more specific pattern or implementing a state machine parser for better performance with large JSON strings </details> </div></td> </tr> <tr> <td>🟡 Warning</td> <td><code>npm/src/agent/schemaUtils.js:874</code></td> <td><div>Recursive searchObject function could cause stack overflow on deeply nested JSON structures <details><summary>💡 <strong>Suggestion</strong></summary>Replace recursion with iterative approach using a stack to handle arbitrarily deep JSON structures safely </details> </div></td> </tr> <tr> <td>🟡 Warning</td> <td><code>npm/src/agent/schemaUtils.js:950</code></td> <td><div>JSON.stringify() with 2-space indentation creates unnecessary overhead for large JSON objects <details><summary>💡 <strong>Suggestion</strong></summary>Use JSON.stringify() without spacing or implement custom formatting only when needed for display purposes </details> </div></td> </tr> <tr> <td>🟡 Warning</td> <td><code>npm/src/agent/schemaUtils.js:970</code></td> <td><div>Multiple regex operations on the same string for code block detection could be optimized <details><summary>💡 <strong>Suggestion</strong></summary>Combine regex patterns or cache results to avoid redundant string processing </details> </div></td> </tr> <tr> <td>🟡 Warning</td> <td><code>npm/src/agent/schemaUtils.js:1045</code></td> <td><div>Sorting diagrams by start index for every replacement operation is inefficient <details><summary>💡 <strong>Suggestion</strong></summary>Sort once and reuse the sorted order, or process in reverse order without explicit sorting </details> </div></td> </tr> </tbody> </table> ### Quality Issues (6) <table> <thead> <tr> <th>Severity</th> <th>Location</th> <th>Issue</th> </tr> </thead> <tbody> <tr> <td>🟠 Error</td> <td><code>npm/src/agent/schemaUtils.js:997</code></td> <td><div>String replacement logic assumes fullMatch has unescaped newlines, but extraction creates fullMatch with escaped newlines from JSON strings <details><summary>💡 <strong>Suggestion</strong></summary>Fix the string replacement to handle the mismatch between escaped newlines in fullMatch and the target string. Either normalize both to use the same newline format or use a more robust replacement method. </details> <details><summary>🔧 <strong>Suggested Fix</strong></summary><pre><code>const originalString = current[index]; // Normalize both strings to use escaped newlines for matching const normalizedFullMatch = diagram.fullMatch.replace(/ /g, &#39;\ &#39;); const normalizedNewCodeBlock = newCodeBlock.replace(/ /g, &#39;\ &#39;); current[index] = originalString.replace(normalizedFullMatch, normalizedNewCodeBlock);</code></pre> </details> </div></td> </tr> <tr> <td>🟠 Error</td> <td><code>npm/src/agent/schemaUtils.js:1001</code></td> <td><div>Same string replacement issue for object properties - fullMatch has unescaped newlines but target string has escaped newlines <details><summary>💡 <strong>Suggestion</strong></summary>Apply the same fix for object property replacement as for array elements </details> <details><summary>🔧 <strong>Suggested Fix</strong></summary><pre><code>const originalString = current[lastPart]; // Normalize both strings to use escaped newlines for matching const normalizedFullMatch = diagram.fullMatch.replace(/ /g, &#39;\ &#39;); const normalizedNewCodeBlock = newCodeBlock.replace(/ /g, &#39;\ &#39;); current[lastPart] = originalString.replace(normalizedFullMatch, normalizedNewCodeBlock);</code></pre> </details> </div></td> </tr> <tr> <td>🟡 Warning</td> <td><code>npm/tests/unit/mermaidValidation.test.js:604</code></td> <td><div>Test uses hardcoded fullMatch string that may not match actual extraction output format <details><summary>💡 <strong>Suggestion</strong></summary>Derive the expected fullMatch from the actual test data or use a more flexible matching approach that doesn&#39;t depend on exact string format </details> <details><summary>🔧 <strong>Suggested Fix</strong></summary><pre><code>// Extract the actual fullMatch from the original response const originalResult = extractMermaidFromJson(originalResponse); const correctedDiagrams = [{ content: &#39;graph TD A --&gt; B B --&gt; C&#39;, fullMatch: originalResult.diagrams[0].fullMatch, isInJson: true, jsonPath: &#39;text&#39;, attributes: &#39;&#39; }];</code></pre> </details> </div></td> </tr> <tr> <td>🟡 Warning</td> <td><code>npm/tests/unit/mermaidValidation.test.js:630</code></td> <td><div>Test uses hardcoded fullMatch that may not match actual extraction format <details><summary>💡 <strong>Suggestion</strong></summary>Use the actual extracted fullMatch instead of hardcoded value </details> <details><summary>🔧 <strong>Suggested Fix</strong></summary><pre><code>// Extract the actual fullMatch from the original response const originalResult = extractMermaidFromJson(originalResponse); const correctedDiagrams = [{ content: &#39;graph TD A --&gt; B --&gt; C&#39;, fullMatch: originalResult.diagrams[0].fullMatch, isInJson: true, jsonPath: &#39;text&#39;, attributes: &#39;&#39; }];</code></pre> </details> </div></td> </tr> <tr> <td>🟡 Warning</td> <td><code>npm/tests/unit/mermaidValidation.test.js:652</code></td> <td><div>Test uses hardcoded fullMatch that may not match actual extraction format <details><summary>💡 <strong>Suggestion</strong></summary>Use the actual extracted fullMatch instead of hardcoded value </details> <details><summary>🔧 <strong>Suggested Fix</strong></summary><pre><code>// Extract the actual fullMatch from the original response const originalResult = extractMermaidFromJson(originalResponse); const correctedDiagrams = [{ content: &#39;graph LR X --&gt; Y --&gt; Z&#39;, fullMatch: originalResult.diagrams[0].fullMatch, isInJson: true, jsonPath: &#39;diagram&#39;, attributes: &#39;&#39; }];</code></pre> </details> </div></td> </tr> <tr> <td>🟡 Warning</td> <td><code>npm/tests/integration/validationFlow.test.js:49</code></td> <td><div>Test assumes new behavior (isValid: true) without verifying the implementation actually works correctly <details><summary>💡 <strong>Suggestion</strong></summary>Add more comprehensive assertions to verify the JSON-aware extraction is working correctly, not just that it returns true </details> <details><summary>🔧 <strong>Suggested Fix</strong></summary><pre><code>expect(mermaidValidation.isValid).toBe(true); // Mermaid diagrams in JSON strings are now detected and validated expect(mermaidValidation.diagrams).toHaveLength(1); expect(mermaidValidation.diagrams[0].isInJson).toBe(true); expect(mermaidValidation.diagrams[0].diagramType).toBe(&#39;flowchart&#39;); // Verify the diagram content was properly extracted from JSON expect(mermaidValidation.diagrams[0].content).toContain(&#39;graph TD&#39;); expect(mermaidValidation.diagrams[0].jsonPath).toBeDefined();</code></pre> </details> </div></td> </tr> </tbody> </table> <details> <summary>🐛 Debug Information</summary> **Provider:** anthropic **Model:** glm-4.6 **API Key Source:** ANTHROPIC_API_KEY **Processing Time:** 204590ms **Timestamp:** 2025-11-16T17:31:16.093Z **Prompt Length:** 48878 characters **Response Length:** 10794 characters **JSON Parse Success:** ✅ ### Debug Details ⚠️ Debug information is too large for GitHub comments. 📁 **Full debug information saved to artifact:** `visor-debug-2025-11-16T17-31-19-408Z.md` 🔗 **Download Link:** [visor-debug-595](https://github.com/probelabs/probe/actions/runs/19409346874) 💡 Go to the GitHub Action run above and download the debug artifact to view complete prompts and responses. </details> --- *Powered by [Visor](https://probelabs.com/visor) from [Probelabs](https://probelabs.com)* *Last updated: 2025-11-16T17:31:19.640Z | Triggered by: opened | Commit: 5766baa* 💡 **TIP:** You can chat with Visor using `/visor ask <your question>` <!-- /visor-comment-id:pr-review-295-review -->