Skip to content

TRCLI-253: Upload Test Result support for AI Evaluation Template#415

Merged
acuanico-tr-galt merged 4 commits into
release/1.14.2from
feature/TRCLI-253
May 1, 2026
Merged

TRCLI-253: Upload Test Result support for AI Evaluation Template#415
acuanico-tr-galt merged 4 commits into
release/1.14.2from
feature/TRCLI-253

Conversation

@acuanico-tr-galt
Copy link
Copy Markdown
Collaborator

@acuanico-tr-galt acuanico-tr-galt commented Apr 23, 2026

Solution description

Added support for TestRail's AI Evaluation Template allowing users to upload test results with multi-dimensional quality ratings (0-5 stars across up to 15 categories like factual_accuracy, security, reliability, etc.) and AI-specific context fields (ai_input, ai_output, ai_traces, ai_latency). Quality ratings are parsed from JUnit XML properties and Robot Framework metadata, validated, and uploaded at the result root level per TestRail API requirements.

Changes

  • New Parser Module: trcli/data_classes/quality_rating_parser.py - Validates and parses quality ratings from JSON strings with comprehensive error handling
  • Data Class Updates: Added optional quality_rating field to TestRailResult with serde skip_if_default=True for backward compatibility
  • JUnit Parser Enhancement: Extended junit_xml.py to extract and validate quality_rating from test case properties
  • Robot Parser Enhancement: Extended robot_xml.py to extract and validate quality ratings from test metadata

Potential impacts

None. All changes are backwards compatible.

Steps to test

Please see updated README file 488-578

PR Tasks

  • PR reference added to issue
  • README updated
  • Unit tests added/updated

@acuanico-tr-galt acuanico-tr-galt merged commit d7608c3 into release/1.14.2 May 1, 2026
3 checks passed
@acuanico-tr-galt acuanico-tr-galt deleted the feature/TRCLI-253 branch May 1, 2026 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant