A Model-Agnostic, Input-Agnostic, and Output-Agnostic Metamorphic Testing Framework
for Regression-Based Autonomous Driving AI/ML Models.

AutoMR is a metamorphic testing framework designed to evaluate the robustness and reliability of AI/ML models without requiring ground-truth labels.

Instead of checking whether predictions exactly match expected outputs, AutoMR verifies whether a model behaves consistently under controlled transformations that should preserve expected behavior.

The framework automatically applies transformations, validates metamorphic relations, analyzes failures, and generates comprehensive reports — all with zero boilerplate.

• Model-Agnostic Testing — works with TensorFlow, Keras, PyTorch, scikit-learn, XGBoost, or any custom model
• Input-Agnostic Architecture — supports images, time-series, sequential, and tabular data
• Output-Agnostic Validation — handles regression, continuous, and numerical outputs
• Built-in Metamorphic Relations — 11 ready-to-use relations covering weather, geometric, behavioral, and temporal properties
• Automated Transformation Pipeline — end-to-end execution with configurable parameters
• Parameter Range Testing — sweep transformation parameters across configurable ranges
• Epsilon Sensitivity Analysis — automatically evaluates model robustness across multiple epsilon thresholds
• Automatic Epsilon Recommendation — identifies first failure, stabilization, and recommended epsilon values
• Interactive Live Dashboard — real-time webcam/video testing with configurable metamorphic relations and epsilon
• Failure Detection and Localization — pinpoints the exact conditions where models break
• Severity Analysis — ranks failures by output deviation magnitude
• Failure Region Identification — isolates parameter ranges with highest instability
• Worst-Case Sample Discovery — surfaces samples with the largest prediction deviations
• CSV, JSON, and Text Report Generation — comprehensive reporting with reproducible outputs
• Verification Artifact Generation — transformed samples saved automatically
• Progress Tracking — optional live progress bars for long-running evaluations
• Extensible Plugin Architecture — easily add custom transformations and relations

pip install automr

git clone https://github.com/CharithManaujayaMUTEC/AutoMR-Framework.git
cd AutoMR-Framework

python -m venv venv
venv\Scripts\activate

pip install -r requirements.txt

from automr.api import AutoMR

automr = AutoMR(
    model=model,
    task="regression",
    input_type="image",
    epsilon=0.05,
    range_threshold=5.0
)

df, results = automr.run_full_test(
    dataset=dataset,
    max_samples=500,
    samples_per_mr=5,
    epsilon_min=0.01,
    epsilon_max=0.20,
    epsilon_count=4
)

Load Dataset
      ↓
Load Model
      ↓
Apply Transformations
      ↓
Generate Predictions
      ↓
Validate Metamorphic Relations
      ↓
Analyze Failures
      ↓
Run Epsilon Sensitivity Analysis
      ↓
Generate Reports
      ↓
Interactive Live Dashboard
      ↓
Export Results

=== AutoMR Results ===

                            total  passed  failed  failure_rate

DarkVisibilityRelation       100      89      11        0.11
TranslationRelation           50      47       3        0.06
RotationRelation              50      49       1        0.02
BrightnessRelation            50      50       0        0.00
ContrastRelation              50      50       0        0.00
BlurRelation                  50      50       0        0.00
FogRelation                   50      50       0        0.00
RainRelation                  50      50       0        0.00
SnowRelation                  50      50       0        0.00

AutoMR can automatically evaluate a model across multiple comparator thresholds.

Instead of manually selecting an epsilon value, the framework performs repeated metamorphic testing over a configurable epsilon range and reports:

- First Failure Epsilon
- Recommended Epsilon
- Stabilization Epsilon
- Maximum Failure Rate

Example console output
========== EPSILON ANALYSIS ==========

First Failure Epsilon : 0.01
Recommended Epsilon : 0.1367
Stabilization Epsilon : 0.1367
Maximum Failure Rate : 6.25%

======================================


Generated files
results/
├── epsilon_summary.csv
└── epsilon_report.txt

All reports are automatically saved to the results/ directory.

Transformation samples are saved per relation under results/transformation_samples/:

transformation_samples/
├── metadata.csv
├── transformation_summary.csv
├── brightness/
├── contrast/
├── blur/
├── rotation/
├── translation/
├── noise/
├── rain/
├── snow/
├── fog/
├── visibility/
└── darkness/

AutoMR automatically computes the following after each test run:

• Failure Rate — per metamorphic relation, across all samples
• Severity Analysis — average and maximum output deviation
• Worst-Case Failures — samples with the largest prediction deviations
• Failure Regions — parameter ranges where the model is most unstable
• Parameter Sensitivity — how model behavior shifts with transformation intensity
• Range Stability Analysis — identifies safe vs. unstable transformation ranges
• Prediction Trace Analysis — tracks prediction drift across all transformations
• Epsilon Sensitivity Analysis
• Automatic Epsilon Recommendation

AutoMR includes a real-time dashboard for evaluating metamorphic relations on webcam or video streams.

Features

- Live webcam/video inference
- Adjustable epsilon threshold
- Configurable metamorphic relations
- Interactive parameter range selection
- Real-time failure detection
- Automatic violation image capture
- Continuous CSV logging
- Summary statistics during execution

Launch

```python
from automr.dashboard import run_live_dashboard

run_live_dashboard(
    automr,
    model,
    video_source=0
)

Dashboard Controls

Control	Purpose
MR Index	Select relation
Enable	Enable/Disable relation
Tests	Number of parameter samples
Range %	Scale transformation range
Epsilon	Comparator threshold
Frame Skip	Processing frequency
R	Run benchmark
ESC	Exit

---

## 5. Update Generated Reports

Add the new files.

### Core Reports
automr_results.csv
prediction_trace.csv
failure_summary.csv
severity_summary.csv
worst_cases.csv
failure_regions.txt
range_summary.csv
range_analysis.csv
epsilon_summary.csv
epsilon_report.txt


### Dashboard Reports

Add a new subsection.

```markdown
### Live Dashboard Reports
results/live_dashboard/
├── dashboard_results.csv
├── dashboard_summary.csv
└── violations/


Each dashboard record stores the epsilon value used during evaluation, allowing experiments to be reproduced even when the threshold changes interactively.

Any model implementing a predict(x) interface is compatible:

output = model.predict(input)

Supported frameworks include TensorFlow, Keras, PyTorch, scikit-learn, XGBoost, and fully custom models.

AutoMR accepts any input type — images, time-series, sequential data, tabular data, or custom formats. Transformations are applied modularly and do not depend on input structure.

AutoMR does not perform preprocessing. Users must provide inputs in the format expected by their model. This ensures the original model pipeline is evaluated without modification.

AutoMR supports regression outputs, continuous predictions, numerical outputs, and custom scalar outputs. No assumptions are made about output scale or range — the comparator is configurable via the epsilon parameter.

AutoMR-Framework/
│
├── automr/
│   ├── __init__.py
│   ├── api.py
│   │
│   ├── analysis/
│   │   ├── __init__.py
│   │   └── analyzer.py
│   │
│   ├── comparators/
│   │   ├── __init__.py
│   │   ├── base.py
│   │   └── regression.py
│   │
│   ├── core/
│   │   ├── __init__.py
│   │   ├── range_tester.py
│   │   ├── failure_analysis.py
│   │   └── validation_runner.py
│   │
│   ├── dashboard/
│   │   ├── __init__.py
│   │   ├── live_dashboard.py
│   │   ├── video_runner.py
│   │   ├── control_panel.py
│   │   ├── dashboard_utils.py
│   │   └── graph_panel.py
│   │
│   ├── epsilon/
│   │   ├── __init__.py
│   │   ├── sensitivity.py
│   │   ├── summary.py
│   │   └── utils.py
│   │
│   ├── evaluation/
│   │   ├── __init__.py
│   │   └── baseline.py
│   │
│   ├── input_handlers/
│   │   ├── __init__.py
│   │   ├── base.py
│   │   ├── image.py
│   │   ├── tabular.py
│   │   └── sequence.py
│   │
│   ├── logging/
│   │   ├── __init__.py
│   │   └── logger.py
│   │
│   ├── models/
│   │   ├── __init__.py
│   │   └── wrapper.py
│   │
│   ├── registry/
│   │   ├── __init__.py
│   │   ├── transformation_registry.py
│   │   └── relation_registry.py
│   │
│   ├── relations/
│   │   ├── image_relations.py
│   │   ├── weather_relations.py
│   │   ├── behavioral_relations.py
│   │   └── temporal_relations.py
│   │
│   ├── transforms/
│   │   ├── image_transforms.py
│   │   ├── weather_transforms.py
│   │   ├── behavioral_transforms.py
│   │   └── temporal_transforms.py
│   │
│   └── verification/
│       ├── __init__.py
│       └── transformation_saver.py
│
├── examples/
│   ├── run_test.py
│   ├── webcam_automr_live.py
│   └── custom_relation_example.py
│
├── results/
│
├── README.md
├── LICENSE
├── pyproject.toml
├── requirements.txt
└── .gitignore

• Transformation suite is primarily focused on image-based inputs
• Classification-specific metamorphic relations are still under development
• Automatic (epsilon) recommendation is heuristic-based and should be validated for domain-specific safety requirements.
• Runtime depends on model inference speed
• Large datasets may require longer execution times

• NLP and text transformation extensions
• Tabular data transformation support
• Classification-specific metamorphic relations
• Native web dashboard
• Automatic epsilon optimization strategies
• Multi-camera live testing
• GPU-accelerated batch validation
• NLP and tabular metamorphic relations
• Distributed testing across multiple machines
• Cross-model MR comparison
• Automated result visualizations (plots and charts)
• Distributed and parallel testing support
• Web-based reporting interface

AutoMR provides the following contributions for regression-based autonomous driving systems:

• Automated metamorphic testing without ground-truth labels
• Label-free robustness validation under realistic conditions
• Parameterized MR evaluation with range sweep support
• Failure region detection and severity-based ranking
• Reusable and extensible testing infrastructure

Charith Manujaya — github.com/CharithManaujayaMUTEC

Raveesha Peiris — github.com/RaveeshaPeiris

Final Year Project — Metamorphic Testing Framework for Regression-Based Autonomous Driving AI/ML Models

@software{automr2025,
  title={AutoMR: A Metamorphic Testing Framework for Regression-Based Autonomous Driving Models},
  author={Charith Manujaya and Raveesha Peiris},
  year={2025}
}

Released under the MIT License.