Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions src/components/EnterpriseInstallReminder.js
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ export default function EnterpriseInstallReminder() {
};

return (
<div
style={containerStyle}
>
<h3 style={{marginTop: 0, color: isDark ? "#fff" : "#222"}}>Don’t have Keploy installed yet?</h3>
<div style={containerStyle}>
<h3 style={{marginTop: 0, color: isDark ? "#fff" : "#222"}}>
Don’t have Keploy installed yet?
</h3>
<p style={{color: isDark ? "#ccc" : undefined}}>
Before running this sample, make sure Keploy Enterprise version is
installed on your system.
Expand Down
8 changes: 4 additions & 4 deletions src/components/InstallReminder.js
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ export default function InstallReminder() {
};

return (
<div
style={containerStyle}
>
<h3 style={{marginTop: 0, color: isDark ? "#fff" : "#222"}}>Don’t have Keploy installed yet?</h3>
<div style={containerStyle}>
<h3 style={{marginTop: 0, color: isDark ? "#fff" : "#222"}}>
Don’t have Keploy installed yet?
</h3>
<p style={{color: isDark ? "#ccc" : undefined}}>
Before running this sample, make sure Keploy is installed on your
system.
Expand Down
8 changes: 8 additions & 0 deletions versioned_docs/version-3.0.0/running-keploy/cli-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,8 @@ keploy gen [flags]

The `normalize` cmd in Keploy allows user to change the response of the testcases according to the latest test run response that is executed by the user, this is useful when the API response of the testcases are changed due to code change or any other intentional change in the application.

This command is now “risk-aware” to prevent accidental acceptance of breaking API changes. By default, it will automatically update tests that failed with Low or Medium risk, but it will refuse to normalize any test that failed with a High risk, printing a warning instead.

<b> Usage: </b>

```bash
Expand Down Expand Up @@ -299,6 +301,12 @@ keploy normalize [flags]
keploy normalize -p "./tests" --test-run "test-run-10" --tests "test-set-1:test-case-1 test-case-2,test-set-2:test-case-1 test-case-2"
```

- `--allow-high-risk` - Allow normalization of high-risk test failures. This flag overrides the default safe behavior and updates all failed tests, including those with breaking changes.

```bash
keploy normalize --allow-high-risk
```

## [rerecord](#rerecord)

The `rerecord` command allows users to record new Keploy test cases and mocks from existing test cases for the given testset(s).
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
---
id: risk-profile-analysis
title: Risk Profile Analysis
sidebar_label: Risk Profile Analysis
description: Discover Keploy's Risk Profile Analysis feature, which automatically classifies API test failures as High, Medium, or Low risk, helping you instantly distinguish between breaking changes and minor updates.
tags:
- risk-analysis
- change-management
- breaking-changes
keywords:
- risk analysis
- breaking changes
- High Risk
- Medium Risk
- Low Risk
- Risk Categories
- Schema Changed
- Schema Broken
---

When API contracts change, tests are expected to fail. However, not all failures are equal. A minor, backward-compatible change (like adding a new optional field) is very different from a major, breaking change (like removing a field or changing a data type).

The **Risk Profile Analysis** feature addresses this by automatically categorizing each test failure based on its potential impact. It assigns a risk level of **High**, **Medium**, or **Low**, giving developers immediate insight into the severity and nature of API changes. This helps distinguish between intentional contract updates and unintentional bugs right from the test report.

### Key Concepts

When a test fails, it is now assigned a risk level:

- **HIGH**: Indicates a likely breaking change to the API contract. This is the highest level of risk.

- **Triggers**: Status code changes, `Content-Type` header changes, removing fields from a JSON body, or changing the data type of a field (e.g., string to number).

- **MEDIUM**: Indicates a change that might affect consumers but is not a direct contract violation.

- **Triggers**: Changes in header values (other than `Content-Type`), or changes to field values within a JSON body while new fields are also being added.

- **LOW**: Indicates a backward-compatible, non-breaking change.
- **Triggers**: Only adding new, optional fields to a JSON body.

To provide more detail, failures are also assigned one or more categories:

- `SCHEMA_BROKEN`: A breaking change occurred in the response body (field removed, type changed).
- `SCHEMA_ADDED`: Only new fields were added to the response body.
- `SCHEMA_UNCHANGED`: The response body schema is identical, but values within it have changed.
- `STATUS_CODE_CHANGED`: The HTTP status code was different from the expected one.
- `HEADER_CHANGED`: One or more headers were different.

##### How Risk Profiling Works During Testing

1. **Failure Detection**: Keploy runs tests as usual. When a response mismatch is detected, the new analysis logic is triggered.
2. **Deep Comparison**: Keploy performs a detailed comparison of the expected versus actual responses:
- **Status Code**: Checks for any mismatch.
- **Headers**: Pays special attention to the `Content-Type` header.
- **JSON Body**: Analyzes the _nature_ of the difference by comparing the structure of the expected and actual JSON payloads to identify added fields, removed fields, and data type changes.
3. **Risk Assessment**: Based on the comparison, a `RiskLevel` and one or more `FailureCategory` tags are assigned to the test result.
4. **Enhanced Reporting**: The test report (`test-run-*-report.yaml`) is updated to include these new details. The summary now includes counts for high, medium, and low-risk failures, and each individual failed test specifies its risk and category.

**Example Test Report Snippet:**

```yaml
version: 2.0.0
name: test-set-0
status: FAILED
success: 0
failure: 12
high-risk: 7
medium-risk: 4
low-risk: 1
total: 12
tests:
- name: test-1
status: FAILED
# ... other fields
failure_info:
risk: HIGH
category:
- SCHEMA_BROKEN
- name: test-2
status: FAILED
# ... other fields
failure_info:
risk: LOW
category:
- SCHEMA_ADDED
```
3 changes: 2 additions & 1 deletion versioned_sidebars/version-3.0.0-sidebars.json
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,8 @@
"running-keploy/docker-tls",
"running-keploy/configuration-file",
"running-keploy/custom-mocks",
"running-keploy/keploy-templatize"
"running-keploy/keploy-templatize",
"running-keploy/risk-profile-analysis"
]
},
{
Expand Down