Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: New Version Updates #3023

Merged
merged 5 commits into from
Aug 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/docs/cli/creating-environments.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
This page showcases how to create and edit environments with the CLI.

:::tip
[To read more about environments check out environment concepts.](../concepts/environments.md)
[To read more about environments check out environment concepts.](../concepts/variable-sets.md)
:::

Just like Data Stores, you can also manage your environments using the CLI and definition files.
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/cli/creating-test-outputs.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Defining Test Outputs in Text Files

Outputs are really useful when running [Transactions](../concepts/transactions). They allow for exporting values from a test so they become available in the [Environment Variables](../concepts/environments.md) of the current transaction.
Outputs are really useful when running [Transactions](../concepts/transactions). They allow for exporting values from a test so they become available in the [Environment Variables](../concepts/variable-sets.md) of the current transaction.

## Outputs are Expression Results

Expand Down
66 changes: 66 additions & 0 deletions docs/docs/concepts/variable-sets.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Variable Sets

A common use case for tests is to assert the same behavior across multiple environments (dev, staging, and production, for example). To make sure all of these environments will have the same behavior, it is important that the tests executed against those environments test the same aspects. To reduce the risks of diverging tests, Tracetest allows you to organize different environments configurations using global objects called **Variable Sets**.

## How Variable Sets Work

Variable sets are objects containing variables that can be referenced by tests. You can use a single test and provide the information on which environment object will be used to run the test. To illustrate this, consider an app that is deployed in three stages: `dev`, `staging`, and `production`. We can execute the same test in all those environments, however, both `URL` and `credentials` change from environment to environment. To run the same test against the three deployments of the app, you can create three variable sets:

```yaml
# dev.env
type: Environment
spec:
name: dev.env
id: dev.env
values:
- key: URL
value: https://app-dev.com
- key: API_TOKEN
value: dev-key
```

```yaml
# staging.env
type: Environment
spec:
name: staging.env
id: staging.env
values:
- key: URL
value: https://app-staging.com
- key: API_TOKEN
value: staging-key
```

```yaml
# production.env
type: Environment
spec:
name: production.env
id: production.env
values:
- key: URL
value: https://app-prod.com
- key: API_TOKEN
value: prod-key
```

Now consider the following test:

```yaml
type: Test
specs:
name: Test user creation
trigger:
type: http
httpRequest:
url: "${env:URL}/api/users"
method: POST
body: '{}'
authentication:
type: bearer
bearer:
token: "${env:API_TOKEN}"
```

Both `env:URL` and `env:API_TOKEN` would be replaced by the variables defined in the chosen environment where the test will run. So, if the chosen environment was `dev.env`, its values would be replaced by `https://app-dev.com` and `dev-key`, respectively.
10 changes: 9 additions & 1 deletion docs/docs/configuration/tracetest-analyzer.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,12 @@ In the Tracetest UI, go to **Settings** and the **Analyzer** tab:

![Analyzer Settings](../img/analyzer-settings.png)

This feature will be available in the CLI very soon. Stay tuned!
Toggle `Enable Analyzer for All Tests` off to disable the Analyzer.

Here, you can also set the thresholds for `Otel Semantic Conventions`, `Common Problems` and `Security` analyzer settings.

![Analyzer Settings 2](../img/analyzer-settings-2.png)

## Tracetest Analyzer in the CLI

You can use Tracetest Analyzer in the CLI to analyze per individual test. Visit the [Creating Transactions](/docs/docs/web-ui/creating-transactions.md) page for details.
Binary file added docs/docs/img/analyzer-settings-2.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/docs/img/analyzer-settings.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/docs/img/automate-tab.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/docs/img/running-transaction.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ So in this case, we need to trigger four tests in sequence to achieve test the e

## Building a Transaction for This Scenario

Using Tracetest, we can do that by [creating a test](../../../web-ui/creating-tests.md) for each step and later grouping these tests as [transactions](../../../web-ui/creating-transactions.md) that have an [environment](../../../concepts/environments.md)](../../../concepts/environments.md).
Using Tracetest, we can do that by [creating a test](../../../web-ui/creating-tests.md) for each step and later grouping these tests as [transactions](../../../web-ui/creating-transactions.md) that have an [variable set](../../../concepts/variable-sets.md)](../../../concepts/variable-sets.md).

We can do that by creating the tests and transactions through the Web UI or using the CLI. In this example, we will use the CLI to create an environment and then create the transaction with all tests needed. The [assertions](../../../concepts/assertions.md) that we will check are the same for every single test.
We can do that by creating the tests and transactions through the Web UI or using the CLI. In this example, we will use the CLI to create a Variable Set and then create the transaction with all tests needed. The [assertions](../../../concepts/assertions.md) that we will check are the same for every single test.

### Mapping Environment Variables

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/web-ui/creating-environments.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
This page showcases how to create and edit environments in the Web UI.

:::tip
[To read more about environments check out environment concepts.](../concepts/environments.md)
[To read more about variable sets check out the variable sets concepts guide.](../concepts/variable-sets.md)
:::

![](https://res.cloudinary.com/djwdcmwdz/image/upload/v1685967319/docs/demo.tracetest.io_transaction_QZ3ejgl4R_run_2_oproo4.png)
Expand Down
12 changes: 10 additions & 2 deletions docs/docs/web-ui/creating-transactions.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,17 @@ Next, select which tests to run in the transaction and click **Create & Run**:

The transaction will start:

![Running Transaction](https://res.cloudinary.com/djwdcmwdz/image/upload/v1685713352/docs/beta.tracetest.io__page_1_2_oqwazx.png)
![Running Transaction](../img/running-transaction.png)

<!--- https://res.cloudinary.com/djwdcmwdz/image/upload/v1685713352/docs/beta.tracetest.io__page_1_2_oqwazx.png) -->

When the transaction is finished, you will get the following results:
On the automate tab, you find methods to automate the current transaction, including the YAML test file and the CLI command for Tracetest.

![Automate Tab](../img/automate-tab.png)

With all of the toggles `Off`, each criteria is tested. Toggle individual criteria `On`, to select on certain checks for the selected test.

When the transaction is finished, you will get the following result:

![Finished Transaction](https://res.cloudinary.com/djwdcmwdz/image/upload/v1685713712/docs/demo.tracetest.io__x0o1gu.png)

Expand Down
4 changes: 2 additions & 2 deletions docs/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -445,8 +445,8 @@ const sidebars = {
// },
{
type: "doc",
id: "concepts/environments",
label: "Environments",
id: "concepts/variable-sets",
label: "Variable Sets",
},
{
type: "doc",
Expand Down
6 changes: 6 additions & 0 deletions node_modules/.package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.