docs(cli+webui normalization): Match both CLI and Web UI features (#2646

) * docs(cli+webui normalization): Added transactions and envs pages for webui * docs(cli+webui normalization): edit code blocks * docs(cli+webui normalization): added webui creating data stores, added cli running transactions * docs(cli+webui normalization): split cli/creating-tests into multiple pages * Update docs/docs/cli/creating-test-outputs.md Co-authored-by: Julianne Fermi <julianne@kubeshop.io> * Update docs/docs/cli/creating-test-outputs.md Co-authored-by: Julianne Fermi <julianne@kubeshop.io> * Update docs/docs/cli/creating-test-outputs.md Co-authored-by: Julianne Fermi <julianne@kubeshop.io> * Update docs/docs/cli/running-tests.md Co-authored-by: Julianne Fermi <julianne@kubeshop.io> * docs(cli+webui normalization): typo * docs(cli+webui normalization): added undef vars in CLI, create dedicated ad hoc testing page under concepts --------- Co-authored-by: Julianne Fermi <julianne@kubeshop.io>
kubeshop · Jun 7, 2023 · 4f94356 · 4f94356
1 parent 0af6764
commit 4f94356
Show file tree

Hide file tree

Showing 15 changed files with 572 additions and 250 deletions.
diff --git a/docs/docs/cli/creating-data-stores.md b/docs/docs/cli/creating-data-stores.md
@@ -2,6 +2,8 @@
 
 You might have multiple Tracetest instances that need to be connected to the same data stores. An easy way of sharing the configuration is by using a configuration file that can be applied to your Tracetest instance.
 
+## Supported Trace Data Stores
+
 ### Jaeger
 
 ```yaml

diff --git a/docs/docs/cli/creating-environments.md b/docs/docs/cli/creating-environments.md
@@ -1,4 +1,10 @@
-# Creating Environments
+# Defining Environments as Text Files
+
+This page showcases how to create and edit environments with the CLI.
+
+:::tip
+[To read more about environments check out environment concepts.](../concepts/environments.md)
+:::
 
 Just like Data Stores, you can also manage your environments using the CLI and definition files.
 
@@ -18,7 +24,7 @@ spec:
 
 In order to apply this configuration to your Tracetest instance, make sure to have your [CLI configured](./configuring-your-cli.md) and run:
 
-```
+```sh
 tracetest apply environment -f <environment.yaml>
 ```
 

diff --git a/docs/docs/cli/creating-test-outputs.md b/docs/docs/cli/creating-test-outputs.md
@@ -0,0 +1,79 @@
+# Defining Test Outputs in Text Files
+
+Outputs are really useful when running [Transactions](../concepts/transactions). They allow for exporting values from a test so they become available in the [Environment Variables](../concepts/environments.md) of the current transaction.
+
+## Outputs are Expression Results
+
+An output exports the result of an [Expression](../concepts/expressions) and assigns it to a name, so it can be injected into the environment variables of a running transaction.
+A `selector` is needed only if the provided expression refers to a/some span/s attribute or meta attributes.
+
+It can be defined using the following YAML definition:
+
+```yaml
+outputs:
+  - name: USER_ID
+    selector: span[name = "user creation"]
+    value: attr:myapp.users.created_id
+```
+
+The `value` attribute is an `expression` and is a very powerful tool.
+
+## Examples
+
+### Basic Expression
+
+You can output basic expressions:
+
+```yaml
+outputs:
+
+- name: ARITHMETIC_RESULT
+  value: 1 + 1
+  # results in ARITHMETIC_RESULT = 2
+
+- name: INTERPOLATE_STRING
+  # assume PRE_EXISTING_VALUE=someValue from env vars
+  value: "the value ${env:PRE_EXISTING_VALUE} comes from the env var PRE_EXISTING_VALUE"
+  # results in INTERPOLATE_STRING = "the value someValue comes from the env var PRE_EXISTING_VALUE
+```
+
+### Extract a Value from a JSON
+
+Imagine a hypotetical `/users/create` endpoint that returns the full `user` object, including the new ID, when the operation is successful.
+
+```yaml
+outputs:
+- name: USER_ID
+  selector: span[name = "POST /user/create"]
+  value: attr:http.response.body | json_path '.id'
+```
+
+### Multiple Values
+
+Using the same hypotethical user creation endpoint, a user creation might result on multiple sql queries, for example:
+
+- `INSERT INTO users ...`
+- `INSERT INTO permissions...`
+- `SELECT remaining_users FROM accounts`
+- `UPDATE accounts SET remaining_users ...`
+
+In this case, the service is instrumented so that each query generates a span of type `database`.
+You can get a list of SQL operations:
+
+```yaml
+outputs:
+- name: SQL_OPS
+  selector: span[tracetest.span.type = "database"]
+  value: attr:sql.operation
+  # result: SQL_OPS = ["INSERT", "INSERT", "SELECT", "UPDATE"]
+```
+
+Since the value is an array, you can also apply filters to it:
+
+```yaml
+outputs:
+- name: LAST_SQL_OP
+  selector: span[tracetest.span.type = "database"]
+  value: attr:sql.operation | get_index 'last'
+  # result: LAST_SQL_OP = "INSERT"
+```
diff --git a/docs/docs/cli/creating-test-specifications.md b/docs/docs/cli/creating-test-specifications.md
@@ -0,0 +1,88 @@
+# Defining Test Specifications in Text Files
+
+Test Specifications may be added to a trace to set a value for a step in the trace to determine success or failure.
+
+## Assertions and Selectors
+
+Assertions are as important as how you trigger your test. Without them, your test is just a fancy way of executing a request using a CLI command. In this section, we will discuss how you can declare your assertions in your definition file.
+
+Before we start, there are two concepts that you must understand to write your tests:
+
+- [Selectors](../concepts/selectors.md)
+- [Assertions](../concepts/assertions.md)
+
+### Selectors
+
+**Selectors** are queries that are executed against your trace tree and select a set of spans based on some attributes. They are responsible for defining which spans will be tested against your assertions.
+
+### Assertions
+
+**Assertions** are tests against a specific span based on its attributes. A practical example might be useful:
+
+Imagine you have to ensure that all your `database select statements` take `less than 500ms`. To write a test for that you must:
+
+1. Select all spans in your trace related to `select statements`.
+2. Check if all those spans lasted `less than 500ms`.
+
+For the first task, we use a selector: `span[db.statement contains "SELECT"]`. While the second one is achieved by using an assertion: `attr:tracetest.span.duration < 500ms`.
+
+> **Note:** When asserting time fields, you can use the following time units: `ns` (nanoseconds), `us` (microseconds), `ms` (milliseconds), `s` (seconds), `m` (minutes), and `h` (hours). Instead of defining `attr:tracetest.span.duration <= 3600s`, you can set it as `attr:tracetest.span.duration <= 1h`.
+
+To write that in your test definition, you can define the following YAML definition:
+
+```yaml
+specs:
+- selector: span[db.statement contains "SELECT"]
+  assertions:
+    - attr:tracetest.span.duration < 500ms
+```
+
+As you probably noticed in the test definition structure, you can have multiple assertions for the same selector. This is useful to group related validations. For example, ensuring that all your HTTP calls are successful and take less than 1000ms:
+
+```yaml
+specs:
+- selector: span[tracetest.span.type="http"]
+  assertions:
+    - attr:http.status_code >= 200
+    - attr:http.status_code < 300
+    - attr:tracetest.span.duration < 1000ms
+```
+
+#### Referencing Other Fields from the Same Span
+
+You also can reference fields from the same span in your assertions. For example, you can define an assertion to ensure the output number is greater than the input number.
+
+```yaml
+specs:
+- selector: span[name = "my operation"]
+  assertions:
+    - attr:myapp.output > attr:myapp.input
+```
+
+You also can use basic arithmetic expressions in your assertions:
+
+```yaml
+assertions:
+  - attr:myapp.output = myapp.input + 1
+```
+
+:::note
+This does not take into account the order of operators yet. So an expression `1 + 2 * 3` will be resolved as `9` instead of `7`. This will be fixed in future releases.
+:::
+
+Available operations in an expression are: `+`, `-`, `*`, and `/`.
+
+For more information about selectors or assertions, take a look at the documentation for those topics.
+
+## Available Operations
+
+| Operator              | Description |
+| :------------- | ------------------------------------------------------------------------------------------------------------------------ |
+| `=`            | Check if two values are equal.                                                                                            |
+| `!=`           | Check if two values have different values.                                                                                |
+| `<`            | Check if value from left side is smaller than the one on the right side of the operation.                                 |
+| `<=`           | Check if value from left side is smaller or equal to the one on the right side of the operation.                          |
+| `>`            | Check if value from left side is larger than the one on the right side of the operation.                                  |
+| `>=`           | Check if value from left side is larger or equal to the one on the right side of the operation.                           |
+| `contains`     | Check if value on the right side of the operation is contained inside of the value of the left side of the operation.     |
+| `not-contains` | Check if value on the right side of the operation is not contained inside of the value of the left side of the operation. |
diff --git a/docs/docs/cli/creating-tests.md b/docs/docs/cli/creating-tests.md
@@ -9,9 +9,10 @@ As Tracetest is mainly a visual tool, this might make it difficult to update tes
 Imagine that you were assigned a ticket to improve your application database usage. You notice that every time a specific endpoint is called, your application executes `N+1` select statements on the database instead of only one statement. You probably already have a test in place to ensure the correct functionality of that endpoint: it inserts the necessary information into the database, calls that specific endpoint using our tool and ensures you get the expected results using the trace generated by your application. It works fine, but there is a problem. That test is managed by Tracetest on its server and the test cannot be changed until the new patch is deployed. Otherwise, if the test is run using a non-patched version of the application, the test would fail.
 
 To solve that, the best approach would be to enable developers to define their tests as text files and allow them to run those tests using a CLI, so you can integrate the execution of those tests to your existing CI pipeline. There are many benefits of this functionality for your tests:
- - Peers can review your tests before merging them to the main branch.
- - Ensure your test works before merging it to the main branch.
- - Have different versions of the same test running in parallel in different branches, so you and your peers can work on the same code modules and update the same test without interfering with each other.
+
+- Peers can review your tests before merging them to the main branch.
+- Ensure your test works before merging it to the main branch.
+- Have different versions of the same test running in parallel in different branches, so you and your peers can work on the same code modules and update the same test without interfering with each other.
 
 ## Definition
 
@@ -91,6 +92,7 @@ trigger:
 Currently, we support three authentication methods for HTTP requests: `basic authentication`, `api key`, and `bearer token`. Here is one example of each authentication method:
 
 **Basic Authentication**
+
 ```yaml
 trigger:
     type: http
@@ -105,6 +107,7 @@ trigger:
 ```
 
 **API Key Authentication**
+
 ```yaml
 trigger:
     type: http
@@ -120,6 +123,7 @@ trigger:
 ```
 
 **Bearer Token Authentication**
+
 ```yaml
 trigger:
     type: http
@@ -166,154 +170,10 @@ Available functions:
 | `creditCardExpDate()` | Generates a random credit card expiration date (mm/yy).                                |
 | `randomInt(min, max)` | Generates a random integer contained in the closed interval defined by [`min`, `max`]. |
 
+:::tip
+[Continue reading about Test Specs, here.](./creating-test-specifications.md)
+:::
 
-## Assertions
-
-Assertions are as important as how you trigger your test. Without them, your test is just a fancy way of executing a request using a CLI command. In this section, we will discuss how you can declare your assertions in your definition file.
-
-Before we start, there are two concepts that you must understand to write your tests: [selectors](../concepts/selectors) and assertions.
-
-**Selectors** are queries that are executed against your trace tree and select a set of spans based on some attributes. They are responsible for defining which spans will be tested against your assertions.
-
-**Assertions** are tests against a specific span based on its attributes. A practical example might be useful:
-
-Imagine you have to ensure that all your `database select statements` take `less than 500ms`. To write a test for that you must:
-
-1. Select all spans in your trace related to `select statements`.
-2. Check if all those spans lasted `less than 500ms`.
-
-For the first task, we use a selector: `span[db.statement contains "SELECT"]`. While the second one is achieved by using an assertion: `attr:tracetest.span.duration < 500ms`.
-
-> **Note:** When asserting time fields, you can use the following time units: `ns` (nanoseconds), `us` (microseconds), `ms` (milliseconds), `s` (seconds), `m` (minutes), and `h` (hours). Instead of defining `attr:tracetest.span.duration <= 3600s`, you can set it as `attr:tracetest.span.duration <= 1h`.
-
-To write that in your test definition, you can define the following YAML definition:
-
-```yaml
-specs:
-- selector: span[db.statement contains "SELECT"]
-  assertions:
-    - attr:tracetest.span.duration < 500ms
-```
-
-As you probably noticed in the test definition structure, you can have multiple assertions for the same selector. This is useful to group related validations. For example, ensuring that all your HTTP calls are successful and take less than 1000ms:
-
-```yaml
-specs:
-- selector: span[tracetest.span.type="http"]
-  assertions:
-    - attr:http.status_code >= 200
-    - attr:http.status_code < 300
-    - attr:tracetest.span.duration < 1000ms
-```
-
-#### Referencing Other Fields from the Same Span**
-You also can reference fields from the same span in your assertions. For example, you can define an assertion to ensure the output number is greater than the input number.
-
-```yaml
-specs:
-- selector: span[name = "my operation"]
-  assertions:
-    - attr:myapp.output > attr:myapp.input
-```
-
-You also can use basic arithmetic expressions in your assertions:
-```yaml
-assertions:
-  - attr:myapp.output = myapp.input + 1
-```
-> **Note:** This does not take into account the order of operators yet. So an expression `1 + 2 * 3` will be resolved as `9` instead of `7`. This will be fixed in future releases.
-
-Available operations in an expression are: `+`, `-`, `*`, and `/`.
-
-For more information about selectors or assertions, take a look at the documentation for those topics.
-
-### Available Operations
-
-| Operator              | Description |
-| :------------- | ------------------------------------------------------------------------------------------------------------------------ |
-| `=`            | Check if two values are equal.                                                                                            |
-| `!=`           | Check if two values have different values.                                                                                |
-| `<`            | Check if value from left side is smaller than the one on the right side of the operation.                                 |
-| `<=`           | Check if value from left side is smaller or equal to the one on the right side of the operation.                          |
-| `>`            | Check if value from left side is larger than the one on the right side of the operation.                                  |
-| `>=`           | Check if value from left side is larger or equal to the one on the right side of the operation.                           |
-| `contains`     | Check if value on the right side of the operation is contained inside of the value of the left side of the operation.     |
-| `not-contains` | Check if value on the right side of the operation is not contained inside of the value of the left side of the operation. |
-
-
-## Outputs
-
-Outputs are really useful when running [Transactions](../concepts/transactions). They allow for exporting values from a test so they become available in the [Environment Variables](../concepts/environments.md) of the current transaction.
-
-An ouptut exports the result of an [Expression](../concepts/expressions) and assigns it to a name, so it can be injected into the environment variables of a running transaction.
-A `selector` is needed only if the provided expression refers to a/some span/s attribute or meta attributes.
-
-It can be defined using the following YAML definition:
-
-```yaml
-outputs:
-  - name: USER_ID
-    selector: span[name = "user creation"]
-    value: attr:myapp.users.created_id
-```
-
-The `value` attribute is an `expression`, and is a very powerful tool.
-
-## Examples
-
-### Basic expression
-
-You can output basic expressions:
-
-```yaml
-outputs:
-
-- name: ARITHMETIC_RESULT
-  value: 1 + 1
-  # results in ARITHMETIC_RESULT = 2
-
-- name: INTERPOLATE_STRING
-  # assume PRE_EXISTING_VALUE=someValue from env vars
-  value: "the value ${env:PRE_EXISTING_VALUE} comes from the env var PRE_EXISTING_VALUE"
-  # results in INTERPOLATE_STRING = "the value someValue comes from the env var PRE_EXISTING_VALUE
-```
-
-### Extract a Value from a JSON
-
-Imagine a hypotetical `/users/create` endpoint that returns the full `user` object, including the new ID, when the operation is successful.
-
-```yaml
-outputs:
-- name: USER_ID
-  selector: span[name = "POST /user/create"]
-  value: attr:http.response.body | json_path '.id'
-```
-
-### Multiple values
-
-Using the same hypotethical user creation endpoint, a user creation might result on multiple sql queries, for example:
-- `INSERT INTO users ...`
-- `INSERT INTO permissions...`
-- `SELECT remaining_users FROM accounts`
-- `UPDATE accounts SET remaining_users ...`
-
-In this case, the service is instrumented so that each query generates a span of type `database`.
-You can get a list of SQL operations:
-
-```yaml
-outputs:
-- name: SQL_OPS
-  selector: span[tracetest.span.type = "database"]
-  value: attr:sql.operation
-  # result: SQL_OPS = ["INSERT", "INSERT", "SELECT", "UPDATE"]
-```
-
-Since the value is an array, you can also apply filters to it:
-
-```yaml
-outputs:
-- name: LAST_SQL_OP
-  selector: span[tracetest.span.type = "database"]
-  value: attr:sql.operation | get_index 'last'
-  # result: LAST_SQL_OP = "INSERT"
-```
+:::tip
+[Continue reading about Test Outputs, here.](./creating-test-outputs.md)
+:::