Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Epic] CLI Improvements #2182

Closed
15 of 19 tasks
xoscar opened this issue Mar 16, 2023 · 19 comments
Closed
15 of 19 tasks

[Epic] CLI Improvements #2182

xoscar opened this issue Mar 16, 2023 · 19 comments
Labels

Comments

@xoscar
Copy link
Collaborator

xoscar commented Mar 16, 2023

Our current implementation of the CLI is not consistent:

  • We are using 2 different approaches to manage different resources,
  • not all commands share a standard for flags,
  • output format is not consistent,
  • etc,

This epic aims to make the CLI consistent.

Issues by type

Standardize what we have

Deprecation of the old architecture

New Features

@schoren schoren changed the title [Epic] CLI Improvements DRAFT [Epic] CLI Improvements Mar 27, 2023
@xoscar xoscar changed the title DRAFT [Epic] CLI Improvements [Epic] CLI Improvements Mar 29, 2023
@schoren
Copy link
Collaborator

schoren commented Mar 30, 2023

Global Flags

All commands share some common flags:

-c, --config string         config file will be used by the CLI (default "config.yml")
-s, --server-url string     override the target server url
-o, --output-format string  output format [pretty|json] (default "pretty")
    --output string        output format [pretty|json] (default "pretty") -- DEPRECATED!!! use --output-format instead
-h, --help                  help for tracetest

These flags MUST NOT be overriden by subcommands both in the short and long form.

For example, no subcommand can use the -s short flag for anything other than the global behaviour of overriding the server url.

@schoren
Copy link
Collaborator

schoren commented Mar 30, 2023

PollingProfile

tracetest get pollingprofile

Description: get the currently active polling profile
Required flags: none
Optional Flags none
Pretty Output Example:

Strategy Details
periodic Retry Delay: 500ms
Timeout: 50s

JSON Output Example:

{
  "type": "PollingProfile",
  "spec":{
    "id":"current",
    "name":"default",
    "default":true,
    "strategy":"periodic",
    "periodic":{
      "retryDelay":"500ms",
      "timeout":"50s"
    }
  }
}

YAML Output Example:

---
type: PollingProfile
spec:
  id: current
  name: default
  default: true
  strategy: periodic
  periodic:
    retryDelay: 500ms
    timeout: 50s

tracetest list pollingprofile

Description: PollingProfile is a single resource, so list is not available
Required flags: none
Optional Flags none
Output: For any format, do not print anything to stdout, print to StdErr:

PollingProfile does not support listing. Try `tracetest get pollingprofile`

tracetest delete pollingprofile

Description: reset the pollingprofile to the default state
Required flags: none
Optional Flags none
Pretty Output Example:

PollingProfile successfully reset to default

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "PollingProfile successfully reset to default"
}

YAML Output Example:

---
sucess: true
message: PollingProfile successfully reset to default
errors: []

tracetest apply pollingprofile --file [path]

Description: update the pollingprofile settings
Required flags: --file|-f input file
Optional Flags none
Validation:

  • if more than one document is present on the file: Only one document per file is allowed
  • if document's type is not PollingProfile: Invalid type [actual type]. Expected PollingProfile
    Pretty Output Example:
PollingProfile successfully updated

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "PollingProfile successfully updated"
}

YAML Output Example:

---
sucess: true
message: PollingProfile successfully updated
errors: []

tracetest run pollingprofile

Description: PollingProfile is not runnable, so run is not available
Required flags: none
Optional Flags none
Output: For any format, do not print anything to stdout, print to StdErr:

PollingProfile does not support running. try `tracetest help run` for more information on the run command

@schoren
Copy link
Collaborator

schoren commented Mar 30, 2023

Config

tracetest get config

Description: get the currently active config
Required flags: none
Optional Flags none
Pretty Output Example:

Analytics Enabled
yes

JSON Output Example:

{
  "type": "Config",
  "spec": {
    "id": "current",
    "name": "Current",
    "analyticsEnabled": true
  }
}

YAML Output Example:

---
type: Config
spec:
  id: current
  name: Current
  analyticsEnabled: true

tracetest list config

Description: Config is a single resource, so list is not available
Required flags: none
Optional Flags none
Output: For any format, do not print anything to stdout, print to StdErr:

Config does not support listing. Try `tracetest get config`

tracetest delete config

Description: reset the config to the default state
Required flags: none
Optional Flags none
Pretty Output Example:

Config successfully reset to default

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "Config successfully reset to default"
}

YAML Output Example:

---
sucess: true
message: Config successfully reset to default
errors: []

tracetest apply config --file [path]

Description: update the config settings
Required flags: --file|-f input file
Optional Flags none
Validation:

  • if more than one document is present on the file: Only one document per file is allowed
  • if document's type is not Config: Invalid type [actual type]. Expected Config

Pretty Output Example:

Config successfully updated

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "Config successfully updated"
}

YAML Output Example:

---
sucess: true
message: Config successfully updated
errors: []

tracetest run config

Description: Config is not runnable, so run is not available
Required flags: none
Optional Flags none
Output: For any format, do not print anything to stdout, print to StdErr:

Config does not support running. try `tracetest help run` for more information on the run command

@schoren
Copy link
Collaborator

schoren commented Mar 30, 2023

Demo

tracetest get demo --id [the id]

Description: Get a demo by ID
Required flags: --id
Optional Flags: none
Pretty Output Example:

ID Name Type Details Enabled
WZGNTOBVg pokeshop pokeshop - http Endpoint: http://demo-pokemon-api.demo
- grpc Endpoint: demo-pokemon-api.demo
yes

JSON Output Example:

{
  "type": "Demo",
  "spec": {
    "id": "WZGNTOBVg",
    "name": "pokeshop",
    "enabled": true,
    "pokeshop": {
      "httpEndpoint": "http://demo-pokemon-api.demo",
      "grpcEndpoint": "demo-pokemon-api.demo"
    }
  }
}

YAML Output example:

---
type: Demo
spec:
  id: WZGNTOBVg
  name: pokeshop
  enabled: true
  pokeshop:
    httpEndpoint: http://demo-pokemon-api.demo
    grpcEndpoint: demo-pokemon-api.demo

tracetest list demo

Description: List Demos
Required flags: none
Optional Flags:

  • --all Get all results, not paginated. If passed, --take and --skip do nothing
  • --take N take N records [default: 20]
  • --skip N skip the first N records [default: 0]
  • --query get resources matching the given query string

Pretty Output Example:

Total avialable Demos: 3

ID Name type Details Enabled
WZGNTOBVg Pokeshop pokeshop - http Endpoint: http://demo-pokemon-api.demo
- grpc Endpoint: demo-pokemon-api.demo
yes
WZGNRenVg Pokeshop pokeshop - http Endpoint: http://old-url.demo
- grpc Endpoint: old-url.demo
no
BaGNTOBdW otel opentelemetryStore - cart Endpoint: http://otel-cartservice.otel-demo:7070
- checkout Endpoint: http://otel-checkoutservice.otel-demo:5050
- frontend Endpoint: http://otel-frontend.otel-demo:8084
- product Catalog Endpoint: http://otel-productcatalogservice.otel-demo:3550
yes

JSON Output Example:

{
  "count": 3,
  "items": [
    {
      "type": "Demo",
      "spec": {
        "id": "WZGNTOBVg",
        "name": "pokeshop",
        "enabled": true,
        "pokeshop": {
          "httpEndpoint": "http://demo-pokemon-api.demo",
          "grpcEndpoint": "demo-pokemon-api.demo"
        }
      }
    },
    {
      "type": "Demo",
      "spec": {
        "id": "WZGNRenVg",
        "name": "pokeshop",
        "type": "pokeshop",
        "enabled": false,
        "pokeshop": {
          "httpEndpoint": "http://old-url.demo",
          "grpcEndpoint": "old-url.demo"
        }
      }
    },
    {
      "type": "Demo",
      "spec": {
        "id": "WZGNRenVg",
        "name": "pokeshop",
        "type": "opentelemetryStore",
        "opentelemetryStore": {
          "cartEndpoint": "http://otel-cartservice.otel-demo:7070",
          "checkoutEndpoint": "http://otel-checkoutservice.otel-demo:5050",
          "frontendEndpoint": "http://otel-frontend.otel-demo:8084",
          "productCatalogEndpoint": "http://otel-productcatalogservice.otel-demo:3550"
        }
      }
    }
  ]
}

YAML Output example:

---
type: Demo
spec:
  id: WZGNTOBVg
  name: pokeshop
  enabled: true
  pokeshop:
    httpEndpoint: http://demo-pokemon-api.demo
    grpcEndpoint: demo-pokemon-api.demo
---
type: Demo
spec:
  id: WZGNRenVg
  name: pokeshop
  type: pokeshop
  enabled: false
  pokeshop:
    httpEndpoint: http://old-url.demo
    grpcEndpoint: old-url.demo
---
type: Demo
spec:
  id: WZGNRenVg
  name: pokeshop
  type: opentelemetryStore
  opentelemetryStore:
    cartEndpoint: http://otel-cartservice.otel-demo:7070
    checkoutEndpoint: http://otel-checkoutservice.otel-demo:5050
    frontendEndpoint: http://otel-frontend.otel-demo:8084
    productCatalogEndpoint: http://otel-productcatalogservice.otel-demo:3550

tracetest get demo --id [the id]

Description: Delete a demo by ID
Required flags: --id
Optional Flags: none
Pretty Output Example:

Demo successfully deleted

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "Demo successfully deleted"
}

YAML Output Example:

---
sucess: true
message: Demo successfully deleted
errors: []

tracetest apply demo --file [path]

Description: Save a demo to the server. If it does not yet exists on the server, it is created. Otherwise it is updated. Input can be a file with one or more documents, or a directory with files with one or more documents. This is not transactional, so if an error happens, all changes to resources applied until the error are kept.
Required flags: --file|-f input file or directory
Optional Flags: none
Validation:

  • if the given file does not have a spec.id set: The resource file does not have a ID defined in the "spec" object. Please provide a unique one. For example, you can use [SHOW A RANDOM VALID ID]
  • if a document's type is not Demo: Invalid type [actual type]. Expected Demo

Pretty Output Example:

Demo successfully applied

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "Demo successfully applied"
}

YAML Output Example:

---
sucess: true
message: Demo successfully applied
errors: []

tracetest run demo

Description: Demo is not runnable, so run is not available
Required flags: none
Optional Flags: none
Output: For any format, do not print anything to stdout, print to StdErr:

Demo does not support running. try `tracetest help run` for more information on the run command

@schoren
Copy link
Collaborator

schoren commented Mar 30, 2023

Environment

tracetest get environment --id [the id]

Description: Get an environment by ID
Required flags: --id
Optional Flags: none
Pretty Output Example:

ID Name Description Values
dev dev my dev environment - hostname: localhost
- app_port: 8080
- pokemon_id: 22

JSON Output Example:

{
  "type": "Environment",
  "spec": {
    "id": "dev",
    "name": "dev",
    "description": "my dev environment",
    "values": [
      {
        "key": "hostname",
        "value": "localhost"
      },
      {
        "key": "app_port",
        "value": "8080"
      },
      {
        "key": "pokemon_id",
        "value": "22"
      }
    ]
  }
}

YAML Output example:

---
type: Environment
spec:
  id: dev
  name: dev
  description: my dev environment
  values:
  - key: hostname
    value: localhost
  - key: app_port
    value: "8080"
  - key: pokemon_id
    value: "22"

tracetest list environment

Description: List Environments
Required flags: none
Optional Flags:

  • --all Get all results, not paginated. If passed, --take and --skip do nothing
  • --take N take N records [default: 20]
  • --skip N skip the first N records [default: 0]
  • --query get resources matching the given query string

Pretty Output Example:

Total avialable Environments: 2

ID Name Description Values
dev dev my dev environment - hostname: localhost
- app_port: 8080
- pokemon_id: 22
prod prod my prod environment - hostname: myapp.com
- app_port: 8080
- pokemon_id: 100

JSON Output Example:

{
  "count": 2,
  "items": [
    {
      "type": "Environment",
      "spec": {
        "id": "dev",
        "name": "dev",
        "description": "my dev environment",
        "values": [
          {
            "key": "hostname",
            "value": "localhost"
          },
          {
            "key": "app_port",
            "value": "8080"
          },
          {
            "key": "pokemon_id",
            "value": "22"
          }
        ]
      }
    },
    {
      "type": "Environment",
      "spec": {
        "id": "prod`",
        "name": "prod",
        "description": "my prod environment",
        "values": [
          {
            "key": "hostname",
            "value": "myapp.com"
          },
          {
            "key": "app_port",
            "value": "8080"
          },
          {
            "key": "pokemon_id",
            "value": "100"
          }
        ]
      }
    }
  ]
}

YAML Output example:

---
type: Environment
spec:
  id: dev
  name: dev
  description: my dev environment
  values:
  - key: hostname
    value: localhost
  - key: app_port
    value: "8080"
  - key: pokemon_id
    value: "22"
---
type: Environment
spec:
  id: dev
  name: dev
  description: my dev environment
  values:
  - key: hostname
    value: localhost
  - key: app_port
    value: "8080"
  - key: pokemon_id
    value: "22"

tracetest get environment --id [the id]

Description: Delete an environment by ID
Required flags: --id
Optional Flags: none
Pretty Output Example:

Environment successfully deleted

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "Environment successfully deleted"
}

YAML Output Example:

---
sucess: true
message: Environment successfully deleted
errors: []

tracetest apply environment --file [path]

Description: Save an environment to the server. If it does not yet exists on the server, it is created. Otherwise it is updated.
Required flags: --file|-f input file
Optional Flags: none
Validation:

  • if the given file does not have a spec.id set: The resource file does not have a ID defined in the "spec" object. Please provide a unique one. For example, you can use [SHOW A RANDOM VALID ID]

Pretty Output Example:

Environment successfully applied

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "Environment successfully applied"
}

YAML Output Example:

---
sucess: true
message: Environment successfully applied
errors: []

tracetest run environment

Description: Environment is not runnable, so run is not available
Required flags: none
Optional Flags: none
Output: For any format, do not print anything to stdout, print to StdErr:

Environment does not support running. try `tracetest help run` for more information on the run command

@schoren
Copy link
Collaborator

schoren commented Mar 30, 2023

DataStore

tracetest get datastore

Description: get the currently active dataStore
Required flags: none
Optional Flags none
Pretty Output Example:

Type Settings
Jaeger - endpoint: jaeger-query.tracetest:16685
- tls.insecure: true

JSON Output Example:

{
  "type": "DataStore",
  "spec": {
    "id": "current",
    "name": "Current",
    "type": "jaeger",
    "default": true,
    "jaeger": {
      "endpoint": "jaeger-query.tracetest:16685",
      "tls": {
        "insecure": true
      }
    },
    "createdAt": "2023-03-09T17:53:10.256383Z"
  }
}

YAML Output example:

---
type: DataStore
spec:
  id: current
  name: Current
  type: jaeger
  default: true
  jaeger:
    endpoint: jaeger-query.tracetest:16685
    tls:
      insecure: true

tracetest list datastore

Description: DataStore is a single resource, so list will print only one item
Required flags: none
Optional Flags none
Output: For any format, print:

ID NAME DEFAULT
current jaeger *

tracetest delete datastore

Description: remove the configured DataStore and put tracetest in no-tracing mode
Required flags: none
Optional Flags none
Pretty Output Example:

DataStore removed. Defaulting back to no-tracing mode

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "DataStore removed. Defaulting back to no-tracing mode"
}

YAML Output Example:

---
sucess: true
message: DataStore removed. Defaulting back to no-tracing mode
errors: []

tracetest apply datastore --file [path]

Description: update the DataStore settings
Required flags: --file|-f input file
Optional Flags none
Validation:

  • if more than one document is present on the file: Only one document per file is allowed
  • if document's type is not DataStore: Invalid type [actual type]. Expected DataStore
    Pretty Output Example:
DataStore successfully updated

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "DataStore successfully updated"
}

YAML Output Example:

---
sucess: true
message: DataStore successfully updated
errors: []

tracetest run datastore

Description: DataStore is not runnable, so run is not available
Required flags: none
Optional Flags none
Output: For any format, do not print anything to stdout, print to StdErr:

DataStore does not support running. try `tracetest help run` for more information on the run command

@schoren
Copy link
Collaborator

schoren commented Mar 30, 2023

Test

tracetest get test --id [the id]

Description: Get a test by ID
Required flags: --id
Optional Flags: none
Pretty Output Example:

ID Name Version Trigger Type Runs Last Run Time Last Run Successes Last Run Fails URL
WZGNTOBVg Pokeshop - List 1 http 5 2hs ago 1 0 http://localhost:11633/test/WZGNTOBVg

JSON Output Example:

{
	"type": "Test",
	"spec": {
		"id": "uivSWYf4g",
		"name": "Pokeshop - List",
		"description": "Get a Pokemon",
		"version": 1,
		"createdAt": "2023-03-27T20:08:14.171787Z",
		"serviceUnderTest": {
			"triggerType": "http",
			"triggerSettings": {
				"http": {
					"url": "http://demo-pokemon-api.demo.svc.cluster.local/not-exists",
					"method": "GET",
					"headers": [{
						"key": "Content-Type",
						"value": "application/json"
					}]
				}
			}
		},
		"specs": {
			"specs": [{
				"name": "",
				"selector": {
					"query": "span[tracetest.span.type=\"general\" name=\"Tracetest trigger\"]",
					"structure": [{
						"filters": [{
								"property": "tracetest.span.type",
								"operator": "=",
								"value": "general"
							},
							{
								"property": "name",
								"operator": "=",
								"value": "Tracetest trigger"
							}
						]
					}]
				},
				"assertions": [
					"attr:tracetest.response.status = 200"
				]
			}]
		},
		"summary": {
			"runs": 5,
			"lastRun": {
				"time": "2023-03-27T20:08:14.366933Z",
				"passes": 1
			}
		}
	}
}

YAML Output example:

---
type: Test
spec:
  id: uivSWYf4g
  name: Pokeshop - List
  description: Get a Pokemon
  trigger:
    type: http
    http:
    	url: http://demo-pokemon-api.demo.svc.cluster.local/not-exists
      method: GET
      headers:
      - key: Content-Type
      value: application/json
  specs:
    - selector: span[tracetest.span.type="general" name="Tracetest trigger"]
      assertions:
      - attr:tracetest.response.status = 200

tracetest list test

Description: List Tests
Required flags: none
Optional Flags:

  • --all Get all results, not paginated. If passed, --take and --skip do nothing
  • --take N take N records [default: 20]
  • --skip N skip the first N records [default: 0]
  • --query get resources matching the given query string

Pretty Output Example:

Total avialable Tests: 2

ID Name Version Trigger Type Runs Last Run Time Last Run Successes Last Run Fails URL
WZGNTOBVg Pokeshop - List 1 http 5 2hs ago 1 0 http://localhost:11633/test/WZGNTOBVg
WZGNTOBVg Pokeshop - List 1 http 5 2hs ago 1 0 http://localhost:11633/test/WZGNTOBVg

JSON Output Example:

{
  "items": [
    {
      "type": "Test",
      "spec": {
        "id": "uivSWYf4g",
        "name": "Pokeshop - List",
        "description": "Get a Pokemon",
        "version": 1,
        "createdAt": "2023-03-27T20:08:14.171787Z",
        "serviceUnderTest": {
          "triggerType": "http",
          "triggerSettings": {
            "http": {
              "url": "http://demo-pokemon-api.demo.svc.cluster.local/not-exists",
              "method": "GET",
              "headers": [{
                "key": "Content-Type",
                "value": "application/json"
              }]
            }
          }
        },
        "specs": {
          "specs": [{
            "name": "",
            "selector": {
              "query": "span[tracetest.span.type=\"general\" name=\"Tracetest trigger\"]",
              "structure": [{
                "filters": [{
                    "property": "tracetest.span.type",
                    "operator": "=",
                    "value": "general"
                  },
                  {
                    "property": "name",
                    "operator": "=",
                    "value": "Tracetest trigger"
                  }
                ]
              }]
            },
            "assertions": [
              "attr:tracetest.response.status = 200"
            ]
          }]
        },
        "summary": {
          "runs": 5,
          "lastRun": {
            "time": "2023-03-27T20:08:14.366933Z",
            "passes": 1
          }
        }
      }
    },
    {
      "type": "Test",
      "spec": {
        "id": "uivSWYf4g",
        "name": "Pokeshop - List",
        "description": "Get a Pokemon",
        "version": 1,
        "createdAt": "2023-03-27T20:08:14.171787Z",
        "serviceUnderTest": {
          "triggerType": "http",
          "triggerSettings": {
            "http": {
              "url": "http://demo-pokemon-api.demo.svc.cluster.local/not-exists",
              "method": "GET",
              "headers": [{
                "key": "Content-Type",
                "value": "application/json"
              }]
            }
          }
        },
        "specs": {
          "specs": [{
            "name": "",
            "selector": {
              "query": "span[tracetest.span.type=\"general\" name=\"Tracetest trigger\"]",
              "structure": [{
                "filters": [{
                    "property": "tracetest.span.type",
                    "operator": "=",
                    "value": "general"
                  },
                  {
                    "property": "name",
                    "operator": "=",
                    "value": "Tracetest trigger"
                  }
                ]
              }]
            },
            "assertions": [
              "attr:tracetest.response.status = 200"
            ]
          }]
        },
        "summary": {
          "runs": 5,
          "lastRun": {
            "time": "2023-03-27T20:08:14.366933Z",
            "passes": 1
          }
        }
      }
    }
  ],

}

YAML Output example:

---
type: Test
spec:
  id: uivSWYf4g
  name: Pokeshop - List
  description: Get a Pokemon
  trigger:
    type: http
    http:
    	url: http://demo-pokemon-api.demo.svc.cluster.local/not-exists
      method: GET
      headers:
      - key: Content-Type
      value: application/json
  specs:
    - selector: span[tracetest.span.type="general" name="Tracetest trigger"]
      assertions:
      - attr:tracetest.response.status = 200
---
type: Test
spec:
  id: uivSWYf4g
  name: Pokeshop - List
  description: Get a Pokemon
  trigger:
    type: http
    http:
    	url: http://demo-pokemon-api.demo.svc.cluster.local/not-exists
      method: GET
      headers:
      - key: Content-Type
      value: application/json
  specs:
    - selector: span[tracetest.span.type="general" name="Tracetest trigger"]
      assertions:
      - attr:tracetest.response.status = 200

tracetest delete test --id [the id]

Description: Delete a test by ID
Required flags: --id
Optional Flags: none
Pretty Output Example:

Test successfully deleted

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "Test successfully deleted"
}

YAML Output Example:

---
sucess: true
message: Test successfully deleted
errors: []

tracetest apply test --file [path]

Description: Save a test to the server. If it does not yet exists on the server, it is created. Otherwise it is updated.
Input can be a file with one or more documents, or a directory with files with one or more documents. This is not transactional, so if an error happens, all changes to resources applied until the error are kept.
Required flags: --file|-f input file or directory
Optional Flags: none
Validation:

  • if the given file does not have a spec.id set: The resource file does not have a ID defined in the "spec" object. Please provide a unique one. For example, you can use [SHOW A RANDOM VALID ID]
  • if a document's type is not Test: Invalid type [actual type]. Expected Test

Pretty Output Example:

Test successfully applied

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "Test successfully applied"
}

YAML Output Example:

---
sucess: true
message: Test successfully applied
errors: []

tracetest run test --id [the id] or tracetest run test --file [path]

Description: Run a test by ID or from a definition file
Required flags: one of --id or --file|-f
Optional Flags:

  • --environment|-e [the id] environment ID to use
  • --wait-for-results|-w wait for results
  • --junit [path] save the results as junit in the given path. Requires --wait-for-results|-w

Validation:

  • if running from --file
    • if the given file does not have a spec.id set: The resource file does not have a ID defined in the "spec" object. Please provide a unique one. For example, you can use [SHOW A RANDOM VALID ID]

Output
Keep current behavior

@schoren
Copy link
Collaborator

schoren commented Mar 30, 2023

Transaction

tracetest get transaction --id [the id]

Description: Get a transaction by ID
Required flags: --id
Optional Flags: none
Pretty Output Example:

ID Name Version Steps Runs Last Run Time Last Run Successes Last Run Fails URL
RRlVDYB4R Shopping Cart 1 2 5 2hs ago 1 0 http://localhost:11633/transaction/RRlVDYB4R

JSON Output Example:

{
  "type": "Transaction",
  "spec": {
    "id": "RRlVDYB4R",
    "name": "Shopping Cart flow",
    "version": 1,
    "steps": [
      {
        "id": "uivSWYf4g",
        "name": "Login",
        "description": "Login",
        "version": 1,
        "createdAt": "2023-03-27T20:08:14.171787Z",
        "serviceUnderTest": {
          "triggerType": "http",
          "triggerSettings": {
            "http": {
              "url": "http://demo-pokemon-api.demo.svc.cluster.local/login",
              "method": "GET",
              "headers": [
                {
                  "key": "Content-Type",
                  "value": "application/json"
                }
              ]
            }
          }
        },
        "summary": {
          "runs": 2,
          "lastRun": {
            "time": "2023-03-27T21:52:50.978947Z"
          }
        }
      },
      {
        "id": "ZIM6hOf4g",
        "name": "Add to cart",
        "description": "add a product to the cart",
        "version": 1,
        "createdAt": "2023-03-23T17:01:51.203541Z",
        "serviceUnderTest": {
          "triggerType": "http",
          "triggerSettings": {
            "http": {
              "url": "http://demo-pokemon-api.demo.svc.cluster.local/pokemon?take=20&skip=0",
              "method": "GET",
              "headers": [
                {
                  "key": "Content-Type",
                  "value": "application/json"
                }
              ]
            }
          }
        },
        "summary": {
          "runs": 2,
          "lastRun": {
            "time": "2023-03-27T21:52:57.138436Z"
          }
        }
      }
    ],
    "createdAt": "2023-03-27T21:52:50.17695Z",
    "summary": {
      "runs": 5,
      "lastRun": {
        "time": "2023-03-27T21:52:50.872917Z"
      }
    }
  }
}

YAML Output example:

---
type: Transaction
spec:
  id: RRlVDYB4R
  name: Shopping Cart flow
  steps:
  - uivSWYf4g
  - ZIM6hOf4g

tracetest list transaction

Description: List Transactions
Required flags: none
Optional Flags:

  • --all Get all results, not paginated. If passed, --take and --skip do nothing
  • --take N take N records [default: 20]
  • --skip N skip the first N records [default: 0]
  • --query get resources matching the given query string

Pretty Output Example:

Total avialable Transactions: 2

ID Name Version Steps Runs Last Run Time Last Run Successes Last Run Fails URL
RRlVDYB4R Shopping Cart 1 2 5 2hs ago 1 0 http://localhost:11633/transaction/RRlVDYB4R
RRlVDYB4R Shopping Cart 1 2 5 2hs ago 1 0 http://localhost:11633/transaction/RRlVDYB4R

JSON Output Example:

{
  "count": 2,
  "items": {
    {
      "type": "Transaction",
      "spec": {
        "id": "RRlVDYB4R",
        "name": "Shopping Cart flow",
        "version": 1,
        "steps": [
          {
            "id": "uivSWYf4g",
            "name": "Login",
            "description": "Login",
            "version": 1,
            "createdAt": "2023-03-27T20:08:14.171787Z",
            "serviceUnderTest": {
              "triggerType": "http",
              "triggerSettings": {
                "http": {
                  "url": "http://demo-pokemon-api.demo.svc.cluster.local/login",
                  "method": "GET",
                  "headers": [
                    {
                      "key": "Content-Type",
                      "value": "application/json"
                    }
                  ]
                }
              }
            },
            "summary": {
              "runs": 2,
              "lastRun": {
                "time": "2023-03-27T21:52:50.978947Z"
              }
            }
          },
          {
            "id": "ZIM6hOf4g",
            "name": "Add to cart",
            "description": "add a product to the cart",
            "version": 1,
            "createdAt": "2023-03-23T17:01:51.203541Z",
            "serviceUnderTest": {
              "triggerType": "http",
              "triggerSettings": {
                "http": {
                  "url": "http://demo-pokemon-api.demo.svc.cluster.local/pokemon?take=20&skip=0",
                  "method": "GET",
                  "headers": [
                    {
                      "key": "Content-Type",
                      "value": "application/json"
                    }
                  ]
                }
              }
            },
            "summary": {
              "runs": 2,
              "lastRun": {
                "time": "2023-03-27T21:52:57.138436Z"
              }
            }
          }
        ],
        "createdAt": "2023-03-27T21:52:50.17695Z",
        "summary": {
          "runs": 5,
          "lastRun": {
            "time": "2023-03-27T21:52:50.872917Z"
          }
        }
      }
    },
    {
      "type": "Transaction",
      "spec": {
        "id": "RRlVDYB4R",
        "name": "Shopping Cart flow",
        "version": 1,
        "steps": [
          {
            "id": "uivSWYf4g",
            "name": "Login",
            "description": "Login",
            "version": 1,
            "createdAt": "2023-03-27T20:08:14.171787Z",
            "serviceUnderTest": {
              "triggerType": "http",
              "triggerSettings": {
                "http": {
                  "url": "http://demo-pokemon-api.demo.svc.cluster.local/login",
                  "method": "GET",
                  "headers": [
                    {
                      "key": "Content-Type",
                      "value": "application/json"
                    }
                  ]
                }
              }
            },
            "summary": {
              "runs": 2,
              "lastRun": {
                "time": "2023-03-27T21:52:50.978947Z"
              }
            }
          },
          {
            "id": "ZIM6hOf4g",
            "name": "Add to cart",
            "description": "add a product to the cart",
            "version": 1,
            "createdAt": "2023-03-23T17:01:51.203541Z",
            "serviceUnderTest": {
              "triggerType": "http",
              "triggerSettings": {
                "http": {
                  "url": "http://demo-pokemon-api.demo.svc.cluster.local/pokemon?take=20&skip=0",
                  "method": "GET",
                  "headers": [
                    {
                      "key": "Content-Type",
                      "value": "application/json"
                    }
                  ]
                }
              }
            },
            "summary": {
              "runs": 2,
              "lastRun": {
                "time": "2023-03-27T21:52:57.138436Z"
              }
            }
          }
        ],
        "createdAt": "2023-03-27T21:52:50.17695Z",
        "summary": {
          "runs": 5,
          "lastRun": {
            "time": "2023-03-27T21:52:50.872917Z"
          }
        }
      }
    }
  }
}

YAML Output example:

---
type: Transaction
spec:
  id: RRlVDYB4R
  name: Shopping Cart flow
  steps:
  - uivSWYf4g
  - ZIM6hOf4g
---
type: Transaction
spec:
  id: RRlVDYB4R
  name: Shopping Cart flow
  steps:
  - uivSWYf4g
  - ZIM6hOf4g

tracetest delete transaction --id [the id]

Description: Delete a transaction by ID
Required flags: --id
Optional Flags: none
Pretty Output Example:

Transaction successfully deleted

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "Transaction successfully deleted"
}

YAML Output Example:

---
sucess: true
message: Transaction successfully deleted
errors: []

tracetest apply transaction --file [path]

Description: Save a transaction to the server. If it does not yet exists on the server, it is created. Otherwise it is updated.Input can be a file with one or more documents, or a directory with files with one or more documents. This is not transactional, so if an error happens, all changes to resources applied until the error are kept.
Required flags: --file|-f input file or directory
Optional Flags: none
Validation:

  • if the given file does not have a spec.id set: The resource file does not have a ID defined in the "spec" object. Please provide a unique one. For example, you can use [SHOW A RANDOM VALID ID]
  • if a document's type is not Transaction: Invalid type [actual type]. Expected Transaction

Pretty Output Example:

Transaction successfully applied

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "Transaction successfully applied"
}

YAML Output Example:

---
sucess: true
message: Transaction successfully applied
errors: []

tracetest run transaction --id [the id] or tracetest run transaction --file [path]

Description: Run a transaction by ID or from a definition file
Required flags: one of --id or --file|-f
Optional Flags:

  • --environment|-e [the id] environment ID to use
  • --wait-for-results|-w wait for results
  • --junit [path] save the results as junit in the given path. Requires --wait-for-results|-w

Validation:

  • if running from --file
    • if the given file does not have a spec.id set: The resource file does not have a ID defined in the "spec" object. Please provide a unique one. For example, you can use [SHOW A RANDOM VALID ID]

Pretty Output Example:
✅ Transaction [transaction ID] successfully executed.

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "Transaction [transaction ID] successfully executed."
}

YAML Output Example:

sucess: true
message: Transaction [transaction ID] successfully executed.
errors: []

@schoren
Copy link
Collaborator

schoren commented Mar 30, 2023

All Resources

tracetest list all

Description: List all resources
Required flags: none
Optional Flags:

  • --all Get all results, not paginated. If passed, --take and --skip do nothing
  • --take N take N records [default: 20]
  • --skip N skip the first N records [default: 0]
  • --query get resources matching the given query string

Pretty Output Example:

Total avialable Resources: 4

Type ID Name Details
Config current Current - analytics enabled: yes
DataStore current Current - type: jaeger
- endpoint: jaeger-query.tracetest:16685
- tls.insecure: true
Test WZGNTOBVg Pokeshop - List - url: http://localhost:11633/test/RRlVDYB4R
- trigger type: http
Transaction RRlVDYB4R Shopping Cart - url: http://localhost:11633/transaction/RRlVDYB4R

JSON Output Example:

{
  "count": 4,
  "items": {
    {
      "type": "Config",
      "spec": {
        "id": "current",
        "name": "Current",
        "analyticsEnabled": true
      }
    },
    {
      "type": "DataStore",
      "spec": {
        "id": "current",
        "name": "Current",
        "type": "jaeger",
        "isDefault": true,
        "jaeger": {
          "endpoint": "jaeger-query.tracetest:16685",
          "tls": {
            "insecure": true
          }
        },
        "createdAt": "2023-03-09T17:53:10.256383Z"
      }
    },
    {
      "type": "Transaction",
      "spec": {
        "id": "RRlVDYB4R",
        "name": "Shopping Cart flow",
        "version": 1,
        "steps": [
          {
            "id": "uivSWYf4g",
            "name": "Login",
            "description": "Login",
            "version": 1,
            "createdAt": "2023-03-27T20:08:14.171787Z",
            "serviceUnderTest": {
              "triggerType": "http",
              "triggerSettings": {
                "http": {
                  "url": "http://demo-pokemon-api.demo.svc.cluster.local/login",
                  "method": "GET",
                  "headers": [
                    {
                      "key": "Content-Type",
                      "value": "application/json"
                    }
                  ]
                }
              }
            },
            "summary": {
              "runs": 2,
              "lastRun": {
                "time": "2023-03-27T21:52:50.978947Z"
              }
            }
          },
          {
            "id": "ZIM6hOf4g",
            "name": "Add to cart",
            "description": "add a product to the cart",
            "version": 1,
            "createdAt": "2023-03-23T17:01:51.203541Z",
            "serviceUnderTest": {
              "triggerType": "http",
              "triggerSettings": {
                "http": {
                  "url": "http://demo-pokemon-api.demo.svc.cluster.local/pokemon?take=20&skip=0",
                  "method": "GET",
                  "headers": [
                    {
                      "key": "Content-Type",
                      "value": "application/json"
                    }
                  ]
                }
              }
            },
            "summary": {
              "runs": 2,
              "lastRun": {
                "time": "2023-03-27T21:52:57.138436Z"
              }
            }
          }
        ],
        "createdAt": "2023-03-27T21:52:50.17695Z",
        "summary": {
          "runs": 5,
          "lastRun": {
            "time": "2023-03-27T21:52:50.872917Z"
          }
        }
      }
    },
    {
      "type": "Test",
      "spec": {
        "id": "uivSWYf4g",
        "name": "Pokeshop - List",
        "description": "Get a Pokemon",
        "version": 1,
        "createdAt": "2023-03-27T20:08:14.171787Z",
        "serviceUnderTest": {
          "triggerType": "http",
          "triggerSettings": {
            "http": {
              "url": "http://demo-pokemon-api.demo.svc.cluster.local/not-exists",
              "method": "GET",
              "headers": [{
                "key": "Content-Type",
                "value": "application/json"
              }]
            }
          }
        },
        "specs": {
          "specs": [{
            "name": "",
            "selector": {
              "query": "span[tracetest.span.type=\"general\" name=\"Tracetest trigger\"]",
              "structure": [{
                "filters": [{
                    "property": "tracetest.span.type",
                    "operator": "=",
                    "value": "general"
                  },
                  {
                    "property": "name",
                    "operator": "=",
                    "value": "Tracetest trigger"
                  }
                ]
              }]
            },
            "assertions": [
              "attr:tracetest.response.status = 200"
            ]
          }]
        },
        "summary": {
          "runs": 5,
          "lastRun": {
            "time": "2023-03-27T20:08:14.366933Z",
            "passes": 1
          }
        }
      }
    }
  }
}

YAML Output example:

---
type: Config
spec:
  id: current
  name: Current
  analyticsEnabled: true
---
type: DataStore
spec:
  id: current
  name: Current
  type: jaeger
  isDefault: true
  jaeger:
    endpoint: jaeger-query.tracetest:16685
    tls:
      insecure: true
---
type: Test
spec:
  id: uivSWYf4g
  name: Pokeshop - List
  description: Get a Pokemon
  trigger:
    type: http
    http:
    	url: http://demo-pokemon-api.demo.svc.cluster.local/not-exists
      method: GET
      headers:
      - key: Content-Type
      value: application/json
  specs:
    - selector: span[tracetest.span.type="general" name="Tracetest trigger"]
      assertions:
      - attr:tracetest.response.status = 200
---
type: Transaction
spec:
  id: RRlVDYB4R
  name: Shopping Cart flow
  steps:
  - uivSWYf4g
  - uivSWYf4g

tracetest delete all

Description: Delete all resources in the Tracetest server, resetting it to a default empty state
Required flags: none
Optional Flags: --yes|-y
Confirmation

  • if running on an interactive environment (env CI unset, empty, or otherwise false-y), start an interaction:
Are you sure you want to delete all resources in the server? 
You will lose all tests, transactions, results, environments and configurations.
This cannot be reverted.

  Yes
> No
  • if running on a non-interactive environment (env CI set to truth-y), the --yes flag is required.
    if it is not present, show an error: You need to confirm this action. use the "--yes" flag to proceed

Pretty Output Example:

Tracetest successfully reset

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "Tracetest successfully reset"
}

YAML Output Example:

---
sucess: true
message: Tracetest successfully reset
errors: []

tracetest apply all --file [path]

Description: Applies all resources in the given file or directory. This is not transactional, so if an error happens, all changes to resources applied until the error are kept.
Required flags: --file|-f input file
Optional Flags: none
Validation:

  • for each resource:
    • if the given document does not have a spec.id set: The resource number [index of the document] in the [path to file] file does not have a ID defined in the "spec" object. Please provide a unique one. For example, you can use [SHOW A RANDOM VALID ID]

Pretty Output Example:

Resources successfully applied

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "Resources successfully applied"
}

YAML Output Example:

---
sucess: true
message: Resources successfully applied
errors: []

@schoren
Copy link
Collaborator

schoren commented Mar 30, 2023

Dashboard

tracetest dashboard

Description: Open the web UI dashboard in the default browser
Required flags: none
Optional Flags: none
Output: none to StdOut, print to StdErr

Opening "[server url]" in the default browser

@olha23
Copy link

olha23 commented Apr 5, 2023

Transaction

tracetest get transaction --id [the id]

Description: Get a transaction by ID Required flags: --id Optional Flags: none Pretty Output Example:

ID Name Version Steps Runs Last Run Time Last Run Successes Last Run Fails URL
RRlVDYB4R Shopping Cart 1 2 5 2hs ago 1 0 http://localhost:11633/transaction/RRlVDYB4R
JSON Output Example:

{
  "type": "Transaction",
  "spec": {
    "id": "RRlVDYB4R",
    "name": "Shopping Cart flow",
    "version": 1,
    "steps": [
      {
        "id": "uivSWYf4g",
        "name": "Login",
        "description": "Login",
        "version": 1,
        "createdAt": "2023-03-27T20:08:14.171787Z",
        "serviceUnderTest": {
          "triggerType": "http",
          "triggerSettings": {
            "http": {
              "url": "http://demo-pokemon-api.demo.svc.cluster.local/login",
              "method": "GET",
              "headers": [
                {
                  "key": "Content-Type",
                  "value": "application/json"
                }
              ]
            }
          }
        },
        "summary": {
          "runs": 2,
          "lastRun": {
            "time": "2023-03-27T21:52:50.978947Z"
          }
        }
      },
      {
        "id": "ZIM6hOf4g",
        "name": "Add to cart",
        "description": "add a product to the cart",
        "version": 1,
        "createdAt": "2023-03-23T17:01:51.203541Z",
        "serviceUnderTest": {
          "triggerType": "http",
          "triggerSettings": {
            "http": {
              "url": "http://demo-pokemon-api.demo.svc.cluster.local/pokemon?take=20&skip=0",
              "method": "GET",
              "headers": [
                {
                  "key": "Content-Type",
                  "value": "application/json"
                }
              ]
            }
          }
        },
        "summary": {
          "runs": 2,
          "lastRun": {
            "time": "2023-03-27T21:52:57.138436Z"
          }
        }
      }
    ],
    "createdAt": "2023-03-27T21:52:50.17695Z",
    "summary": {
      "runs": 5,
      "lastRun": {
        "time": "2023-03-27T21:52:50.872917Z"
      }
    }
  }
}

YAML Output example:

---
type: Transaction
spec:
  id: RRlVDYB4R
  name: Shopping Cart flow
  steps:
  - uivSWYf4g
  - ZIM6hOf4g

tracetest list transaction

Description: List Transactions Required flags: none Optional Flags:

  • --all Get all results, not paginated. If passed, --take and --skip do nothing
  • --take N take N records [default: 20]
  • --skip N skip the first N records [default: 0]
  • --query get resources matching the given query string

Pretty Output Example:

Total avialable Transactions: 2

ID Name Version Steps Runs Last Run Time Last Run Successes Last Run Fails URL
RRlVDYB4R Shopping Cart 1 2 5 2hs ago 1 0 http://localhost:11633/transaction/RRlVDYB4R
RRlVDYB4R Shopping Cart 1 2 5 2hs ago 1 0 http://localhost:11633/transaction/RRlVDYB4R
JSON Output Example:

{
  "count": 2,
  "items": {
    {
      "type": "Transaction",
      "spec": {
        "id": "RRlVDYB4R",
        "name": "Shopping Cart flow",
        "version": 1,
        "steps": [
          {
            "id": "uivSWYf4g",
            "name": "Login",
            "description": "Login",
            "version": 1,
            "createdAt": "2023-03-27T20:08:14.171787Z",
            "serviceUnderTest": {
              "triggerType": "http",
              "triggerSettings": {
                "http": {
                  "url": "http://demo-pokemon-api.demo.svc.cluster.local/login",
                  "method": "GET",
                  "headers": [
                    {
                      "key": "Content-Type",
                      "value": "application/json"
                    }
                  ]
                }
              }
            },
            "summary": {
              "runs": 2,
              "lastRun": {
                "time": "2023-03-27T21:52:50.978947Z"
              }
            }
          },
          {
            "id": "ZIM6hOf4g",
            "name": "Add to cart",
            "description": "add a product to the cart",
            "version": 1,
            "createdAt": "2023-03-23T17:01:51.203541Z",
            "serviceUnderTest": {
              "triggerType": "http",
              "triggerSettings": {
                "http": {
                  "url": "http://demo-pokemon-api.demo.svc.cluster.local/pokemon?take=20&skip=0",
                  "method": "GET",
                  "headers": [
                    {
                      "key": "Content-Type",
                      "value": "application/json"
                    }
                  ]
                }
              }
            },
            "summary": {
              "runs": 2,
              "lastRun": {
                "time": "2023-03-27T21:52:57.138436Z"
              }
            }
          }
        ],
        "createdAt": "2023-03-27T21:52:50.17695Z",
        "summary": {
          "runs": 5,
          "lastRun": {
            "time": "2023-03-27T21:52:50.872917Z"
          }
        }
      }
    },
    {
      "type": "Transaction",
      "spec": {
        "id": "RRlVDYB4R",
        "name": "Shopping Cart flow",
        "version": 1,
        "steps": [
          {
            "id": "uivSWYf4g",
            "name": "Login",
            "description": "Login",
            "version": 1,
            "createdAt": "2023-03-27T20:08:14.171787Z",
            "serviceUnderTest": {
              "triggerType": "http",
              "triggerSettings": {
                "http": {
                  "url": "http://demo-pokemon-api.demo.svc.cluster.local/login",
                  "method": "GET",
                  "headers": [
                    {
                      "key": "Content-Type",
                      "value": "application/json"
                    }
                  ]
                }
              }
            },
            "summary": {
              "runs": 2,
              "lastRun": {
                "time": "2023-03-27T21:52:50.978947Z"
              }
            }
          },
          {
            "id": "ZIM6hOf4g",
            "name": "Add to cart",
            "description": "add a product to the cart",
            "version": 1,
            "createdAt": "2023-03-23T17:01:51.203541Z",
            "serviceUnderTest": {
              "triggerType": "http",
              "triggerSettings": {
                "http": {
                  "url": "http://demo-pokemon-api.demo.svc.cluster.local/pokemon?take=20&skip=0",
                  "method": "GET",
                  "headers": [
                    {
                      "key": "Content-Type",
                      "value": "application/json"
                    }
                  ]
                }
              }
            },
            "summary": {
              "runs": 2,
              "lastRun": {
                "time": "2023-03-27T21:52:57.138436Z"
              }
            }
          }
        ],
        "createdAt": "2023-03-27T21:52:50.17695Z",
        "summary": {
          "runs": 5,
          "lastRun": {
            "time": "2023-03-27T21:52:50.872917Z"
          }
        }
      }
    }
  }
}

YAML Output example:

---
type: Transaction
spec:
  id: RRlVDYB4R
  name: Shopping Cart flow
  steps:
  - uivSWYf4g
  - ZIM6hOf4g
---
type: Transaction
spec:
  id: RRlVDYB4R
  name: Shopping Cart flow
  steps:
  - uivSWYf4g
  - ZIM6hOf4g

tracetest delete transaction --id [the id]

Description: Delete a transaction by ID Required flags: --id Optional Flags: none Pretty Output Example:

Transaction successfully deleted

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "Transaction successfully deleted"
}

YAML Output Example:

---
sucess: true
message: Transaction successfully deleted
errors: []

tracetest apply transaction --file [path]

Description: Save a transaction to the server. If it does not yet exists on the server, it is created. Otherwise it is updated.Input can be a file with one or more documents, or a directory with files with one or more documents. This is not transactional, so if an error happens, all changes to resources applied until the error are kept. Required flags: --file|-f input file or directory Optional Flags: none Validation:

  • if the given file does not have a spec.id set: The resource file does not have a ID defined in the "spec" object. Please provide a unique one. For example, you can use [SHOW A RANDOM VALID ID]
  • if a document's type is not Transaction: Invalid type [actual type]. Expected Transaction

Pretty Output Example:

Transaction successfully applied

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "Transaction successfully applied"
}

YAML Output Example:

---
sucess: true
message: Transaction successfully applied
errors: []

tracetest run transaction --id [the id] or tracetest run transaction --definition [path]

Description: Run a transaction by ID or from a definition Required flags: one of --id or --defintion|-d Optional Flags:

  • --environment|-e [the id] environment ID to use
  • --wait-for-results|-w wait for results
  • --junit [path] save the results as junit in the given path. Requires --wait-for-results|-w

Validation:

  • if running from --definition

    • if the given file does not have a spec.id set: The resource file does not have a ID defined in the "spec" object. Please provide a unique one. For example, you can use [SHOW A RANDOM VALID ID]

Output TODO (@olha23 any input appreciated)

@schoren i think OUTPUT can be:
Pretty Output Example:
✅ Transaction [transaction ID] successfully executed.

JSON Output Example:

{
  "success": true,
  "errors": [],
  "message": "Transaction [transaction ID] successfully executed."
}

YAML Output Example:

sucess: true
message: Transaction [transaction ID] successfully executed.
errors: []

@schoren
Copy link
Collaborator

schoren commented Apr 5, 2023

@olha23 thanks! We would still need to handle the --wait-for-result outpout. for the tests, it waits for the test to be done (shortly to be updated by a live update as things happen) and shows a list of assertions with success/fails etc.

Transactions are sequential, meaning a test starts, finishes, and then the next one is started. We could do something like:

Runnning test [testID/name] #1st test
# Show the same output as `tracetest run test --id 123` would

Runnning test [testID/name] # 2nd test
# Show the same output as `tracetest run test --id 123` would

What could we show if there is an error between test 1 and test2, for example? like what the output would look like?

This is just for the pretty output, other formats can wait for the transaction to finish and show the encoded raw information, as it does with other resources.

@olha23
Copy link

olha23 commented Apr 5, 2023

@olha23 thanks! We would still need to handle the --wait-for-result outpout. for the tests, it waits for the test to be done (shortly to be updated by a live update as things happen) and shows a list of assertions with success/fails etc.

Transactions are sequential, meaning a test starts, finishes, and then the next one is started. We could do something like:

Runnning test [testID/name] #1st test
# Show the same output as `tracetest run test --id 123` would

Runnning test [testID/name] # 2nd test
# Show the same output as `tracetest run test --id 123` would

What could we show if there is an error between test 1 and test2, for example? like what the output would look like?

This is just for the pretty output, other formats can wait for the transaction to finish and show the encoded raw information, as it does with other resources.

Maybe something like this:

`tracetest run transaction --id [transactionID/name] --wait-for-results

🏃 Running transaction [transactionID/name]

🏃 Running test [testID/name] #1
✅ Assertion 1 passed
❌ Assertion 2 failed: Expected "ABC", but got "DEF"

❌ Transaction failed: Test [testID/name] has failed.

🏃 Running test [testID/name] #2
✅ Assertion 1 passed
✅ Assertion 2 passed

✅ Transaction completed successfully.
`

we can use emojis to represent the different statuses and actions in the output. This makes it easier to quickly scan the output and understand the results of the transaction and each test case.

@schoren what do you think?

@schoren
Copy link
Collaborator

schoren commented Apr 5, 2023

@olha23 makes sense to me. Would love @kdhamric feedback here, but it looks good.

@kdhamric
Copy link
Collaborator

kdhamric commented Apr 5, 2023

It looks good. A few things:

  • The example did not look quite right, as I am assuming if one of the tests in the transaction fails the entire transaction fails.
  • Would it be useful to add some indentation and maybe some dashes to provide some logical grouping so your eye can easy pick out what is part of a test, what is a transaction.
    I took a stab at both items below - not sure if it is right. (@olha23 may be able to improve)

`tracetest run transaction --id [transactionID/name] --wait-for-results

🏃 Running transaction [transactionID/name]

🏃 Running test [testID/name] #1
✅ Assertion 1 passed
❌ Assertion 2 failed: Expected "ABC", but got "DEF"

❌ Test [testID/name] has failed.

🏃 Running test [testID/name] #2
✅ Assertion 1 passed
✅ Assertion 2 passed

✅ Test [testID/name] #2 has passed.

===================================
❌ Transaction [transactionID/name] has failed.

@olha23
Copy link

olha23 commented Apr 6, 2023

@schoren @kdhamric couple options:

first:

`tracetest run transaction --id [transactionID/name] --wait-for-results

🏃 Running transaction [transactionID/name]
------------------------------------------------

🏃 Running test [testID/name] #1
  ----------------------------------------
  ✅ Assertion 1 passed
  ❌ Assertion 2 failed: Expected "ABC", but got "DEF"

  ❌ Test [testID/name] has failed.

🏃 Running test [testID/name] #2
  ----------------------------------------
  ✅ Assertion 1 passed
  ✅ Assertion 2 passed

  ✅ Test [testID/name] #2 has passed.
  
  
❌ Transaction [testID/name] has failed.
`

Second:

`tracetest run transaction --id [transactionID/name] --wait-for-results

🏃 Running transaction [transactionID/name]

------ Test [testID/name] #1 ------
✅ Assertion 1 passed
❌ Assertion 2 failed: Expected "ABC", but got "DEF"

❌ Test [testID/name] #1 failed.

------ Test [testID/name] #2 ------
✅ Assertion 1 passed
✅ Assertion 2 passed

✅ Test [testID/name] #2 has passed.

❌ Transaction [testID/name] has failed.
`

@xoscar
Copy link
Collaborator Author

xoscar commented Apr 10, 2023

@schoren By reading through the issues, I think we can separate the tasks into three types:

  1. Standardize what we have (new arch/verbs, run command, outputs)
  2. Deprecating old architecture/verbs
  3. Adding new features (version mismatch, multi-file support, dashboard command)

That way we can identify dependencies, possibly have multiple releases, and most importantly the team is aware of what can be started right away

@kdhamric
Copy link
Collaborator

@olha23 I think I like the second choice for output when running a transaction.

@schoren
Copy link
Collaborator

schoren commented Jul 17, 2023

We're closing this epic as it's mostly done. The remaining tasks can be tracked independantly.

@schoren schoren closed this as completed Jul 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

7 participants