This repository has been archived by the owner on Jun 5, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
af330fa
commit df003f8
Showing
9 changed files
with
5,230 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
# EditorConfig helps developers define and maintain consistent | ||
# coding styles between different editors and IDEs | ||
# editorconfig.org | ||
|
||
root = true | ||
|
||
[*] | ||
|
||
indent_style = space | ||
indent_size = 4 | ||
|
||
end_of_line = lf | ||
charset = utf-8 | ||
trim_trailing_whitespace = true | ||
insert_final_newline = true | ||
|
||
[{.babelrc,.eslintrc,.codeclimate.yml,.travis.yml,**/*.json,.eslintrc.js,.github/**/*.yml}] | ||
indent_size = 2 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -102,3 +102,6 @@ dist | |
|
||
# TernJS port file | ||
.tern-port | ||
|
||
# MacOS | ||
.DS_Store |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
src | ||
test | ||
yarn.lock | ||
*.log | ||
package-lock.json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,124 @@ | ||
# testcafe-reporter-prometheus-multi | ||
Prometheus reporter for TestCafe allowing for alerts based on multiple suite executions across time. | ||
|
||
Prometheus reporter for [TestCafe](https://devexpress.github.io/testcafe/) allowing for alerts based on multiple suite executions across time. | ||
|
||
## Installation | ||
|
||
```sh | ||
$ npm install --save-exact --save-dev testcafe-reporter-prometheus-multi | ||
``` | ||
|
||
## Usage | ||
|
||
### Setup | ||
|
||
Specify the reporter in your TestCafe config, e.g. | ||
|
||
```json | ||
reporter: [ | ||
"spec", | ||
{ | ||
"name": "prometheus-multi", | ||
"output": "report.txt" | ||
} | ||
] | ||
``` | ||
|
||
Use option `-r prometheus-multi:report.txt` if you're running TestCafe from command line. | ||
|
||
### Labeling tests | ||
|
||
The reporter makes use of _fixtures_' and _tests_' `meta.app` and `meta.owner`. | ||
If `app` or `owner` is not specified for a _test_, the `meta` from the _fixture_ is used. | ||
|
||
#### Example fixture with correct `meta` tags | ||
```javascript | ||
fixture('My fixture') | ||
.meta({ | ||
app: 'my-app', | ||
owner: 'me', | ||
}) | ||
.page(...); | ||
|
||
test('some test about my-app', async (testcafe) => { | ||
... | ||
}); | ||
|
||
test.meta({ app: 'someone-elses-app' })('a test about someone-elses-app', async (testcafe) => { | ||
... | ||
}); | ||
|
||
test.meta({ app: 'someone-elses-app', owner: 'someone-else' })('a test about someone-elses-app with a different owner', async (testcafe) => { | ||
... | ||
}); | ||
``` | ||
|
||
For more information, refer to [TestCafe documentation](https://devexpress.github.io/testcafe/documentation/using-testcafe/common-concepts/reporters.html#using-the-reporters). | ||
|
||
### Sending data to Prometheus | ||
|
||
The easiest way to send the report to Prometheus, is via [Pushgateway](https://github.com/prometheus/pushgateway). Just `curl` the report file to the Pushgateway endpoint: | ||
|
||
```sh | ||
curl --data-binary @report.txt http://address.of/pushgateway | ||
``` | ||
|
||
## Output format | ||
|
||
The aim of this project is for the report format to be compatible with [Prometheus data model](https://prometheus.io/docs/concepts/data_model/). Output contains the number of _passed_, _skipped_ and _failed_ tests for every combination of `app` and `owner` from the test suite. | ||
|
||
Metric names (`e2e_passed_tests`, `e2e_skipped_tests`, `e2e_failed_tests`) are hardcoded, because TestCafe doesn't provide an option to pass any parameters to the reporter. If you need to use different metric names, you'll have to fork the project and create a new node.js package. | ||
|
||
#### Example | ||
``` | ||
# HELP e2e_passed_tests Passed tests | ||
# TYPE e2e_passed_tests gauge | ||
e2e_passed_tests{app="my-app",owner="me"} 1 | ||
e2e_passed_tests{app="someone-elses-app",owner="someone-else"} 1 | ||
e2e_passed_tests{app="my-app",owner="someone-else"} 1 | ||
# HELP e2e_skipped_tests Skipped tests | ||
# TYPE e2e_skipped_tests gauge | ||
e2e_skipped_tests{app="my-app",owner="me"} 0 | ||
e2e_skipped_tests{app="someone-elses-app",owner="someone-else"} 0 | ||
e2e_skipped_tests{app="my-app",owner="someone-else"} 0 | ||
# HELP e2e_failed_tests Failed tests | ||
# TYPE e2e_failed_tests gauge | ||
e2e_failed_tests{app="my-app",owner="me"} 1 | ||
e2e_failed_tests{app="someone-elses-app",owner="someone-else"} 0 | ||
e2e_failed_tests{app="my-app",owner="someone-else"} 1 | ||
``` | ||
|
||
## Development | ||
|
||
Use NPM [`link`](https://docs.npmjs.com/cli/link) command to make this package available for use on your local machine. | ||
|
||
First, link this package to your global `node_modules` folder: | ||
|
||
```sh | ||
cd /path/to/testcafe-reporter-prometheus-multi | ||
npm install | ||
npm link | ||
``` | ||
|
||
Then, go to your project with the test suite and link the package from the global folder: | ||
|
||
```sh | ||
cd /path/to/your-project-with-tests | ||
npm link testcafe-reporter-prometheus-multi # or your changed package name | ||
``` | ||
|
||
## FAQ | ||
|
||
#### Why is the name so long? | ||
|
||
TestCafe requires all reporters to have `testcafe-reporter-` prefix, and `testcafe-reporter-prometheus` is [already taken](https://www.npmjs.com/package/testcafe-reporter-prometheus). | ||
|
||
#### Why is the package not owned by saucelabs in NPM? | ||
|
||
Same as above, all reporters must have `testcafe-reporter-` prefix. `@saucelabs/testcafe-reporter-` is no good. 😞 | ||
|
||
#### Can I customize the metric names? | ||
|
||
No, TestCafe currently doesn't offer a possibility to pass parameters to reporters, but you can always fork the project. |
Oops, something went wrong.