Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bench mark report #280

Merged
merged 46 commits into from
Apr 7, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
40c22e0
test(raw data changes): add bench marks end point
OrFrenkelZooz Mar 22, 2020
736c36b
test(unit-tests): add unit-tests bench mark
OrFrenkelZooz Mar 31, 2020
a56e1dc
test(unit-tests): fix change
OrFrenkelZooz Mar 31, 2020
829cd38
Merge branch 'master' into banch-mark
OrFrenkelZooz Mar 31, 2020
594e5fd
test(): add integration tests
OrFrenkelZooz Mar 31, 2020
3a38dc8
test(integration test): add integration tests and swagger validations
OrFrenkelZooz Mar 31, 2020
956ab34
test(integration test): add unit-tests
OrFrenkelZooz Mar 31, 2020
db5e642
test(integration test): temp commit
OrFrenkelZooz Mar 31, 2020
f115e70
test(temp push): temp push
OrFrenkelZooz Apr 1, 2020
17b9cbc
test(fix): fix typo
OrFrenkelZooz Apr 1, 2020
2e94c2b
test(fix): fix typo and all review changes
OrFrenkelZooz Apr 1, 2020
63b0aa9
Merge branch 'banch-mark' into bench-mark-report
OrFrenkelZooz Apr 1, 2020
fe0e4dd
test(fix): temp push reset mac
OrFrenkelZooz Apr 2, 2020
f16ff6e
Merge branch 'master' into bench-mark-report
OrFrenkelZooz Apr 5, 2020
9a1c6af
feat(benchmark): add benchmark
OrFrenkelZooz Apr 5, 2020
745d38d
feat(benchmark): fix lint
OrFrenkelZooz Apr 5, 2020
5529aed
test(unit-tests): add unit-tests
OrFrenkelZooz Apr 5, 2020
a454730
test(integration tests): fix swagger for integration tests
OrFrenkelZooz Apr 5, 2020
62f4e42
test(integration tests): fix swagger for integration tests
OrFrenkelZooz Apr 5, 2020
7223a1a
feat(code review): code review cahnges and add calaualte func
OrFrenkelZooz Apr 6, 2020
d931d68
test(unit-tests): fix unit-tests
OrFrenkelZooz Apr 6, 2020
fb6bb00
test(unit-tests): fix unit-tests
OrFrenkelZooz Apr 6, 2020
abded93
test(unit-tests): fix unit-tests
OrFrenkelZooz Apr 6, 2020
e51d011
test(unit-tests): add integration tests
OrFrenkelZooz Apr 6, 2020
8c30db1
test(unit-tests): fix integration tests
OrFrenkelZooz Apr 6, 2020
9e75918
test(unit-tests): add integration tests
OrFrenkelZooz Apr 6, 2020
1c1ad7b
test(unit-tests): add integration tests
OrFrenkelZooz Apr 6, 2020
64fce3b
feat(data-type): change type to blob
OrFrenkelZooz Apr 7, 2020
f2c5726
feat(data-type): change type Sequelize.DataTypes.TEXT('long')
OrFrenkelZooz Apr 7, 2020
d4583ba
test(sqllite): fix sqlite
OrFrenkelZooz Apr 7, 2020
59c12eb
feat(bench-mark): add default value to bench_mark config
OrFrenkelZooz Apr 7, 2020
9af9943
test(sqllite): fix last report score
OrFrenkelZooz Apr 7, 2020
c08df43
test(sqllite): fix last report score
OrFrenkelZooz Apr 7, 2020
b14418f
test(sqllite): fix last report score
OrFrenkelZooz Apr 7, 2020
2c2fc14
test(sqllite): revert marked changes
OrFrenkelZooz Apr 7, 2020
1d55280
test(): triger push again
OrFrenkelZooz Apr 7, 2020
b604cbb
test(): remove comments
OrFrenkelZooz Apr 7, 2020
b04d47e
test(): fix config unit tests
OrFrenkelZooz Apr 7, 2020
3b39f33
feat(): change to percentile_ninety_five
OrFrenkelZooz Apr 7, 2020
a68dd42
Merge branch 'default-config-benchmark' into bench-mark-report
OrFrenkelZooz Apr 7, 2020
1ef217a
feat(): change to percentile_ninety_five
OrFrenkelZooz Apr 7, 2020
4dc11f0
feat(): change to percentile_ninety_five
OrFrenkelZooz Apr 7, 2020
20e079b
feat(): change to percentile_ninety_five
OrFrenkelZooz Apr 7, 2020
65895d2
Merge branch 'default-config-benchmark' into bench-mark-report
OrFrenkelZooz Apr 7, 2020
1edc554
feat(): change to percentile_ninety_five
OrFrenkelZooz Apr 7, 2020
961d665
feat(): change to percentile_ninety_five
OrFrenkelZooz Apr 7, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 55 additions & 15 deletions docs/devguide/docs/swagger-docs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -319,6 +319,40 @@ paths:
$ref: '#/components/schemas/benchmark_request'
description: The bench mark to add
required: true
get:
enudler marked this conversation as resolved.
Show resolved Hide resolved
tags:
- Tests
parameters:
- in: path
name: test_id
description: The test id.
required: true
schema:
type: string
format: uuid
example: 4bf5d7ab-f310-4a64-8ec2-d65c06188ec1
summary: get a bench mark for test
description: Get a benchmark for a test
responses:
'200':
description: Success
content:
application/json:
schema:
$ref: '#/components/schemas/benchmark_request'
'404':
description: Not found
content:
application/json:
schema:
$ref: '#/components/schemas/error_response'
'500':
description: Internal server error
content:
application/json:
schema:
$ref: '#/components/schemas/error_response'

'/v1/tests/{test_id}':
get:
operationId: retrieve-a-test
Expand Down Expand Up @@ -1182,6 +1216,11 @@ components:
benchmark_request:
type: object
additionalProperties: false
required:
- errors
- codes
- rps
- latency
properties:
errors:
type: object
Expand All @@ -1196,24 +1235,22 @@ components:
500: 53
rps:
type: object
required:
- mean
properties:
count:
type: number
mean:
type: number
latency:
type: object
required:
- median
- p95
properties:
min:
type: number
max:
type: number
median:
type: number
p95:
type: number
p99:
type: number

dsl:
description: A test that is made of scenarios base on domain specific language
allOf:
Expand Down Expand Up @@ -1770,6 +1807,13 @@ components:
type: number
description: The duration of the test in seconds.
example: 60
score:
type: number
description: The score of a report, calculated based on a benchmark and benchmark weights
example: 9.25
benchmark_weights_data:
type: object
description: The way score calualated
arrival_rate:
type: number
description: The arrival rate that was set for the test. This is the number of times per second that the test scenarios will run.
Expand Down Expand Up @@ -1911,14 +1955,14 @@ components:
type: object
description: benchmark_weights
required:
- percentile_ninety
- percentile_ninety_five
- percentile_fifty
- server_errors
- client_errors
- rps
properties:
percentile_ninety:
description: P90 benchmark weight
percentile_ninety_five:
description: P95 benchmark weight
allOf:
- $ref: '#/components/schemas/benchmark_weights'
percentile_fifty:
Expand All @@ -1940,12 +1984,8 @@ components:
benchmark_weights:
type: object
required:
- factor
- percentage
properties:
factor:
type: number
description: benchmark factor weight
percentage:
type: number
description: benchmark percentage weight
71 changes: 55 additions & 16 deletions docs/openapi3.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -323,6 +323,40 @@ paths:
$ref: '#/components/schemas/benchmark_request'
description: The bench mark to add
required: true
get:
tags:
- Tests
parameters:
- in: path
name: test_id
description: The test id.
required: true
schema:
type: string
format: uuid
example: 4bf5d7ab-f310-4a64-8ec2-d65c06188ec1
summary: get a bench mark for test
description: Get a benchmark for a test
responses:
'200':
description: Success
content:
application/json:
schema:
$ref: '#/components/schemas/benchmark_request'
'404':
description: Not found
content:
application/json:
schema:
$ref: '#/components/schemas/error_response'
'500':
description: Internal server error
content:
application/json:
schema:
$ref: '#/components/schemas/error_response'

'/v1/tests/file/{file_id}':
get:
operationId: retrieve-a-file
Expand Down Expand Up @@ -1465,6 +1499,11 @@ components:
benchmark_request:
type: object
additionalProperties: false
required:
- errors
- codes
- rps
- latency
properties:
errors:
type: object
Expand All @@ -1479,24 +1518,21 @@ components:
500: 53
rps:
type: object
required:
- mean
properties:
count:
type: number
mean:
type: number
latency:
type: object
required:
- median
- p95
properties:
min:
type: number
max:
type: number
median:
type: number
p95:
type: number
p99:
type: number

dsl:
description: A test that is made of scenarios base on domain specific language
Expand Down Expand Up @@ -2047,6 +2083,13 @@ components:
type: number
description: The duration of the test in seconds.
example: 60
score:
type: number
description: The score of a report, calculated based on a benchmark and benchmark weights
example: 9.25
benchmark_weights_data:
type: object
description: The way score calualated
arrival_rate:
type: number
description: The arrival rate that was set for the test. This is the number of times per second that the test scenarios will run.
Expand Down Expand Up @@ -2192,14 +2235,14 @@ components:
type: object
description: benchmark_weights
required:
- percentile_ninety
- percentile_ninety_five
- percentile_fifty
- server_errors
- client_errors
- rps
properties:
percentile_ninety:
description: P90 benchmark weight
percentile_ninety_five:
description: P95 benchmark weight
allOf:
- $ref: '#/components/schemas/benchmark_weights'
percentile_fifty:
Expand All @@ -2221,12 +2264,8 @@ components:
benchmark_weights:
type: object
required:
- factor
- percentage
- percentage
properties:
factor:
type: number
description: benchmark factor weight
percentage:
type: number
description: benchmark percentage weight
10 changes: 9 additions & 1 deletion src/configManager/helpers/configDataMap.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
const constConfig = require('../../common/consts').CONFIG;

const BENCHMARK_WEIGHTS_DEFAULT = {
percentile_ninety_five: { percentage: 20 },
percentile_fifty: { percentage: 20 },
server_errors: { percentage: 20 },
client_errors: { percentage: 20 },
rps: { percentage: 20 }
};

let configDataMap = {
[constConfig.GRFANA_URL]: { value: process.env.GRAFANA_URL },
[constConfig.DELAY_RUNNER_MS]: { value: process.env.DELAY_RUNNER_MS || 0, type: 'int' },
Expand Down Expand Up @@ -32,7 +40,7 @@ let configDataMap = {
},
[constConfig.BENCHMARK_THRESHOLD]: { value: process.env.BENCHMARK_THRESHOLD, type: 'int' },
[constConfig.BENCHMARK_THRESHOLD_WEBHOOK_URL]: { value: process.env.BENCHMARK_THRESHOLD_WEBHOOK_URL, type: 'string' },
[constConfig.BENCHMARK_WEIGHTS]: { value: process.env.BENCHMARK_WEIGHTS, type: 'json' }
[constConfig.BENCHMARK_WEIGHTS]: { value: process.env.BENCHMARK_WEIGHTS || JSON.stringify(BENCHMARK_WEIGHTS_DEFAULT), type: 'json' }
};

module.exports.getConstType = (configValue) => {
Expand Down
4 changes: 2 additions & 2 deletions src/configManager/helpers/validators.js
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
module.exports.validateBenchmarkWeights = (req, res, next) => {
const benchmarkWeights = req.body.benchmark_weights;
if (benchmarkWeights){
const p90Percentage = benchmarkWeights.percentile_ninety.percentage;
const p95Percentage = benchmarkWeights.percentile_ninety_five.percentage;
const p50Percentage = benchmarkWeights.percentile_fifty.percentage;
const serverErrorsPercentage = benchmarkWeights.server_errors.percentage;
const clientErrorsPercentage = benchmarkWeights.client_errors.percentage;
const rpsPercentage = benchmarkWeights.rps.percentage;

const percentageSum = p90Percentage + p50Percentage + serverErrorsPercentage + clientErrorsPercentage + rpsPercentage;
const percentageSum = p95Percentage + p50Percentage + serverErrorsPercentage + clientErrorsPercentage + rpsPercentage;
if (percentageSum !== 100){
const error = new Error('Benchmark weights needs to sum up to 100%');
error.statusCode = 422;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
ALTER TABLE reports_summary ADD benchmark_weights_data text;
---
ALTER TABLE reports_summary ADD score float;
---
ALTER TABLE last_reports ADD benchmark_weights_data text;
---
ALTER TABLE last_reports ADD score float;

Loading