Client for Timings API to support Python based test environments
NOTE: you need to have a timings API server running in your network to use this client!
- Sending performance data from functional tests to the timings API.
- This client makes it easy to communicate with the API without the need to setup your own curl/request/etc. calls.
- The response contains the necessary fields to validate/assert the performance results (look for the
assert
field!). - The API stores the results in ElasticSearch and can be visualized with Kibana.
- This helps get better visibility into performance improvements/regression trends before moving into production.
To learn more about ELK (Elastic Search, LogStash, Kibana). Click Here https://www.elastic.co/products/kibana
Installation of the client can be done with pip
or easy_install
:
pip install timingsclient
After installation you first need to copy the configuration file and change it to your needs. The client includes a sample config file that can be found here: https://github.com/godaddy/timings-client-py/blob/master/timingsclient/default.yaml
. You will pass the location of this config file when you initiate the client (see below)
Add a custom config file to your project's root folder and edit to match your situation. Example:
PERF_API_URL: "http://localhost/v2/api/cicd/" # the full fqdn of the API server
api_timeout: 2 # timeout for the API in seconds (valid range: 2-30 seconds)
api_params: # the "api parameters" that will be send to the API in the POST body
sla: # This is the "flat SLA" - valid keys are "pageLoadTime" or "visualCompleteTime"
pageLoadTime: 2000 # Flat maximum threshold - this will be used if baseline (+ padding) is greater or flags.assertBaseline=false!
baseline: # These settings will be used by the API to calculate the baseline
days: 7 # Number of days to calculate the baseline for
perc: 75 # Percentile to calculate
padding: 1.2 # Extra padding on top of the calculated baseline (gives some wiggle-room)
flags: # These booleans will tell the API what to do
assertBaseline: true # Whether or not to compare against baseline
debug: false # Request extra debug info from the API
esTrace: false # Request elasticsearch output from API
esCreate: false # Save results to elasticsearch
passOnFailedAssert: false # Pass the test, even when the performance is above the threshold
log: # These key-value pairs will be stored in elasticsearch and can be used to slice & dice the data in Kibana
test_info: "Demo test" # Info about the test(-step)
env_tester: "Windows" # Environment of the test machine (local, saucelabs, selenium grid, etc.)
browser: "Demo browser" # Browser used to run the test with
env_target: "PROD" # Environment of the target app (usually dev, test, or prod)
team: "DEMO TEAM" # THe name of the (test-)team
NOTE: it is recommended that you dynamically populate log fields like browser
and env_target
.
To use the client, import it like you would with any python module:
from timingsclient import Perf
Then, you can initiate the client, along with the location of your config file, as follows:
# Setup custom config
CONFIG_FILE = os.path.join(os.path.abspath(os.path.dirname(__file__)),'', 'custom.yaml')
PERF = Perf(CONFIG_FILE)
With the client initiated, you can now call the different methods from your script.
Here is an example of a complete test 'navtiming' script:
"""
Selenium Grid with Hub and Node test
"""
import os
import platform
from selenium import webdriver
from timingsclient import Perf
BROWSER = webdriver.Chrome(
'/Users/mverkerk/selenium/chromedriver_2.34.exe')
BROWSER.get('http://www.seleniumconf.de/test?test=strip_this')
# Perform functional assert
try:
BROWSER.find_element_by_class_name('section__heading')
print("FUNCTIONAL: Page looks good!")
except:
print("FUNCTIONAL: Page not working!")
# Setup custom config for PERF
CONFIG_FILE = os.path.join(
os.path.abspath(os.path.dirname(__file__)),
'', 'config.yaml')
# Initiate client
PERF = Perf(CONFIG_FILE)
# Collect inject code from API (strip query string = True)
INJECT_CODE = PERF.injectjs(inject_type='navtiming', mark='visual_complete', strip_qs=True)
# Set API parameters
API_PARAMS = PERF.getapiparams(es_create=False, log={
'browser': BROWSER.name, 'env_tester': platform.system()})
if INJECT_CODE is not False:
# Collect perf data from browser
TIMING = BROWSER.execute_script(INJECT_CODE)
# Send perf data with parameters to API
NAV_RESP = PERF.navtiming(TIMING, API_PARAMS)
# Assert perf
if NAV_RESP is not False:
print(
'PERFORMANCE of [' + NAV_RESP["export"]["dl"] + '] was ',
('GOOD' if NAV_RESP['assert'] is True else "BAD") + '! - ',
str(NAV_RESP['export']['perf']['measured']),
'/ ' + str(NAV_RESP['export']['perf']['threshold'])
)
else:
print("NAV_RESP problem!")
else:
print("INJECT_CODE problem!")
BROWSER.close()
Collect or overwrite the default parameters (see above) to be send to the API. None of the parameters are required. If you don't submit any arguments, the defaults will be used. All arguments have to be 'keyed' arguments.
param | type | default | description |
---|---|---|---|
sla | dict | - | Overwrite the default sla settings |
debug | boolean | false |
Receive extra debug information from the API |
esTrace | boolean | false |
Request Elasticsearch query information from the API |
esCreate | boolean | true |
Save the result to elasticsearch |
days | number | 7 |
Number of days to calculate the baseline for |
perc | number | 75 |
Percentile of the baseline to be calculated |
padding | number | 1.2 |
Multiplier to calculate extra padding on top of the baseline |
searchUrl | string | '' |
Wildcard to use for baseline (instead of using the submitted URL) |
log | dict | - | Object that holds the keys to be logged. Can be used to overwrite the defaults or add extra keys! |
Example:
getapiparams(es_create=False, debug=True, log={'browser': BROWSER.name, 'env_tester': platform.system()})
Returns:
{
"sla": {
"pageLoadTime": 2000
},
"baseline": {
"days": 7,
"perc": 75,
"padding": 1.2
},
"flags": {
"assertBaseline": true,
"debug": true,
"esTrace": false,
"esCreate": false,
"passOnFailedAssert": false
},
"log": {
"test_info": "Sample test_info",
"env_tester": "Windows",
"browser": "firefox",
"env_target": "Sample target",
"team": "SAMPLE TEAM"
}
}
Notice that the flags.debug
, log.browser
and the log.env_tester
keys were changed when compared to the defaults!
Get the "inject code" from the API
param | type | required | default | description |
---|---|---|---|---|
inject_type | string | Yes | - | The type of measurement you are performing. Valid options are navtiming and usertiming |
mark | No | string | false |
The name of the visual complete mark |
strip_query_string | No | boolean | false |
Indicates whether you want to strip the querystring from the URL you are testing |
Example:
injectjs( inject_type="navtiming", mark="visual_complete", strip_query_string=True )
Returns:
{
"status": 200,
"inject_code": "var%20visualCompleteTime%20%3D%200%3B%0Aif%20(performance.getEntriesByName('visual_complete').length)%20%7B%0A%20%20visualCompleteTime%20%3D%20parseInt(performance.getEntriesByName('visual_complete')%5B0%5D.startTime)%3B%0A%20%20window.performance.clearMarks()%3B%0A%7D%3B%0Areturn%20%7Btime%3Anew%20Date().getTime()%2C%20timing%3Awindow.performance.timing%2C%20visualCompleteTime%3A%20visualCompleteTime%2C%20url%3A%20document.location.href.split(%22%3F%22)%5B0%5D%2C%20resources%3A%20window.performance.getEntriesByType('resource')%7D%3B"
}
Post navtiming performance data to the API
param | type | required | default | description |
---|---|---|---|---|
inject_js | dict | Yes | - | Contains the full response that you received from the browser after injecting the injectjs code |
api_params | dict | Yes | - | Contains the API params that you retrieved from the getapiparams() method |
Post usertiming performance data to the API
param | type | required | default | description |
---|---|---|---|---|
inject_js | dict | Yes | - | Contains the full response that you received from the browser after injecting the injectjs code |
api_params | dict | Yes | - | Contains the API params that you retrieved from the getapiparams() method |
Post apitiming performance data to the API
param | type | required | default | description |
---|---|---|---|---|
timing | dict | Yes | - | Contains the start/stop timestamps that you set before and after running your API call. Example: {"startTime": 1515443109031, "endTime": 1515443109046} |
url | string | Yes | - | The URL of the API you're testing |
api_params | dict | Yes | - | Contains the API params that you retrieved from the getapiparams() method |
For more information about the API: https://github.com/godaddy/timings