CMS load testing setup

[lbeaufort]

Set up CMS locust testing

• Figure out load (volume and specific requests) from 4/15: 240/30 seconds, 8 requests/second
• Install locust in CMS env
• Gather @source.type=RTR logs (https://logs.fr.cloud.gov/goto/5743f0824bd4353529156d064c307f08)
• Generate requests in locust format (in progress, WIP script below. Need to tweak for parameters (data ajax calls))
• Put in a PR with queries to CMS

Proof of concept

• Make sure everything actually works by running a sample test on stage as-is

[lbeaufort]

Log parsing script:

"""Extract API query from Kibana log file"""
import csv
import json

with open("LB_CMS_RTR_requests_4-15-20.csv", "r") as file:
    reader = csv.reader(file, delimiter=',')
    # url/query lookup
    queries = {}
    # count = 0
    for row in reader:
        if "@message" not in row:
            # Column 2 has the queries
            url = row[1].partition("GET ")[2].partition(" ")[0]
            # count += 1
            if "?" in url:
                url = url.partition("?")[0]
                # Get the query param string - everything between the ? and the ' '
                query_parameters = row[1].partition("?")[2].partition(" ")[0]
                # Parse arguments
                query_dict = {}
                if "&" in query_parameters:
                    # Clean up some double &&
                    query_parameters = query_parameters.replace("&&", "&")
                    # Split each query pair out into a list
                    parameter_groups = query_parameters.split("&")
                    # Make a dictionary of the parameters (this is how locust needs them)
                    for result in parameter_groups:
                        # Split the parameters from the values
                        if "=" in result and "api_key" not in result:
                            #print(result)
                            key, value = result.split("=")
                            if not query_dict.get(key):
                                query_dict[key] = [value]
                            else:
                                query_dict[key].append(value)
                if query_dict:
                    # Add to the url/query lookup
                    if not queries.get(url):
                        queries[url] = [query_dict]
                    else:
                        if query_dict not in queries[url]:
                            queries[url].append(query_dict)
            else:
                if url not in queries:
                    queries[url] = {}
        # if count == 100:
        #     break

print(json.dumps(queries, indent=1))
print(len(queries))