Skip to content

System Performance

Prasad Talasila edited this page Jan 22, 2019 · 6 revisions

System Performance

Metrics

  • Response time is the amount of time it takes for the system to process a request from the outside.
  • Responsiveness is about how quickly the system acknowledges a request as opposed to processing it.
  • Latency is the minimum time required to get any form of response, even if the work to be done is nonexistent.
  • Throughput is how much stuff you can do in a given amount of time.
  • Load is a statement of how much stress a system is under, which might be measured in how many users are currently connected to it.
  • Load sensitivity is an expression of how the response time varies with the load.
  • The capacity of a system is an indication of maximum effective throughput or load.
  • Scalability is a measure of how adding resources (usually hardware) affects performance.

AutolabJS Performance

Metric Application Terminology Ideal Actual
Response Time Evaluation Completion Time Max(test case completion times) Sum of test case completion times
Responsiveness Evaluation Status Update 0 Sum of test case completion times
Responsiveness Status Request Time 0 At most sum of test case completion times
Latency Network Delay 0 0
Throughput Concurrent Evaluations Infinite Total number of execution nodes
Load How Many Pending Evaluations Infinite Only limited by RAM
Load Sensitivity Evaluation Completion Time under Load 0 NEGATIVE
Scalability Increased Concurrent Evaluations for Increasing Execution Nodes Uses all available execution nodes Uses all available execution nodes

Order of metrics to improve:

  1. Scalability
    • Improve deployment procedures for executionnodes component
  2. Load Sensitivity
    • Maintain fair evaluation completion time under load by using congestion avoidance
  3. Response Time
    • Concurrent evaluation of test cases
  4. Responsiveness
    • Asynchronous evaluation requests at clients, perhaps in the background

References

  1. Introduction, Pattern Oriented Enterprise Architecture, Martin Fowler.

Plots for Distribution

The desirable performance plots are given below.

  1. System throughput for the following cases.

    1. null case of (near zero execution time)
    2. lab with one second execution time
    3. lab with two second execution time
    4. lab with ten second execution time

    The labs are to be created in C, CPP, Java, Python with a simple sleep function call of specified duration. The throughput indicates the steady state number of jobs AutolabJS can execute with a given number of execution nodes. Ultimately, all of these plots must approximate to the equation:
    no of requests per second = (3 * number of execution nodes) / (4 * evaluation time per request)

  2. Component throughput of main server and load balancer for the following cases.

    1. null case of (near zero execution time)
  3. System resource consumption for the following cases.

    1. null case of (near zero execution time)
    2. lab with one second execution time
    3. lab with two second execution time
    4. lab with ten second execution time

    The labs are to be created in C, CPP, Java, Python with a simple sleep function call of specified duration. The resources to be tracked are: aggregate cpu load, and aggregate memory used. It is better to obtain the resource usage numbers at per container level.

  4. Live performance of the system.

    1. For one sample DSA lab, collect the lab source code, estimated evaluation time, worst case evaluation time and all the student submissions.
    2. Collect all the logs of the system. Based on the logs, draw a plot with time on x-axis and three parameters on y-axis: available number of execution nodes, evaluation time quantiles, latest evaluation time (for the last n evaluations where n is equal to the number of execution nodes), queuing time quantiles, latest queuing time (for the last 6 user requests)
  5. Peak performance of the system

    1. For one sample DSA lab, create one sample student submission and measure the evaluation time in seconds. Let us call this time, (T).
    2. Every 'T' seconds, submit 'k' evaluation requests to autolab setup where 'k' is equal to the number of execution nodes. Repeat the procedure for 100 iterations.
    3. Measure the following metrics: job queue length, number of failed evaluations, percentage quantiles of evaluation time.

We can use AutolabCLI to run single and multiple evaluations and see the execution performance of the system. The tools that might help do this step are:

Clone this wiki locally
You can’t perform that action at this time.