Performance drop introduced by mitmproxy

While working with our current framework, which employs a proxy for instrumentation, I've observed an additional overhead that impacts performance. This slowdown is primarily caused by the mitmproxy artifact. Through my analysis, I identified two critical issues:

1. Non-linear Growth in Processing Time: The processing time for network requests increases non-linearly with the size of the packets. This means that as the packet size grows, the time required to process these requests increases disproportionately.

2. Significant Slowdown with Concurrent Requests: The processing time for network requests slows down dramatically when handling multiple requests simultaneously.

The worst-case scenario occurs when there is a high volume of concurrent requests, especially when some of these requests are significantly large. In such cases, the performance degradation is severe, leading to substantial delays and inefficiencies in processing.

I demonstrated these findings through two experimental settings: 1/ Sequential Requests: The first HTML request loads the JavaScript files one by one. Each request is triggered by the onload event of the previously requested JavaScript file. 2/ Concurrent Requests: The second HTML request loads all the JavaScript files simultaneously. For these experiments, I created 20 JavaScript files ranging from 1MB to 100MB, increasing by 5MB each.

Sequential Requests: Without mitmproxy vs. With mitmproxy

Without mitmproxy:

Name	Size	Loading Time
large_script_1.js	1.0 MB	4 ms
large_script_2.js	6.3 MB	35 ms
large_script_3.js	11.5 MB	83 ms
large_script_4.js	16.8 MB	142 ms
large_script_5.js	22.0 MB	201 ms
large_script_6.js	27.3 MB	254 ms
large_script_7.js	32.5 MB	294 ms
large_script_8.js	37.7 MB	357 ms
large_script_9.js	43.0 MB	412 ms
large_script_10.js	48.2 MB	467 ms
large_script_11.js	53.5 MB	502 ms
large_script_12.js	58.7 MB	578 ms
large_script_13.js	64.0 MB	590 ms
large_script_14.js	69.2 MB	664 ms
large_script_15.js	74.4 MB	744 ms
large_script_16.js	79.7 MB	779 ms
large_script_17.js	84.9 MB	832 ms
large_script_18.js	90.2 MB	883 ms
large_script_19.js	95.4 MB	976 ms
large_script_20.js	101 MB	970 ms

With mitmproxy:

Name	Size	Loading Time
large_script_1.js	1.0 MB	71 ms
large_script_2.js	6.3 MB	62 ms
large_script_3.js	11.5 MB	195 ms
large_script_4.js	16.8 MB	272 ms
large_script_5.js	22.0 MB	416 ms
large_script_6.js	27.3 MB	602 ms
large_script_7.js	32.5 MB	818 ms
large_script_8.js	37.7 MB	1.78 s
large_script_9.js	43.0 MB	2.94 s
large_script_10.js	48.2 MB	4.34 s
large_script_11.js	53.5 MB	5.82 s
large_script_12.js	58.7 MB	7.69 s
large_script_13.js	64.0 MB	9.29 s
large_script_14.js	69.2 MB	11.30 s
large_script_15.js	74.4 MB	13.83 s
large_script_16.js	79.7 MB	16.19 s
large_script_17.js	84.9 MB	18.74 s
large_script_18.js	90.2 MB	20.94 s
large_script_19.js	95.4 MB	24.02 s
large_script_20.js	101 MB	26.58 s

Concurrent Requests: Without mitmproxy vs. With mitmproxy

Without mitmproxy:

Name	Size	Loading Time
large_script_1.js	1.0 MB	4 ms
large_script_2.js	6.3 MB	40 ms
large_script_3.js	11.5 MB	97 ms
large_script_4.js	16.8 MB	152 ms
large_script_5.js	22.0 MB	202 ms
large_script_6.js	27.3 MB	266 ms
large_script_7.js	32.5 MB	332 ms
large_script_8.js	37.7 MB	435 ms
large_script_9.js	43.0 MB	547 ms
large_script_10.js	48.2 MB	719 ms
large_script_11.js	53.5 MB	940 ms
large_script_12.js	58.7 MB	1.05 s
large_script_13.js	64.0 MB	1.17 s
large_script_14.js	69.2 MB	1.27 s
large_script_15.js	74.4 MB	1.64 s
large_script_16.js	79.7 MB	2.10 s
large_script_17.js	84.9 MB	2.41 s
large_script_18.js	90.2 MB	2.54 s
large_script_19.js	95.4 MB	2.77 s
large_script_20.js	101 MB	2.94 s

With mitmproxy:

Name	Size	Loading Time
large_script_1.js	1.0 MB	21 ms
large_script_2.js	6.3 MB	281 ms
large_script_3.js	11.5 MB	700 ms
large_script_4.js	16.8 MB	1.36 s
large_script_5.js	22.0 MB	1.72 s
large_script_6.js	27.3 MB	2.53 s
large_script_7.js	32.5 MB	3.53 s
large_script_8.js	37.7 MB	6.03 s
large_script_9.js	43.0 MB	9.56 s
large_script_10.js	48.2 MB	15.74 s
large_script_11.js	53.5 MB	20.72 s
large_script_12.js	58.7 MB	28.10 s
large_script_13.js	64.0 MB	36.80 s
large_script_14.js	69.2 MB	50.31 s
large_script_15.js	74.4 MB	1.1 min
large_script_16.js	79.7 MB	1.4 min
large_script_17.js	84.9 MB	1.6 min
large_script_18.js	90.2 MB	1.9 min
large_script_19.js	95.4 MB	2.2 min
large_script_20.js	101 MB	2.4 min