Upload API stress/load testing

[mkrystof]

Stress test the following scenarios:

• Multiple clients with multiple simultaneous uploads (one upload per client)
• One client with multiple simultaneous uploads under same client session -- this is covered by issue tus client receives 504 from upload API on uploading of multiple files concurrently  #85
• Single large file upload
• Simultaneous large file uploads

---

• Test DEX -> EDAV copy by azure function for large payloads

Note: please see test results below.

[rmharrison]

Re: large file uploads

Single large file upload
Simultaneous large file uploads

Flagged by Frank Wang

Azure function is limited to 10 min execution, right? I am thinking about larger datasets especially all CDC data coming in using the Upload API and Functions may overload.

We're using a higher-tier, so we don't have a 10 min execution limit. But it's possible than an Azure Function timeout could kill the copy. It depends on whether the function is waiting for a synchronous copy complete response, or just triggers the copy (async). I'm not sure how it's implemented.

[cosmycx]

File Name	File Size	Upload Duration [ms]	Upload Duration [s]	Note
large_file_95MB.txt	95MB	25205	25.205
large_file_95MB.txt	95MB	46841	46.841
large_file_95MB.txt	95MB	24303	24.303
large_file_95MB.txt	95MB	8878	8.878
large_file_95MB.txt	95MB	28440	28.44	concurrent upload
large_file_190MB.txt	190MB	45984	45.984
large_file_190MB.txt	190MB	70151	70.151
large_file_190MB.txt	190MB	35144	35.144
large_file_190MB.txt	190MB	39293	39.293
large_file_190MB.txt	190MB	52680	52.68	concurrent upload
large_file_476MB.txt	476MB	931273	931.273	not on z-scaler, received response patch 500 once
large_file_476MB.txt	476MB	32093	32.093	on z-scaler
large_file_476MB.txt	476MB	30174	30.174	on z-scaler
large_file_476MB.txt	476MB	91478	91.478	not on on z-scaler
large_file_953MB.txt	963MB	261317	261.317	(~4.35 min) on on z-scaler
large_file_4GB.txt	4GB	2212094	2212.094	(~36 min) on z-scaler, received responses patch 500, head 423, upload resumed
large_file_5.56GB.txt	5.56GB	2733818	2733.818	(~45 min) on z-scaler, received multiple responses patch 500, see pics below, upload resumed but lost chunks, eventually and finished

[cosmycx]

For large files, usually larger than 4GB, the upload grinds and restarts in the back:

[rmharrison]

Thx @cosmycx. This is fine for individual file, but doesn't really cover the "one client, many concurrent files" use case.
It looks like you did concurrency of 2x.
A concurrent file test would cover 2-100x files. Sparse is fine [2, 3, 5, 8, 13, 21, 34, 55, 89...].
My current understanding is concurrency fails at ~4.

[cosmycx]

changed function plan:

looks to be working differently and crashing less, still getting 500s