Systematically compare/investigate performance on Zarrs #54
Comments
|
Here are preliminary results for a simple "download subtree part of zarr" using bash script#!/bin/bash
set -eu
ds=000108
asset=0dfbb4b4-be63-45a6-8354-347cb98fdb5b
if false; then
# just ot cut/paste outputs below
curl --silent https://api.dandiarchive.org/api/dandisets/000108/versions/draft/assets/$asset/ | { echo -n "version="; jq -r '.digest["dandi:dandi-zarr-checksum"]'; }
curl --silent https://api.dandiarchive.org/api/dandisets/000108/versions/draft/assets/$asset/ | { echo -n "zarr="; jq -r '.contentUrl[1]' | sed -e 's,.*zarr/,,g' -e 's,/,,g'; }
curl --silent https://api.dandiarchive.org/api/dandisets/000108/versions/draft/assets/$asset/ | { echo -n "path="; jq -r '.path' ; }
fi
# too big already
#zarr=04767811-3cea-43c8-956d-81da5e496f80
#version=WRONG!!
#path=sub-MITU01/ses-20211002h21m40s58/micr/sub-MITU01_ses-20211002h21m40s58_sample-37_stain-YO_run-1_chunk-10_SPIM.ome.zarr
# a little smaller
version=0395d0a3767524377b58da3945b3c063-48379--27115470
zarr=0d5b9be5-e626-4f6a-96da-b6b602954899
path=sub-U01hm15x/ses-20220731h17m24s47/micr/sub-U01hm15x_ses-20220731h17m24s47_sample-mEhmAD031x15R2_ABETA_stain-ABETA_run-1_chunk-1_SPIM.ome.zarr
part=${PART:-0/0/0/0/0/}
: ${CONCUR:=20}
OUT="/tmp/zarr-bm"
rm -rf "$OUT"/* || :
#set -x
# simple download directly from S3
get_aws_s3() {
aws configure set default.s3.max_concurrent_requests $CONCUR
/usr/bin/time chronic aws s3 --no-sign-request sync s3://dandiarchive/zarr/$zarr/$part "$1"
}
get_rclone_s3() {
aws configure set default.s3.max_concurrent_requests $CONCUR
/usr/bin/time rclone sync --transfers $CONCUR DANDI-S3:dandiarchive/zarr/$zarr/$part "$1"
}
get_rclone_dandisets() {
/usr/bin/time rclone sync --transfers $CONCUR DANDI-WEBDAV:dandisets/$ds/draft/$path/$part "$1"
}
get_rclone_zarr_manifest() {
/usr/bin/time rclone sync --transfers $CONCUR DANDI-WEBDAV:zarrs/${zarr:0:3}/${zarr:3:3}/${zarr}/$version/$part "$1"
}
echo "Downloading part $part within zarr $zarr asking for up to $CONCUR processes"
for method in get_aws_s3 get_rclone_s3 get_rclone_dandisets get_rclone_zarr_manifest; do
out="$OUT/$method"
echo "---------------"
echo "$method: $out"
$method "$out"
checksum=$(TQDM_DISABLE=1 zarrsum local "$out" | tail -n 1)
if [ -z "$part" ] && [ $checksum != "$version" ]; then
echo "wrong checksum $checksum != $version"
fi
if [ -n "$part" ]; then
echo "checksum $checksum"
fi
doneso we see that (dev3) yoh@typhon:~/proj/dandi/zarr-benchmarking/tools$ PART=0/0/0/0/ CONCUR=10 ./simple_parallel_get.sh
Downloading part 0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 10 processes
---------------
get_aws_s3: /tmp/zarr-bm/get_aws_s3
22.49user 2.03system 0:29.91elapsed 82%CPU (0avgtext+0avgdata 119156maxresident)k
0inputs+24064outputs (0major+110906minor)pagefaults 0swaps
checksum 742c8d77baf7240437d33117f1d063fb-3008--1684480
---------------
get_rclone_s3: /tmp/zarr-bm/get_rclone_s3
3.64user 0.94system 0:16.16elapsed 28%CPU (0avgtext+0avgdata 85516maxresident)k
0inputs+24064outputs (0major+5286minor)pagefaults 0swaps
checksum 742c8d77baf7240437d33117f1d063fb-3008--1684480
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
4.63user 1.99system 3:16.25elapsed 3%CPU (0avgtext+0avgdata 74280maxresident)k
0inputs+24064outputs (0major+5663minor)pagefaults 0swaps
checksum 742c8d77baf7240437d33117f1d063fb-3008--1684480
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
3.47user 1.55system 0:32.48elapsed 15%CPU (0avgtext+0avgdata 74440maxresident)k
0inputs+24064outputs (0major+5382minor)pagefaults 0swaps
checksum 742c8d77baf7240437d33117f1d063fb-3008--1684480on a serial access we are up to par with the
|
|
@yarikoptic Yes, a dedicated dandi-archive endpoint would definitely help with efficiency. |
with CONCUR=20 rclone scales linearly to cutting in half from 10. rclone of dandidav remains the same -- suggests that there might be limiting factor in parallel handling of requests(dev3) yoh@typhon:~/proj/dandi/zarr-benchmarking/tools$ PART=0/0/0/0/ CONCUR=20 ./simple_parallel_get.sh
Downloading part 0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 20 processes
---------------
get_aws_s3: /tmp/zarr-bm/get_aws_s3
30.80user 2.00system 0:23.41elapsed 140%CPU (0avgtext+0avgdata 127672maxresident)k
0inputs+24064outputs (0major+111824minor)pagefaults 0swaps
checksum 742c8d77baf7240437d33117f1d063fb-3008--1684480
---------------
get_rclone_s3: /tmp/zarr-bm/get_rclone_s3
3.90user 1.00system 0:08.26elapsed 59%CPU (0avgtext+0avgdata 91768maxresident)k
0inputs+24064outputs (0major+5026minor)pagefaults 0swaps
checksum 742c8d77baf7240437d33117f1d063fb-3008--1684480
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
4.68user 1.94system 3:28.99elapsed 3%CPU (0avgtext+0avgdata 74680maxresident)k
0inputs+24064outputs (0major+5036minor)pagefaults 0swaps
checksum 742c8d77baf7240437d33117f1d063fb-3008--1684480
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
3.79user 1.43system 0:32.42elapsed 16%CPU (0avgtext+0avgdata 77672maxresident)k
0inputs+24064outputs (1major+4009minor)pagefaults 0swaps
checksum 742c8d77baf7240437d33117f1d063fb-3008--1684480For /dandisets/ -- seems to stay at about 30% CPUshub@falkor:~$ pidstat -p 3044013 1
Linux 6.1.0-10-amd64 (falkor) 02/13/2024 _x86_64_ (8 CPU)
09:15:06 AM UID PID %usr %system %guest %wait %CPU CPU Command
09:15:07 AM 1001 3044013 34.00 1.00 0.00 0.00 35.00 0 dandidav
09:15:08 AM 1001 3044013 29.00 3.00 0.00 0.00 32.00 0 dandidav
09:15:09 AM 1001 3044013 30.00 2.00 0.00 0.00 32.00 0 dandidav
09:15:10 AM 1001 3044013 29.00 2.00 0.00 0.00 31.00 0 dandidav
09:15:11 AM 1001 3044013 30.00 2.00 0.00 0.00 32.00 0 dandidav
09:15:12 AM 1001 3044013 25.00 4.00 0.00 0.00 29.00 0 dandidav
09:15:13 AM 1001 3044013 26.00 2.00 0.00 0.00 28.00 0 dandidav
For `/zarr/` even lessshub@falkor:~$ pidstat -p 3044013 1
Linux 6.1.0-10-amd64 (falkor) 02/13/2024 _x86_64_ (8 CPU)
09:16:56 AM UID PID %usr %system %guest %wait %CPU CPU Command
09:16:57 AM 1001 3044013 9.00 1.00 0.00 0.00 10.00 0 dandidav
09:16:58 AM 1001 3044013 5.00 1.00 0.00 0.00 6.00 0 dandidav
09:16:59 AM 1001 3044013 10.00 0.00 0.00 0.00 10.00 0 dandidav
09:17:00 AM 1001 3044013 11.00 1.00 0.00 0.00 12.00 0 dandidav
09:17:01 AM 1001 3044013 9.00 0.00 0.00 0.00 9.00 0 dandidav
09:17:02 AM 1001 3044013 8.00 1.00 0.00 0.00 9.00 0 dandidav
09:17:03 AM 1001 3044013 9.00 0.00 0.00 0.00 9.00 0 dandidav
09:17:04 AM 1001 3044013 9.00 2.00 0.00 0.00 11.00 0 dandidav
09:17:05 AM 1001 3044013 9.00 2.00 0.00 0.00 11.00 0 dandidav
09:17:06 AM 1001 3044013 9.00 1.00 0.00 0.00 10.00 0 dandidav
09:17:07 AM 1001 3044013 11.00 1.00 0.00 0.00 12.00 0 dandidav
09:17:08 AM 1001 3044013 10.00 1.00 0.00 0.00 11.00 0 dandidav
09:17:09 AM 1001 3044013 10.00 2.00 0.00 0.00 12.00 0 dandidavAlthough no %wait I wonder if logging (#69) already something effecting it. |
|
@yarikoptic Did you build the |
|
@yarikoptic - just a note that testing throughput could be broken into two parts (reading, reading + verifying). the former is what any zarr reader would do. it wouldn't try to check that it received the correct bytes. the latter is what dandi cli would do to ensure that we do receive the correct bytes. and the tools used above do various things depending on flags. also for parallel read from s3, s5cmd is significantly faster than most other tools (https://github.com/peak/s5cmd). |
|
@satra thanks for the note. In the script/output above I report the zarr checksum we have for all downloads -- all good as long as they match. If it is not partial download of a zarr (like in those above) -- it would explicitly compare to the target overall zarr checksum. Verification is not taken as part of the benchmark. If you have some python / any other code you would like to benchmark zarrs -- please share. re s5cmd, I will add it then. Probably in favor over |
|
FWIW with default settings s5cmd with direct access to s3 at 00.48 definitely beats everything down (note: for aws s3 invocation I just set concurrency to 255) especially if I do not request any specific level of concurrency and s5cmd goes wild to its top 255 etc - it is about 3 times faster than rclone access from our dandidav. But rclone itself is slower than s5cmd in direct access to s3 - and our overhead for it is quite small (2.13 rclone on s3 vs 2.30 rclone on our manifests)yoh@typhon:~/proj/dandi/zarr-benchmarking$ tools/simple_parallel_get.sh
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 without specifying explicit number of processes
---------------
get_aws_s3: /tmp/zarr-bm/get_aws_s3
TIME: 0:10.58 real 104.23 user 0.56 sys 990% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
---------------
get_s5cmd_s3: /tmp/zarr-bm/get_s5cmd_s3
TIME: 0:00.48 real 2.00 user 0.24 sys 464% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
---------------
get_rclone_s3: /tmp/zarr-bm/get_rclone_s3
TIME: 0:02.13 real 0.30 user 0.10 sys 19% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:15.81 real 0.31 user 0.15 sys 2% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.30 real 0.29 user 0.08 sys 16% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280but if I give rclone explicit up to 30 concurrent processes, it improves to 00.59 and actually beats s5cmd at the same level of concurrency making it in 00.61. Unfortunately rclone access to dandidav manifests stays at ceiling of 02.12yoh@typhon:~/proj/dandi/zarr-benchmarking$ CONCUR=30 tools/simple_parallel_get.sh
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 30 processes
---------------
get_aws_s3: /tmp/zarr-bm/get_aws_s3
TIME: 0:08.16 real 23.55 user 0.21 sys 290% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
---------------
get_s5cmd_s3: /tmp/zarr-bm/get_s5cmd_s3
TIME: 0:00.61 real 1.43 user 0.19 sys 262% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
---------------
get_rclone_s3: /tmp/zarr-bm/get_rclone_s3
TIME: 0:00.59 real 0.48 user 0.10 sys 99% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:12.14 real 0.46 user 0.12 sys 4% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.12 real 0.32 user 0.12 sys 21% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280and indeed we reach ceiling at some where around 20 concurrent processesyoh@typhon:~/proj/dandi/zarr-benchmarking$ for c in 1 5 10 20 30 40 50; do CONCUR=$c METHODS=get_rclone_zarr_manifest tools/simple_parallel_get.sh; done
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 1 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:07.93 real 0.32 user 0.11 sys 5% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 5 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.21 real 0.30 user 0.12 sys 19% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 10 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.19 real 0.36 user 0.10 sys 21% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 20 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.12 real 0.36 user 0.12 sys 23% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 30 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.13 real 0.38 user 0.10 sys 23% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 40 processesall prior results were with prior "debug" build. With fresh "release" build situation is qualitatively the same.yoh@typhon:~/proj/dandi/zarr-benchmarking$ for c in 1 5 10 20 30 40 50; do CONCUR=$c METHODS=get_rclone_zarr_manifest tools/simple_parallel_get.sh; done
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 1 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:10.93 real 0.31 user 0.16 sys 4% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 5 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.31 real 0.29 user 0.08 sys 16% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 10 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.12 real 0.33 user 0.09 sys 20% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 20 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.08 real 0.27 user 0.12 sys 19% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 30 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.08 real 0.35 user 0.09 sys 21% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 40 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.09 real 0.37 user 0.12 sys 23% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 50 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.10 real 0.33 user 0.13 sys 22% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
@jwodder said that he already looked into scalability a little but have not found a resolution yet. Given that rclone seems generally can reach s5cmd performance on access to S3, I think solving scalability solution for the |
FWIW: I considered possibility that it is apache as reverse proxy making a limit. It is not the case: ran on falkor against apache and directly against dandidav -- the samefalkor:~/proj/dandi/zarr-benchmarking
$> for c in 1 5 10 20 30 40 50; do CONCUR=$c METHODS=get_rclone_zarr_manifest RCLONE_DANDI_WEBDAV=DANDI-WEBDAV-LOCAL tools/simple_parallel_get.sh; done
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 1 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:07.89 real 0.29 user 0.14 sys 5% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 5 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.22 real 0.27 user 0.12 sys 17% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 10 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.15 real 0.31 user 0.11 sys 19% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 20 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.10 real 0.24 user 0.17 sys 19% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 30 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.17 real 0.39 user 0.15 sys 25% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 40 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.10 real 0.36 user 0.10 sys 21% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 50 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.09 real 0.32 user 0.17 sys 23% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
(dev3.11) (git)/home/yoh/proj/dandi/zarr-benchmarking:[master]
#22 !2680 [0].....................................:Wed Feb 14 08:47:45:.
falkor:~/proj/dandi/zarr-benchmarking
*$> for c in 1 5 10 20 30 40 50; do CONCUR=$c METHODS=get_rclone_zarr_manifest RCLONE_DANDI_WEBDAV=DANDI-WEBDAV tools/simple_parallel_get.sh; done
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 1 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:07.69 real 0.29 user 0.16 sys 6% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 5 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.20 real 0.26 user 0.12 sys 17% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 10 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.14 real 0.25 user 0.16 sys 19% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 20 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.11 real 0.27 user 0.20 sys 22% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 30 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.11 real 0.28 user 0.13 sys 19% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 40 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.12 real 0.24 user 0.20 sys 20% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 50 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.12 real 0.27 user 0.14 sys 19% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
FWIW I have pushed current version of the helper to https://github.com/dandi/zarr-benchmarking/blob/master/tools/simple_parallel_get.sh |
|
@yarikoptic To be clear: You're concluding that |
|
correct. I don't think so since the overall number of requests is constant (number of files in that zarr path) and the only thing changing is how many we are submitting in parallel. So it is a question of throughput really. My trivilized thinking/explanation which simplifies as in submitter not waiting for response (like we do) but overall of the same reasoning. If dandidav can handle max 20 requests in 1 sec, the time to process 100 requests would remain constant 5 seconds, regardless in how many requests (above 20) I would request at once (in parallel). |
|
@yarikoptic I think I misunderstood what you were doing with concurrent processes. I thought you were running, say, 20 copies of |
|
correct! indeed it might also be valuable to test against multiple clients in parallel bombarding the same/different zarrs. |
|
@yarikoptic I'm going to ask about request limits on a Rust forum, and I need to know how many CPUs (not cores) the machine |
|
@yarikoptic Also, could you run |
32
Sure, doing against LOCAL *$> for c in 1 5 10 20 30 40 50; do CONCUR=$c METHODS=get_rclone_zarr_manifest RCLONE_DANDI_WEBDAV=DANDI-WEBDAV-LOCAL tools/simple_parallel_get.sh; done
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 1 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:13.37 real 0.29 user 0.22 sys 3% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 5 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.47 real 0.25 user 0.17 sys 17% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 10 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.15 real 0.36 user 0.10 sys 21% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 20 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.10 real 0.30 user 0.18 sys 23% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 30 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.09 real 0.39 user 0.12 sys 24% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 40 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.14 real 0.36 user 0.14 sys 23% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 50 processes
---------------
get_rclone_zarr_manifest: /tmp/zarr-bm/get_rclone_zarr_manifest
TIME: 0:02.10 real 0.34 user 0.15 sys 23% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
shub@falkor:~$ dandidav-test/dandidav --version
dandidav 0.2.0 (commit: 4998bd8)
shub@falkor:~$ file dandidav-test/dandidav
dandidav-test/dandidav: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=825b59e00515dad24f6ae389e6a79c65765232dd, for GNU/Linux 3.2.0, with debug_info, not strippedif needed next time - to expedite, just run the script on e.g. drogon: it is now on github. I can also make account for you on either typhon (where I did originally) or falkor (webserver), just let me know. |
|
Hi @jwodder , anything from Rust community on the issue of scalability? What to look for? |
|
@yarikoptic No, my post got no attention. The only thing I've learned is that axum (the web framework |
|
but what is it -- limited number of threads ? |
|
@yarikoptic |
|
is it tunable? if we double it and it doesn't have effect -- must be smth else. if not -- we could figure out why 20 and not 32 ;-) |
|
@yarikoptic Yes, it's tunable. If you set the environment variable If you're really paranoid, you can check what Rust thinks the number of CPUs is as follows:
|
|
FWIW. Setting it to 10 caused performance ceiling at 10on server: on client typhon (zarr-manifests) yoh@typhon:~/proj/dandi/zarr-benchmarking$ for c in 1 5 10 15 20 30 40 50; do CONCUR=$c METHODS=get_rclone_dandisets RCLONE_DANDI_WEBDAV=DANDI-WEBDAV tools/simple_parallel_get.sh; done
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 1 processes
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:37.85 real 0.39 user 0.12 sys 1% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 5 processes
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:09.88 real 0.39 user 0.16 sys 5% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 10 processes
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:08.74 real 0.41 user 0.11 sys 6% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 15 processes
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:08.20 real 0.43 user 0.11 sys 6% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 20 processes
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:08.36 real 0.46 user 0.10 sys 6% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 30 processes
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:08.44 real 0.50 user 0.18 sys 8% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 40 processes
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:08.27 real 0.56 user 0.16 sys 8% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 50 processes
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:08.59 real 0.83 user 0.20 sys 12% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
setting to 100 - the same ceiling at 20-30(zarr-manifests) yoh@typhon:~/proj/dandi/zarr-benchmarking$ for c in 1 5 10 15 20 30 40 50; do CONCUR=$c METHODS=get_rclone_dandisets RCLONE_DANDI_WEBDAV=DANDI-WEBDAV tools/simple_parallel_get.sh; done
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 1 processes
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:34.04 real 0.29 user 0.20 sys 1% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 5 processes
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:09.10 real 0.37 user 0.13 sys 5% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 10 processes
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:08.74 real 0.31 user 0.24 sys 6% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 15 processes
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:08.34 real 0.40 user 0.13 sys 6% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 20 processes
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:08.42 real 0.41 user 0.15 sys 6% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 30 processes
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:08.22 real 0.51 user 0.21 sys 8% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 40 processes
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:08.79 real 0.54 user 0.13 sys 7% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
Downloading part 0/0/0/0/0/ within zarr 0d5b9be5-e626-4f6a-96da-b6b602954899 asking for up to 50 processes
---------------
get_rclone_dandisets: /tmp/zarr-bm/get_rclone_dandisets
TIME: 0:08.38 real 0.87 user 0.25 sys 13% CPU
checksum f494a7ab20c6906e4c176e7a0f08c29d-188--105280
so there is indeed more to it |
|
@yarikoptic This is largely a shot in the dark, but on the server that |
|
I even had no clue that there are different kinds of cores. FWIW, to rule out cores effect we could check scalability while running on some other server. Could you please run on typhon which has 32 cores? |
It's a new thing that ARM came out with in 2011, and then Intel made its own version in 2022-ish. (Thus, if the server's CPUs are from 2021 or earlier, then they're just "normal" cores and my theory is wrong.) I only found out about the different types last week or the week before when I was looking at the specs for new MacBooks.
I don't think you ever gave me access to typhon, and I don't know how to access it. (At the very least, |
|
@yarikoptic I ran both
|
|
@yarikoptic I wrote https://github.com/jwodder/batchdav for measuring WebDAV traversal time using a given number of workers. Unlike I've run it at various different points, and the times from different sessions have varied by about an order of magnitude, which I suspect is due to differing amounts of load on Heroku. Some of the fastest times I've ever gotten are:
Make of that what you will. |
|
Just to recall/summarize what we have so far:
|
|
@yarikoptic I'm starting to think that the "20 workers ceiling" is just a statistical illusion resulting from the combination of two factors:
It seems that the decrease from (2) is dominant until some point in the 20-30 concurrent requests range, after which the increase from (1) is dominant, thereby producing a minimum runtime around 20 workers. |
|
it could indeed be the case. But why and how much the response time increases for the |
|
@yarikoptic I'm not entirely sure why it happens, but you can see the increase in the following statistical summary of the individual request times per number of client workers when using batchdav from my MacBook Pro to traverse [1]: The effect also happens when both the |
|
interesting. So indeed it pretty much jumps twice in Avg going from 20 to 40, but not before, i.e. before 20 it grows slower than factor for number of workers (e.g. it is growing twice in going from 1 to 10). Just for academic purposes -- could you produce those Avg times only for our dandidav and smaug only but with step 1? then I would like to look at estimate of Avg[n]/Avg[n/k] (k==2 or some other) -- is it jumping rapidly higher at some specific value and either that value the same for the both, and where it crosses the horizontal line for So indeed your observation was probably right spot on that response time is what keeps us from "going faster", but the question remains why such rapid response time growth? it could again be if there is some hardcoded size of some queue somewhere to process only N requests in parallel, and that would increase average time since N+1st request would need to wait first to get its turn, which would add up to response time. |
Over what range of worker values? |
|
how academic you want to go? ;) if not much -- till 50, if more -- till 100 ;) |
I'm not sure what formula you're trying to describe here. If |
|
The statistics for 1 through 50 workers against webdav.dandiarchive.org are: and the statistics for 1 through 50 workers against dandidav run on smaug are: Would you like me to provide the raw data, containing each of the individual request times? |
|
here what I get for k=2
and smaug
assuming that excursion is noise (in particular for smaug at 15), it remains under 2, so in principle should still scale up although very slowly, and indeed that ratio plato is around around 12-15 in both cases, i.e. somewhere at 24-30 we reach so that ratio of response time at N becomes 1.8 of response time at N/2 parallel requests. if zoom in at smaug one
we see that it pretty much grows linearly until then. So there is IMHO some inherent factor, possibly dependent on instance (what is # of processors/cores on heroku) but not clearly so, which just ceils scalability there., |
|
BTW -- while running on smaug what do you see for the CPU load on dandidav (since no actual download is involved I guess could differ from prior stated)? |
|
@yarikoptic The load steadily increased as the number of workers increased, capping out at around 4.2 for the 5-second load average. |
|
what about cpu%? ie is that task becoming somehow IO bound vs CPU? |
|
@yarikoptic I ran |
|
so may be the ceiling is due to bottleneck in the implementation stack somewhere :-/ Are there convenient profiling tools in the |
|
@yarikoptic There are a number of profilers. I'm starting out with flamegraph, but I'm getting the error: For the record, flamegraph's docs suggest setting |
|
Should be all set (but not for every user; and not persistent across reboots), you were added to the |
|
@yarikoptic Side note: From the flamegraph, I've discovered that the functionality for logging how much memory |
|
@yarikoptic I ended up posting a question about this on axum's repository: tokio-rs/axum#2772 |
@satra did just a quick&dirty test using zarr viewer comparing access directly to S3 URL and then our testbed dandi.centerforopenneuroscience.org instance (on a zarr in 000108 under
/dandisets/not the manifests based one since no manifest for it was there yet) so there are no quantified characteristics available butWe need to come up with
Compare
to identify how much any component contributes and if we could improve any aspect (e.g. parallel access etc). I hope that it is not all simply due to redirects.
Later we need to create some nice "versioned zarr" with a few versions and to use for benchmarks and also benchmark on the
/zarrendpoints.The text was updated successfully, but these errors were encountered: