Is there any parameter which restricts thenumber of connection to harbor ?

[kavyavvv]

Is there any parameter which restricts thenumber of connection to harbor ?
How to finetune harbor for handling 1200 parellel pulls ?

[wy65701436]

what connection do you mean? Did you notice any bottle neck for your concurrence pulling?

[kavyavvv]

yes ,
we are getting some errors like connection reset by peer , sometimes 502 gateway error and few TLS handshake errors as well

[wy65701436]

several points that we need to check,

1. cpu/memory usage of core, db, registry
2. database connection
3. the connection reset happens at which phase, it's closed by nginx, distribution or storage.

[kavyavvv]

[kavyavvv]

we are setting such limits to the containers and sometimes CPU and mem for registry will be 100 percent with above restriction

the connection reset happens at which phase, it's closed by nginx, distribution or storage.- connection reset happens between registry and core

[stonezdj]

What is your installation type docker-compose or harbor helm? Do you have any log information show that connection reset happens between registry and core?
The connect reset /tls handshake errors should occur in Nginx side.

For the concurrent pull test, you could refer https://github.com/goharbor/perf/wiki

[kavyavvv]

it is docker-compose

024-06-03T10:34:30.621804333+03:00 stderr F 2024/06/03 07:34:30 http: TLS handshake error from 172.17.0.127:37762: read tcp 172.17.0.124:5443->172.17.0.127:37762: read: connection reset by peer

Test scenario:
and each image is of ~250 MB , there are one or two image which is upto 1.5 gb as well .We are testing to download 58 such image from harbor in 20 virtual machines together. So concurrency comes around 1200.
Above testing i see concurrency is 500
is there any concurrency limit from harbor side as well ?

[kavyavvv]

i am also getting this errors as well

image download error
Error: initializing source docker://zxzxzxzx:8443/vduproject/docker-dhcp:4.44.1: Requesting bearer token: invalid status code from registry 504 (Gateway Timeout)
time="2024-05-31T18:14:50+03:00" level=warning msg="Failed, retrying in 1s ... (1/3). Error: parsing image configuration: Get "https://xzxzxzxz/v2/vduproject/oamne3sadapt/blobs/sha256:fe435c56ee3225a1b2ebd34cdf693e37272ec3cc11bcdcda5e537cdf2b196ced\": EOF"
Copying blob sha256:ec7250d0b43078843415b0fc080372d03c48afa27dc48e351d353da5ada6d7c3
Copying blob sha256:cd1d83871e52f65d9d2ac52e7486c0e671b8e6c845ef6760710052a58d7315d1

same error in proxy.log

2024-05-31T18:14:48.928049004+03:00 stdout F 2024/05/31 15:14:48 [error] 7#0: *5336 upstream timed out (110: Connection timed out) while reading response header from upstream, client: xx.xxx.xxx.xx, server: , request: "GET /service/token?account=robot%24vduproject%2Bvdu_user&scope=repository%3Avduproject%2Fdocker-dhcp%3Apull&service=harbor-registry HTTP/1.1", upstream: "https://172.17.0.119:8443/service/token?account=robot%24vduproject%2Bvdu_user&scope=repository%3Avduproject%2Fdocker-dhcp%3Apull&service=harbor-registry", host: "xxxxx:8443"t

[13] 11:46:05 [SUCCESS] sprintlab780vm23
Error: initializing source docker://xxx:8443/vduproject/oamconfig:0.300.22229: reading manifest 0.300.22229 in xxxx:8443/vduproject/oamconfig: received unexpected HTTP status: 502 Bad Gateway

  proxy_connect_timeout 600;
  proxy_send_timeout 900;
  proxy_read_timeout 900;