feat: add cache layer for artifact

[chlins]

Implement cache layer for resource artifact and define common
cache manager and workflow. Also add cache related options to
configuration yaml.

Closes: #16739

Signed-off-by: chlins chenyuzh@vmware.com

Comparison

Get the specific artifact

/projects/{project_name}/repositories/{repository_name}/artifacts/{reference}

Get by digest

script: hey -c 2000 -z 1m -H "Authorization: Basic YWRtaW46SGFyYm9yMTIzNDUK" http://harbor.domain/api/v2.0/projects/library/repositories/busybox/artifacts/sha256:14d4f50961544fdb669075c442509f194bdc4c0e344bde06e35dbd55af842a38

2.5:

Summary:
  Total:	62.0148 secs
  Slowest:	16.1262 secs
  Fastest:	0.1568 secs
  Average:	1.9981 secs
  Requests/sec:	980.0083

  Total data:	68736525 bytes
  Size/request:	1131 bytes

Response time histogram:
  0.157 [1]	|
  1.754 [32982]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  3.351 [21843]	|■■■■■■■■■■■■■■■■■■■■■■■■■■
  4.948 [3915]	|■■■■■
  6.545 [1165]	|■
  8.141 [490]	|■
  9.738 [229]	|
  11.335 [106]	|
  12.932 [31]	|
  14.529 [6]	|
  16.126 [7]	|


Latency distribution:
  10% in 0.9956 secs
  25% in 1.2842 secs
  50% in 1.6807 secs
  75% in 2.2534 secs
  90% in 3.3216 secs
  95% in 4.3125 secs
  99% in 7.2571 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0019 secs, 0.1568 secs, 16.1262 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0000 secs
  req write:	0.0000 secs, 0.0000 secs, 0.0360 secs
  resp wait:	1.9959 secs, 0.1468 secs, 16.1262 secs
  resp read:	0.0000 secs, 0.0000 secs, 0.0131 secs

Status code distribution:
  [200]	60775 responses

after:

Summary:
  Total:	61.6996 secs
  Slowest:	4.3489 secs
  Fastest:	0.0143 secs
  Average:	0.5151 secs
  Requests/sec:	3786.3100

  Total data:	264217434 bytes
  Size/request:	1131 bytes

Response time histogram:
  0.014 [1]	|
  0.448 [154944]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.881 [41623]	|■■■■■■■■■■■
  1.315 [19779]	|■■■■■
  1.748 [8368]	|■■
  2.182 [3642]	|■
  2.615 [3284]	|■
  3.049 [1445]	|
  3.482 [512]	|
  3.915 [5]	|
  4.349 [11]	|


Latency distribution:
  10% in 0.1628 secs
  25% in 0.2302 secs
  50% in 0.3355 secs
  75% in 0.5766 secs
  90% in 1.1193 secs
  95% in 1.5810 secs
  99% in 2.5418 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0004 secs, 0.0143 secs, 4.3489 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0000 secs
  req write:	0.0001 secs, 0.0000 secs, 0.0250 secs
  resp wait:	0.5146 secs, 0.0142 secs, 4.3254 secs
  resp read:	0.0000 secs, 0.0000 secs, 0.0156 secs

Status code distribution:
  [200]	233614 responses

In conclusion, QPS improvement is about 3.8X and P99 latency is 2.8X faster than 2.5.

Get by tag

script: hey -c 2000 -z 1m -H "Authorization: Basic YWRtaW46SGFyYm9yMTIzNDUK" http://harbor.domain/api/v2.0/projects/library/repositories/busybox/artifacts/latest

2.5:

Summary:
  Total:	62.1554 secs
  Slowest:	6.1563 secs
  Fastest:	0.0137 secs
  Average:	0.7875 secs
  Requests/sec:	2478.1917

  Total data:	178593918 bytes
  Size/request:	1159 bytes

Response time histogram:
  0.014 [1]	|
  0.628 [100809]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  1.242 [25355]	|■■■■■■■■■■
  1.856 [11940]	|■■■■■
  2.471 [10592]	|■■■■
  3.085 [3322]	|■
  3.699 [1500]	|■
  4.314 [385]	|
  4.928 [97]	|
  5.542 [25]	|
  6.156 [6]	|


Latency distribution:
  10% in 0.3182 secs
  25% in 0.3990 secs
  50% in 0.5095 secs
  75% in 0.8389 secs
  90% in 1.8815 secs
  95% in 2.2722 secs
  99% in 3.2261 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0005 secs, 0.0137 secs, 6.1563 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0000 secs
  req write:	0.0001 secs, 0.0000 secs, 0.0232 secs
  resp wait:	0.7868 secs, 0.0136 secs, 6.1563 secs
  resp read:	0.0000 secs, 0.0000 secs, 0.0114 secs

Status code distribution:
  [200]	148699 responses
  [503]	5333 responses

after:

Summary:
  Total:	61.8543 secs
  Slowest:	3.5923 secs
  Fastest:	0.0316 secs
  Average:	0.5456 secs
  Requests/sec:	3573.9795

  Total data:	250025646 bytes
  Size/request:	1131 bytes

Response time histogram:
  0.032 [1]	|
  0.388 [87192]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.744 [90451]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  1.100 [26736]	|■■■■■■■■■■■■
  1.456 [11829]	|■■■■■
  1.812 [2834]	|■
  2.168 [1513]	|■
  2.524 [330]	|
  2.880 [121]	|
  3.236 [50]	|
  3.592 [9]	|


Latency distribution:
  10% in 0.2686 secs
  25% in 0.3328 secs
  50% in 0.4337 secs
  75% in 0.6489 secs
  90% in 1.0068 secs
  95% in 1.2196 secs
  99% in 1.7883 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0006 secs, 0.0316 secs, 3.5923 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0000 secs
  req write:	0.0001 secs, 0.0000 secs, 0.0634 secs
  resp wait:	0.5447 secs, 0.0316 secs, 3.4957 secs
  resp read:	0.0000 secs, 0.0000 secs, 0.0169 secs

Status code distribution:
  [200]	221066 responses

In conclusion, QPS improvement is about 1.4X and P99 latency is 1.8X faster than 2.5, and also success rate improve to 100%.

[codecov]

Codecov Report

Merging #16593 (8c22313) into main (fbff7ab) will increase coverage by 0.06%.
The diff coverage is 65.54%.

@@            Coverage Diff             @@
##             main   #16593      +/-   ##
==========================================
+ Coverage   67.29%   67.35%   +0.06%     
==========================================
  Files         953      956       +3     
  Lines       79048    79191     +143     
  Branches     2332     2332              
==========================================
+ Hits        53195    53340     +145     
+ Misses      22279    22248      -31     
- Partials     3574     3603      +29     

Flag	Coverage Δ
unittests	67.35% <65.54%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/lib/cache/cache.go	100.00% <ø> (ø)
src/pkg/artifact/manager.go	63.26% <ø> (ø)
src/pkg/cached/artifact/redis/manager.go	52.32% <52.32%> (ø)
src/lib/cache/redis/redis.go	58.73% <54.54%> (-0.89%)	⬇️
src/lib/cache/memory/memory.go	80.85% <70.00%> (-2.94%)	⬇️
src/lib/config/systemconfig.go	43.67% <77.77%> (+3.93%)	⬆️
src/controller/artifact/abstractor.go	66.66% <100.00%> (ø)
src/controller/artifact/controller.go	64.51% <100.00%> (ø)
src/controller/icon/controller.go	61.70% <100.00%> (ø)
src/controller/repository/controller.go	63.07% <100.00%> (ø)
... and 13 more

[wy65701436]

lgtm