Return error when proxy cache get too many request error(429) #18728

stonezdj · 2023-05-24T03:32:11Z

Add 429 too many request error in http error
Fixes #18707

Thank you for contributing to Harbor!

Comprehensive Summary of your change

Issue being fixed

Fixes #(issue)

Please indicate you've done the following:

Well Written Title and Summary of the PR
Label the PR as needed. "release-note/ignore-for-release, release-note/new-feature, release-note/update, release-note/enhancement, release-note/community, release-note/breaking-change, release-note/docs, release-note/infra, release-note/deprecation"
Accepted the DCO. Commits without the DCO will delay acceptance.
Made sure tests are passing and test coverage is added if needed.
Considered the docs impact and opened a new docs issue or PR with docs changes if needed in website repository.

codecov · 2023-05-24T03:35:45Z

Codecov Report

Merging #18728 (0bbe5d9) into main (210186f) will decrease coverage by 0.02%.
The diff coverage is 16.66%.

@@            Coverage Diff             @@
##             main   #18728      +/-   ##
==========================================
- Coverage   67.38%   67.37%   -0.02%     
==========================================
  Files         980      980              
  Lines      106749   106759      +10     
  Branches     2665     2665              
==========================================
- Hits        71937    71932       -5     
- Misses      30948    30965      +17     
+ Partials     3864     3862       -2

Flag	Coverage Δ
unittests	`67.37% <16.66%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/controller/proxy/controller.go	`15.84% <0.00%> (+0.92%)`	⬆️
src/lib/http/error.go	`92.68% <ø> (ø)`
src/pkg/registry/client.go	`42.99% <0.00%> (-0.18%)`	⬇️
src/server/middleware/repoproxy/proxy.go	`2.70% <0.00%> (-0.05%)`	⬇️
src/lib/errors/const.go	`92.30% <100.00%> (+0.64%)`	⬆️

... and 9 files with indirect coverage changes

wy65701436 · 2023-05-24T05:13:21Z

src/server/middleware/repoproxy/proxy.go

@@ -46,6 +46,8 @@ const (
 	ensureTagMaxRetry   = 60
 )

+var tooManyRequestsError = errors.New("too many requests to upstream registry").WithCode(errors.RateLimitCode)


You have exceeded the pull rate limit for the proxy cache.

wy65701436

lgtm

reasonerjt · 2023-05-24T05:19:27Z

src/controller/proxy/controller.go

@@ -155,6 +155,9 @@ func (c *controller) UseLocalManifest(ctx context.Context, art lib.ArtifactInfo,
 	remoteRepo := getRemoteRepo(art)
 	exist, desc, err := remote.ManifestExist(remoteRepo, getReference(art)) // HEAD
 	if err != nil {
+		if errors.IsRateLimitError(err) && a != nil { // if rate limit, use local if it exists, otherwise return error
+			return true, nil, nil
+		}


I don't think we should handle 429 as a special case, we handle any error the same way, i.e. when error happens refreshing the cache and the cache is not expired, we log the error and serve the data in the cache.

IMHO, what needs to be refined is in the path when an error happens and the cache is expired or the data is not cached.

In this case, we better return the error more explicitly instead of a 404. I think a better choice here is to return 500 which wraps the root cause (in this case 429), if 500 can't be rendered in a friendly way for mainstream clients, we find a proper error so most clients can handle it.

To avoid leaking any server information. if return 500 error to the client, it just sends the "internal server error" to the client, no other useful information.
see
https://github.com/goharbor/harbor/blob/cc4c08b0a1cef086fdb47d90be8f6b32efe170cf/src/lib/http/error.go#L55-L54

reasonerjt · 2023-05-25T08:13:49Z

I'm not quite convinced why we fix 429 but leave other error-handling situations as-is.
Returning a 404 for whatever backend error when the local cache does not exist is as confusing as the problem this PR is trying to solve.

That said it's up to @stonezdj to make decision.

stonezdj · 2023-05-25T10:20:19Z

I'm not quite convinced why we fix 429 but leave other error-handling situations as-is. Returning a 404 for whatever backend error when the local cache does not exist is as confusing as the problem this PR is trying to solve.

That said it's up to @stonezdj to make decision.

For any other error:
If the Harbor can return an internal server error 500 to the client when there is error and no cache can be served in local
return 404 if fallback to local and no cache can be served,
It doesn't make too much difference to the client side.

For the least impact to the current system, keep the current implementation.

Add 429 too many request error in http error Fixes goharbor#18707 Signed-off-by: stonezdj <stonezdj@gmail.com>

…or#18728) Add 429 too many request error in http error Fixes goharbor#18707 Signed-off-by: stonezdj <stonezdj@gmail.com> Signed-off-by: Wilfred Almeida <60785452+WilfredAlmeida@users.noreply.github.com>

…or#18728) Add 429 too many request error in http error Fixes goharbor#18707 Signed-off-by: stonezdj <stonezdj@gmail.com>

stonezdj added the release-note/update Update or Fix label May 24, 2023

stonezdj assigned wy65701436 May 24, 2023

stonezdj requested a review from a team as a code owner May 24, 2023 03:32

stonezdj assigned chlins May 24, 2023

github-actions bot assigned AllForNothing May 24, 2023

stonezdj force-pushed the 23may24_return_429_proxy_cache branch from 7a765eb to cc4c08b Compare May 24, 2023 03:47

wy65701436 reviewed May 24, 2023

View reviewed changes

wy65701436 approved these changes May 24, 2023

View reviewed changes

reasonerjt reviewed May 24, 2023

View reviewed changes

stonezdj force-pushed the 23may24_return_429_proxy_cache branch from cc4c08b to e1bc03d Compare May 25, 2023 06:54

stonezdj requested a review from zyyw May 27, 2023 01:09

stonezdj assigned zyyw May 27, 2023

zyyw approved these changes May 29, 2023

View reviewed changes

stonezdj force-pushed the 23may24_return_429_proxy_cache branch from e1bc03d to 8224aaf Compare May 29, 2023 03:08

Return error when proxy cache get too many request error(429)

0bbe5d9

Add 429 too many request error in http error Fixes goharbor#18707 Signed-off-by: stonezdj <stonezdj@gmail.com>

stonezdj force-pushed the 23may24_return_429_proxy_cache branch from 8224aaf to 0bbe5d9 Compare May 29, 2023 04:40

stonezdj merged commit 1b1af4a into goharbor:main May 29, 2023
10 of 12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Return error when proxy cache get too many request error(429) #18728

Return error when proxy cache get too many request error(429) #18728

stonezdj commented May 24, 2023 •

edited

codecov bot commented May 24, 2023 •

edited

wy65701436 May 24, 2023

wy65701436 left a comment

reasonerjt May 24, 2023 •

edited

stonezdj May 25, 2023

reasonerjt commented May 25, 2023

stonezdj commented May 25, 2023 •

edited

Return error when proxy cache get too many request error(429) #18728

Return error when proxy cache get too many request error(429) #18728

Conversation

stonezdj commented May 24, 2023 • edited

Comprehensive Summary of your change

Issue being fixed

codecov bot commented May 24, 2023 • edited

Codecov Report

wy65701436 May 24, 2023

Choose a reason for hiding this comment

wy65701436 left a comment

Choose a reason for hiding this comment

reasonerjt May 24, 2023 • edited

Choose a reason for hiding this comment

stonezdj May 25, 2023

Choose a reason for hiding this comment

reasonerjt commented May 25, 2023

stonezdj commented May 25, 2023 • edited

stonezdj commented May 24, 2023 •

edited

codecov bot commented May 24, 2023 •

edited

reasonerjt May 24, 2023 •

edited

stonezdj commented May 25, 2023 •

edited