Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

when push image concurrently, updating the quota will be stuck for a few seconds #18440

Closed
hcwnbs opened this issue Mar 28, 2023 · 1 comment · Fixed by #18871
Closed

when push image concurrently, updating the quota will be stuck for a few seconds #18440

hcwnbs opened this issue Mar 28, 2023 · 1 comment · Fixed by #18871
Assignees
@chlins
Labels
area/performance priority/high target/2.9.0

Comments

@hcwnbs
Copy link

hcwnbs commented Mar 28, 2023 •
edited

Steps to reproduce the problem:
push 20 image concurrently

  • harbor version: 2.6.0
  • docker engine version: 18.09

Expected behavior and actual behavior:
A clear and concise description of what you expected to happen and what's the actual behavior. If applicable, add screenshots to help explain your problem.

set orm.debug=true
I found that when update quota concurrently, harbor need to wait another request transaction to commit.

this is the orm log:
[ORM]2023/03/28 02:48:12 -[Queries/default] - [ OK / db.Exec / 3.2ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":43}, 2023-03-28 02:48:12, 43, 3, 42
[ORM]2023/03/28 02:48:20 -[Queries/default] - [ OK / db.Exec / 8030.1ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":43}, 2023-03-28 02:48:12, 43, 3, 42
[ORM]2023/03/28 02:48:20 -[Queries/default] - [ OK / db.Exec / 0.3ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":44}, 2023-03-28 02:48:20, 44, 3, 43
[ORM]2023/03/28 02:48:24 -[Queries/default] - [ OK / db.Exec / 11855.8ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":43}, 2023-03-28 02:48:12, 43, 3, 42
[ORM]2023/03/28 02:48:24 -[Queries/default] - [ OK / db.Exec / 2.8ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":45}, 2023-03-28 02:48:24, 45, 3, 44
[ORM]2023/03/28 02:48:26 -[Queries/default] - [ OK / db.Exec / 14335.1ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":43}, 2023-03-28 02:48:12, 43, 3, 42
[ORM]2023/03/28 02:48:26 -[Queries/default] - [ OK / db.Exec / 0.3ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":46}, 2023-03-28 02:48:26, 46, 3, 45
[ORM]2023/03/28 02:48:31 -[Queries/default] - [ OK / db.Exec / 16242.1ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":43}, 2023-03-28 02:48:15, 43, 3, 42
[ORM]2023/03/28 02:48:31 -[Queries/default] - [ OK / db.Exec / 0.3ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":47}, 2023-03-28 02:48:31, 47, 3, 46
[ORM]2023/03/28 02:48:34 -[Queries/default] - [ OK / db.Exec / 19062.1ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":43}, 2023-03-28 02:48:15, 43, 3, 42
[ORM]2023/03/28 02:48:34 -[Queries/default] - [ OK / db.Exec / 0.3ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":48}, 2023-03-28 02:48:34, 48, 3, 47
[ORM]2023/03/28 02:48:37 -[Queries/default] - [ OK / db.Exec / 21549.4ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":43}, 2023-03-28 02:48:15, 43, 3, 42
[ORM]2023/03/28 02:48:37 -[Queries/default] - [ OK / db.Exec / 0.2ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":49}, 2023-03-28 02:48:37, 49, 3, 48
[ORM]2023/03/28 02:48:39 -[Queries/default] - [ OK / db.Exec / 24045.9ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":43}, 2023-03-28 02:48:15, 43, 3, 42
[ORM]2023/03/28 02:48:39 -[Queries/default] - [ OK / db.Exec / 0.4ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":50}, 2023-03-28 02:48:39, 50, 3, 49
[ORM]2023/03/28 02:48:41 -[Queries/default] - [ OK / db.Exec / 25600.2ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":43}, 2023-03-28 02:48:15, 43, 3, 42
[ORM]2023/03/28 02:48:41 -[Queries/default] - [ OK / db.Exec / 0.3ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":51}, 2023-03-28 02:48:41, 51, 3, 50
[ORM]2023/03/28 02:48:44 -[Queries/default] - [ OK / db.Exec / 28516.0ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":43}, 2023-03-28 02:48:15, 43, 3, 42
[ORM]2023/03/28 02:48:44 -[Queries/default] - [ OK / db.Exec / 0.8ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":52}, 2023-03-28 02:48:44, 52, 3, 51
[ORM]2023/03/28 02:48:46 -[Queries/default] - [ OK / db.Exec / 30008.8ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":43}, 2023-03-28 02:48:16, 43, 3, 42
[ORM]2023/03/28 02:48:46 -[Queries/default] - [ OK / db.Exec / 0.4ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":53}, 2023-03-28 02:48:46, 53, 3, 52
[ORM]2023/03/28 02:48:49 -[Queries/default] - [ OK / db.Exec / 32268.1ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":43}, 2023-03-28 02:48:17, 43, 3, 42
[ORM]2023/03/28 02:48:49 -[Queries/default] - [ OK / db.Exec / 0.4ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":54}, 2023-03-28 02:48:49, 54, 3, 53
[ORM]2023/03/28 02:48:53 -[Queries/default] - [ OK / db.Exec / 35709.3ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":43}, 2023-03-28 02:48:17, 43, 3, 42
[ORM]2023/03/28 02:48:53 -[Queries/default] - [ OK / db.Exec / 0.4ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":55}, 2023-03-28 02:48:53, 55, 3, 54
[ORM]2023/03/28 02:48:54 -[Queries/default] - [ OK / db.Exec / 37307.4ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":43}, 2023-03-28 02:48:17, 43, 3, 42
[ORM]2023/03/28 02:48:54 -[Queries/default] - [ OK / db.Exec / 0.6ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":56}, 2023-03-28 02:48:54, 56, 3, 55
[ORM]2023/03/28 02:48:55 -[Queries/default] - [ OK / db.Exec / 37694.6ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":43}, 2023-03-28 02:48:17, 43, 3, 42
[ORM]2023/03/28 02:48:55 -[Queries/default] - [ OK / db.Exec / 0.8ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":57}, 2023-03-28 02:48:55, 57, 3, 56
[ORM]2023/03/28 02:48:56 -[Queries/default] - [ OK / db.Exec / 35799.3ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":44}, 2023-03-28 02:48:20, 44, 3, 43
[ORM]2023/03/28 02:48:56 -[Queries/default] - [ OK / db.Exec / 0.3ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":58}, 2023-03-28 02:48:56, 58, 3, 57
[ORM]2023/03/28 02:48:57 -[Queries/default] - [ OK / db.Exec / 33438.3ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":45}, 2023-03-28 02:48:24, 45, 3, 44
[ORM]2023/03/28 02:48:57 -[Queries/default] - [ OK / db.Exec / 1.1ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":59}, 2023-03-28 02:48:57, 59, 3, 58
[ORM]2023/03/28 02:49:00 -[Queries/default] - [ OK / db.Exec / 33230.3ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":46}, 2023-03-28 02:48:26, 46, 3, 45
[ORM]2023/03/28 02:49:00 -[Queries/default] - [ OK / db.Exec / 1.2ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":60}, 2023-03-28 02:49:00, 60, 3, 59
[ORM]2023/03/28 02:49:03 -[Queries/default] - [ OK / db.Exec / 32046.0ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":47}, 2023-03-28 02:48:31, 47, 3, 46
[ORM]2023/03/28 02:49:03 -[Queries/default] - [ OK / db.Exec / 0.6ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":61}, 2023-03-28 02:49:03, 61, 3, 60
[ORM]2023/03/28 02:49:08 -[Queries/default] - [ OK / db.Exec / 34437.8ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":48}, 2023-03-28 02:48:34, 48, 3, 47
[ORM]2023/03/28 02:49:08 -[Queries/default] - [ OK / db.Exec / 0.5ms] - [UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5] - {"number":62}, 2023-03-28 02:49:08, 62, 3, 61
@chlins chlins mentioned this issue Apr 20, 2023
Closed
@LemonHXY
Copy link

LemonHXY commented May 27, 2023 •
edited

Hi, what is the solution to this problem? I've had similar problems. @chlins

chlins pushed a commit to chlins/harbor that referenced this issue Jun 30, 2023
perf: introduce update quota by redis
76f6bb6 
Introduce the quota update provider, improve the performance of pushing
artifacts to same project with high concurrency by implementing
optimistic lock in redis. By default the function is disabled, open it
by set env 'QUOTA_UPDATE_PROVIDER=Redis' for the core container.

Fixes: goharbor#18440

Signed-off-by: chlins <chenyuzh@vmware.com>
@chlins chlins mentioned this issue Jun 30, 2023
Merged
5 tasks
chlins pushed a commit to chlins/harbor that referenced this issue Jun 30, 2023
perf: introduce update quota by redis
8feaa6b 
Introduce the quota update provider, improve the performance of pushing
artifacts to same project with high concurrency by implementing
optimistic lock in redis. By default the function is disabled, open it
by set env 'QUOTA_UPDATE_PROVIDER=Redis' for the core container.

Fixes: goharbor#18440

Signed-off-by: chlins <chenyuzh@vmware.com>
chlins pushed a commit to chlins/harbor that referenced this issue Jul 10, 2023
perf: introduce update quota by redis
d7fdfc5 
Introduce the quota update provider, improve the performance of pushing
artifacts to same project with high concurrency by implementing
optimistic lock in redis. By default the function is disabled, open it
by set env 'QUOTA_UPDATE_PROVIDER=Redis' for the core container.

Fixes: goharbor#18440

Signed-off-by: chlins <chenyuzh@vmware.com>
chlins pushed a commit that referenced this issue Jul 10, 2023
perf: introduce update quota by redis (#18871)
78799cc 
Introduce the quota update provider, improve the performance of pushing
artifacts to same project with high concurrency by implementing
optimistic lock in redis. By default the function is disabled, open it
by set env 'QUOTA_UPDATE_PROVIDER=Redis' for the core container.

Fixes: #18440

Signed-off-by: chlins <chenyuzh@vmware.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chlins chlins

Labels
area/performance priority/high target/2.9.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

perf: introduce update quota by redis chlins/harbor
5 participants
@wy65701436 @zyyw @hcwnbs @chlins @LemonHXY