Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DB connection spike when too many request comes from kapp-controller #16039

Closed
stonezdj opened this issue Nov 22, 2021 · 3 comments
Closed

DB connection spike when too many request comes from kapp-controller #16039

stonezdj opened this issue Nov 22, 2021 · 3 comments
Labels
area/performance Stale

Comments

@stonezdj
Copy link
Contributor

The kapp controller reconcile the application every 5 minutes, during the reconcile process, it pulls the manifest from the Harbor, the pull manifest operation result in too many database updates on the table

 UPDATE repository  SET  pull_count  =  pull_count + $1, update_time  = $2 WHERE  repository_id  IN ( SELECT T0.repository_id FROM repository T0 WHERE T0.repository_id = $3  )

 UPDATE artifact  SET pull_time  = $1 WHERE id = $2
wy65701436 added a commit to wy65701436/harbor that referenced this issue Nov 24, 2021
@wy65701436
add pull update control env
9bb46d3 
These variables are temporary solution for issue: goharbor#16039
When user disable the pull count/time/audit log, it will decrease the database access, especially in large concurrency pull scenarios.

1, PULL_TIME_UPDATE_DISABLE : The flag to indicate if pull time is disable for pull request.
2, PULL_COUNT_UPDATE_DISABLE : The flag to indicate if pull count is disable for pull request.
3, pull audit log will not create on disabling pull time.

Signed-off-by: Wang Yan <wangyan@vmware.com>
wy65701436 added a commit to wy65701436/harbor that referenced this issue Nov 24, 2021
@wy65701436
add pull update control env
1d0ffb5 
These variables are temporary solution for issue: goharbor#16039
When user disable the pull count/time/audit log, it will decrease the database access, especially in large concurrency pull scenarios.

1, PULL_TIME_UPDATE_DISABLE : The flag to indicate if pull time is disable for pull request.
2, PULL_COUNT_UPDATE_DISABLE : The flag to indicate if pull count is disable for pull request.
3, pull audit log will not create on disabling pull time.

Signed-off-by: Wang Yan <wangyan@vmware.com>
wy65701436 referenced this issue in wy65701436/harbor Nov 25, 2021
@wy65701436
add pull update control env
90a7f52 
These variables are temporary solution for issue: goharbor#16039
When user disable the pull count/time/audit log, it will decrease the database access, especially in large concurrency pull scenarios.

1, PULL_TIME_UPDATE_DISABLE : The flag to indicate if pull time is disable for pull request.
2, PULL_COUNT_UPDATE_DISABLE : The flag to indicate if pull count is disable for pull request.
3, pull audit log will not create on disabling pull time.

Signed-off-by: Wang Yan <wangyan@vmware.com>
@wy65701436 wy65701436 mentioned this issue Nov 26, 2021
Merged
wy65701436 referenced this issue Nov 26, 2021
@wy65701436
add pull update control env (#16061)
1ee3f51 
These variables are temporary solution for issue: goharbor#16039
When user disable the pull count/time/audit log, it will decrease the database access, especially in large concurrency pull scenarios.

1, PULL_TIME_UPDATE_DISABLE : The flag to indicate if pull time is disable for pull request.
2, PULL_COUNT_UPDATE_DISABLE : The flag to indicate if pull count is disable for pull request.
3, pull audit log will not create on disabling pull time.

Signed-off-by: Wang Yan <wangyan@vmware.com>
wy65701436 added a commit that referenced this issue Nov 26, 2021
@wy65701436
add pull update control env (#16051)
7608df4 
These variables are temporary solution for issue: #16039
When user disable the pull count/time/audit log, it will decrease the database access, especially in large concurrency pull scenarios.

1, PULL_TIME_UPDATE_DISABLE : The flag to indicate if pull time is disable for pull request.
2, PULL_COUNT_UPDATE_DISABLE : The flag to indicate if pull count is disable for pull request.
3, pull audit log will not create on disabling pull time.

Signed-off-by: Wang Yan <wangyan@vmware.com>
wy65701436 added a commit to wy65701436/harbor that referenced this issue Nov 29, 2021
@wy65701436
add pull update control env
c24f786 
These variables are temporary solution for issue: goharbor#16039
When user disable the pull count/time/audit log, it will decrease the database access, especially in large concurrency pull scenarios.

1, PULL_TIME_UPDATE_DISABLE : The flag to indicate if pull time is disable for pull request.
2, PULL_COUNT_UPDATE_DISABLE : The flag to indicate if pull count is disable for pull request.
3, pull audit log will not create on disabling pull time.

Signed-off-by: Wang Yan <wangyan@vmware.com>
wy65701436 added a commit to wy65701436/harbor that referenced this issue Nov 29, 2021
@wy65701436
add pull update control env
4849f00 
These variables are temporary solution for issue: goharbor#16039
When user disable the pull count/time/audit log, it will decrease the database access, especially in large concurrency pull scenarios.

1, PULL_TIME_UPDATE_DISABLE : The flag to indicate if pull time is disable for pull request.
2, PULL_COUNT_UPDATE_DISABLE : The flag to indicate if pull count is disable for pull request.
3, pull audit log will not create on disabling pull time.

Signed-off-by: Wang Yan <wangyan@vmware.com>
@wy65701436 wy65701436 mentioned this issue Nov 29, 2021
Merged
@dkulchinsky
Copy link
Contributor

@wy65701436 looks like PULL_TIME_UPDATE_DISABLE will also mitigate:

at least until a more specific fix is available to not update pull time when image pulled by a scanner.

prahaladdarkin pushed a commit to prahaladdarkin/harbor that referenced this issue Dec 4, 2021
@wy65701436 @prahaladdarkin
add pull update control env (goharbor#16051)
be2c9ac 
These variables are temporary solution for issue: goharbor#16039
When user disable the pull count/time/audit log, it will decrease the database access, especially in large concurrency pull scenarios.

1, PULL_TIME_UPDATE_DISABLE : The flag to indicate if pull time is disable for pull request.
2, PULL_COUNT_UPDATE_DISABLE : The flag to indicate if pull count is disable for pull request.
3, pull audit log will not create on disabling pull time.

Signed-off-by: Wang Yan <wangyan@vmware.com>
wy65701436 added a commit that referenced this issue Dec 16, 2021
@wy65701436
add pull update control env (#16063)
2e6650a 
These variables are temporary solution for issue: #16039
When user disable the pull count/time/audit log, it will decrease the database access, especially in large concurrency pull scenarios.

1, PULL_TIME_UPDATE_DISABLE : The flag to indicate if pull time is disable for pull request.
2, PULL_COUNT_UPDATE_DISABLE : The flag to indicate if pull count is disable for pull request.
3, pull audit log will not create on disabling pull time.

Signed-off-by: Wang Yan <wangyan@vmware.com>
stonezdj pushed a commit to stonezdj/harbor that referenced this issue Jan 4, 2022
@wy65701436 @stonezdj
add pull update control env (goharbor#16051)
c6594cb 
These variables are temporary solution for issue: goharbor#16039
When user disable the pull count/time/audit log, it will decrease the database access, especially in large concurrency pull scenarios.

1, PULL_TIME_UPDATE_DISABLE : The flag to indicate if pull time is disable for pull request.
2, PULL_COUNT_UPDATE_DISABLE : The flag to indicate if pull count is disable for pull request.
3, pull audit log will not create on disabling pull time.

Signed-off-by: Wang Yan <wangyan@vmware.com>
prahaladdarkin pushed a commit to prahaladdarkin/harbor that referenced this issue Mar 13, 2022
@wy65701436 @prahaladdarkin
add pull update control env (goharbor#16051)
a5a3bc6 
These variables are temporary solution for issue: goharbor#16039
When user disable the pull count/time/audit log, it will decrease the database access, especially in large concurrency pull scenarios.

1, PULL_TIME_UPDATE_DISABLE : The flag to indicate if pull time is disable for pull request.
2, PULL_COUNT_UPDATE_DISABLE : The flag to indicate if pull count is disable for pull request.
3, pull audit log will not create on disabling pull time.

Signed-off-by: Wang Yan <wangyan@vmware.com>
@stale
Copy link

stale bot commented Apr 16, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the Stale label Apr 16, 2022
@github-actions
Copy link

github-actions bot commented Jul 5, 2022

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

@github-actions github-actions bot closed this as completed Jul 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/performance Stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@stonezdj @dkulchinsky