Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

git LFS and storage type minio: Massive memory usage #15253

Closed
1 of 4 tasks
jochenhz opened this issue Apr 2, 2021 · 3 comments · Fixed by #15264
Closed
1 of 4 tasks

git LFS and storage type minio: Massive memory usage #15253

jochenhz opened this issue Apr 2, 2021 · 3 comments · Fixed by #15264
Labels
performance/memory Performance issues affecting memory use type/upstream This is an issue in one of Gitea's dependencies and should be reported there
Milestone
1.13.7

Comments

@jochenhz
Copy link

jochenhz commented Apr 2, 2021

  • Gitea version (or commit ref): 1.14.0-rc2
  • Git version: 2.30.2
  • Operating system: docker
  • Database (use [x]):
    • PostgreSQL
    • MySQL
    • MSSQL
    • SQLite
  • Log gist (heap, allocs, gitea.log):
    Logs.zip

Description

This is a follow up issue from the discord general channel.
Running git lfs push with very small LFS files (330kb in size total) and storage type minio causes a massive use of memory of gitea (around 2.2 GB). Pushing another round of small files quickly exceeds 5GB.

I’ve attached a zip containing a docker compose and test data so that you can reproduce the issue easily.
docker.zip

Reproduction steps:

  1. Extract docker.zip
  2. Adjust IP address for server.root_url and storage.minio.minio_endpoint in config/app.ini
  3. Run: docker-compose -f docker-compose.yml up -d
  4. Go to localhost and setup gitea
  5. Create a new repo + initialize
  6. Run: git clone
  7. Cd into cloned repo
  8. Run: git lfs install
  9. Run: git lfs track *.png
  10. Copy provided PNG images to repository
  11. Run: git add .
  12. Run: git lfs status - and verify that LFS files are tracked correctly
  13. Run: git commit and push
  14. Observe gitea memory usage spikes from 180 mb to around 2.2 GB
@jochenhz jochenhz mentioned this issue Apr 2, 2021
Closed
@lunny lunny added the performance/memory Performance issues affecting memory use label Apr 3, 2021
@lunny
Copy link
Member

lunny commented Apr 3, 2021

This is upstream related. This may happen when give the size as -1 when upload with minio sdk. Could you send an issue to https://github.com/minio/minio-go/issues/new

@lunny lunny added the type/upstream This is an issue in one of Gitea's dependencies and should be reported there label Apr 3, 2021
@lunny lunny modified the milestones: 1.15.0, 1.13.7 Apr 3, 2021
@jochenhz jochenhz mentioned this issue Apr 3, 2021
Closed
zeripath added a commit to zeripath/gitea that referenced this issue Apr 3, 2021
@zeripath
Alternative to go-gitea#15255 - add size to Save function
89fb6d0 
This PR proposes an alternative solution to go-gitea#15255 - just add the size to the
save function. Yes it is less apparently clean but it may be more correct.

Close go-gitea#15255
Fix go-gitea#15253

Signed-off-by: Andrew Thornton <art27@cantab.net>
@zeripath zeripath mentioned this issue Apr 3, 2021
Merged
@jochenhz
Copy link
Author

jochenhz commented Apr 3, 2021

fyi there is some update on the minio-go issue:
minio/minio-go#1473 (comment)

6543 pushed a commit that referenced this issue Apr 3, 2021
@zeripath
Add size to Save function (#15264)
0bb8bd8 
Fix #15253

Signed-off-by: Andrew Thornton <art27@cantab.net>
@lunny
Copy link
Member

lunny commented Apr 3, 2021

Add PartSize to PutObjectOptions could be another PR.

zeripath added a commit to zeripath/gitea that referenced this issue Apr 3, 2021
@zeripath
Add size to Save function (go-gitea#15264)
15700b3 
This PR proposes an alternative solution to go-gitea#15255 - just add the size to the
save function. Yes it is less apparently clean but it may be more correct.

Close go-gitea#15255
Fix go-gitea#15253

Signed-off-by: Andrew Thornton <art27@cantab.net>
@zeripath zeripath mentioned this issue Apr 3, 2021
Merged
zeripath added a commit to zeripath/gitea that referenced this issue Apr 3, 2021
@zeripath
Add size to Save function (go-gitea#15264)
af941b4 
Backport go-gitea#15264

This PR proposes an alternative solution to go-gitea#15255 - just add the size to the
save function. Yes it is less apparently clean but it may be more correct.

Close go-gitea#15255
Fix go-gitea#15253

Signed-off-by: Andrew Thornton <art27@cantab.net>
@zeripath zeripath mentioned this issue Apr 3, 2021
Merged
techknowlogick pushed a commit that referenced this issue Apr 4, 2021
@zeripath
Add size to Save function (#15264) (#15271)
013639b 
Backport #15264

This PR proposes an alternative solution to #15255 - just add the size to the
save function. Yes it is less apparently clean but it may be more correct.

Close #15255
Fix #15253

Signed-off-by: Andrew Thornton <art27@cantab.net>
techknowlogick pushed a commit that referenced this issue Apr 4, 2021
@zeripath @lunny
Add size to Save function (#15264) (#15270)
af73e1e 
This PR proposes an alternative solution to #15255 - just add the size to the
save function. Yes it is less apparently clean but it may be more correct.

Close #15255
Fix #15253

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
@go-gitea go-gitea locked and limited conversation to collaborators May 13, 2021
AbdulrhmnGhanem pushed a commit to kitspace/gitea that referenced this issue Aug 10, 2021
@zeripath @AbdulrhmnGhanem
Add size to Save function (go-gitea#15264)
8a55646 
Fix go-gitea#15253

Signed-off-by: Andrew Thornton <art27@cantab.net>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
performance/memory Performance issues affecting memory use type/upstream This is an issue in one of Gitea's dependencies and should be reported there
Projects
None yet
Milestone
1.13.7
Development

Successfully merging a pull request may close this issue.

Add size to Save function zeripath/gitea
Some optimizations for uploading to minio lunny/gitea
2 participants
@lunny @jochenhz