Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor editor upload, update and delete to use git plumbing and add LFS support #5702

Merged
merged 29 commits into from
Feb 12, 2019

Conversation

zeripath
Copy link
Contributor

@zeripath zeripath commented Jan 11, 2019

When uploading files using the GUI, Gitea currently does:

a) A full clone - Including the whole object DB
b) A full branch checkout - every file
c) Copies the uploaded file to this cloned repository
d) Add/stages the changes
e) Commits
f) Pushes to the real repository
g) Cleans up the cloned repository - deleting the whole object db and checkout.

This is extremely inefficient.

This PR takes a different approach and, instead of using the git porcelain commands (git add and the like), uses the plumbing commands to (hopefully) significantly speed up this process. See my comments in #601 (#601 (comment)).

The basics of this code was in #5621, however, there has been significant refactoring with code moved out of models.Repository into a module called uploader. LFS support has been added, with transparent recognition of whether a file should be added to the LFS through checking the git attributes.

Issues

  • Currently my approach is to use the git-cli commands which can/could be migrated to go-git commands later. Some of this code could and probably should be moved into the gitea git subproject.
  • There are inefficiencies in the LFS code which could be improved - for example, when files are uploaded to the server, their sha1dcsum and sha256sum should be created at that point rather than read the whole file again - and the LFS ContentStore will currently regenerate the sha256sum and copy the file again.
  • I don't set ORIGINAL_SSH_COMMAND so the push hooks may not actually run. This is to stay in keeping with the current implementation - however, could be very easily set.
  • It's not possible to update LFS files, although it is possible to create new text files that will become LFS files.

Will fix #5600. Will fix #4778. Will fix #3557. Will fix #5727.

@codecov-io
Copy link

codecov-io commented Jan 11, 2019

Codecov Report

❗ No coverage uploaded for pull request base (master@baffea1). Click here to learn what that means.
The diff coverage is 32.51%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master    #5702   +/-   ##
=========================================
  Coverage          ?   38.82%           
=========================================
  Files             ?      344           
  Lines             ?    49355           
  Branches          ?        0           
=========================================
  Hits              ?    19163           
  Misses            ?    27428           
  Partials          ?     2764
Impacted Files Coverage Δ
modules/uploader/diff.go 0% <0%> (ø)
modules/uploader/delete.go 0% <0%> (ø)
models/lfs.go 32.72% <0%> (ø)
models/upload.go 0% <0%> (ø)
modules/uploader/upload.go 0% <0%> (ø)
routers/repo/editor.go 28.67% <26.31%> (ø)
modules/uploader/update.go 35.71% <35.71%> (ø)
models/repo_branch.go 58.27% <56%> (ø)
modules/uploader/repo.go 67.04% <67.04%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update baffea1...9c3603b. Read the comment docs.

@bkcsoft bkcsoft added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Jan 11, 2019
@zeripath
Copy link
Contributor Author

zeripath commented Jan 11, 2019

As previously described in #5621 this provides around a 10x speed up on the current implementation even for very small repositories.


Speed ups for large repositories should be considerably better.

@lunny lunny added the type/refactoring Existing code has been cleaned up. There should be no new functionality. label Jan 12, 2019
@lunny lunny added this to the 1.8.0 milestone Jan 12, 2019
@zeripath zeripath mentioned this pull request Jan 14, 2019
7 tasks
models/upload.go Outdated Show resolved Hide resolved
models/upload.go Outdated Show resolved Hide resolved
models/upload.go Show resolved Hide resolved
    This is a change from the previous code but is more in keeping
    with the default behaviour of git.

Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution.
see go-gitea#5774

Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
@lafriks lafriks added type/enhancement An improvement of existing functionality type/changelog Adds the changelog for a new Gitea version labels Feb 7, 2019
@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Feb 7, 2019
@zeripath
Copy link
Contributor Author

zeripath commented Feb 8, 2019

@lunny needs your review.

@zeripath
Copy link
Contributor Author

Ping @lunny

@zeripath
Copy link
Contributor Author

Make lg TM work

@lafriks lafriks merged commit 296814e into go-gitea:master Feb 12, 2019
@zeripath zeripath deleted the make-upload-not-clone branch February 12, 2019 19:38
@richmahn
Copy link
Contributor

Any way to retain commit history when a file is renamed in the GUI? Now it deletes the old file, makes a new file. That isn't the behavior of GitHub when you change the name or path of an existing file.

@lafriks
Copy link
Member

lafriks commented Feb 16, 2019

@richmahn please submit issue for this

@go-gitea go-gitea locked and limited conversation to collaborators Nov 24, 2020
@delvh delvh removed the type/changelog Adds the changelog for a new Gitea version label Oct 7, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. type/bug type/enhancement An improvement of existing functionality type/refactoring Existing code has been cleaned up. There should be no new functionality.
Projects
None yet
9 participants