Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for sha256 repositories #23894

Merged
merged 38 commits into from
Jan 19, 2024
Merged

Add support for sha256 repositories #23894

merged 38 commits into from
Jan 19, 2024

Conversation

AdamMajer
Copy link
Contributor

@AdamMajer AdamMajer commented Apr 3, 2023

Currently only SHA1 repositories are supported by Gitea. This adds support for alternate SHA256 with the additional aim of easier support for additional hash types in the future.

Fixes: #13794
Limited by: go-git/go-git#899
Depend on: #28138

图片

@AdamMajer
Copy link
Contributor Author

AdamMajer commented Apr 3, 2023

As additional comments for this PR,

  • It's WIP. It compiles now and passes tests (unit tests and the sqlite tests) and I've managed to import a SHA256 repo and things are not exploding anymore
  • Many things are not yet tested
  • This is my first project where I need to use Go :-)
  • This is my first attempt to understand Gitea
  • GoGit is completely unsupported by this patch

Comments are welcome. I do not expect this to land soon and I expect some changes.

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Apr 3, 2023
@techknowlogick techknowlogick added type/feature Completely new functionality. Can only be merged if feature freeze is not active. pr/wip This PR is not ready for review labels Apr 3, 2023
@techknowlogick techknowlogick added this to the 1.20.0 milestone Apr 3, 2023
@techknowlogick
Copy link
Member

Whoa! This is a huge accomplishment! Thank you <3

@AdamMajer
Copy link
Contributor Author

There is a funny issue I've reported upstream (git) now. And it's funny by semi-blocking here.

If you do,

git init --object-format=sha256 a
git clone a b
git-receive-pack --advertise-refs a
git-receive-pack --advertise-refs b

You will see respective results,

00cb0000000000000000000000000000000000000000000000000000000000000000
capabilities^{}report-status report-status-v2 delete-refs side-band-64k quiet atomic ofs-delta object-format=sha256 agent=git/2.35.3

00b10000000000000000000000000000000000000000
capabilities^{}report-status report-status-v2 delete-refs side-band-64k quiet atomic ofs-delta object-format=sha1 agent=git/2.35.3

Basically, the git-clone uses git-upload-pack to fetch the remote objects, but this doesn't send the header if there are no objects. git-receive-pack, used as part of push, does send a NULL OID in case of empty repository, so hash type can be determined. git-upload-pack does not, so git-clone initializes uses default hash, and this is sha1.

Gitea uses git-clone everywhere as ways of checking out a branch and inserting README.md or LICENSE or whatever. This of course fails on an empty repository. So, to make SHA256 usable without errors everywhere, the git-clone needs to have have the hash passed to it. Fortunately, this is possible by using GIT_DEFAULT_HASH env variable, otherwise there would be no workaround. But, this will make the patch unnecessarily more intrusive, and it will not fix the case where you do a git clone of remote empty sha256 repository to initialize it. You have to do a git init --object-type=sha256 instead. At least until upsteam git issue is fixed.

TL;DR: This means that if you create a SHA256 repository with nothing in it and clone it, you will get a SHA1 repository. This then will fail to push.

@AdamMajer
Copy link
Contributor Author

Looks like we have a preliminary patch for this issue upstream now git/git@8b214c2

@AdamMajer AdamMajer force-pushed the sha256 branch 2 times, most recently from 19c5107 to 05d320a Compare April 6, 2023 12:49
models/repo/repo.go Outdated Show resolved Hide resolved
@AdamMajer
Copy link
Contributor Author

Looks like go-git/go-git#707 is starting point for SHA256 go-git implementation. I'll refactor my patches to align more with those changes.

@pull-request-size pull-request-size bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label May 10, 2023
@AdamMajer AdamMajer force-pushed the sha256 branch 2 times, most recently from dbaf01f to 93219a8 Compare May 12, 2023 15:16
models/git/commit_status.go Outdated Show resolved Hide resolved
@wxiaoguang wxiaoguang removed this from the 1.20.0 milestone May 27, 2023
@AdamMajer AdamMajer force-pushed the sha256 branch 3 times, most recently from 8f5987a to 9fd63ce Compare June 20, 2023 11:55
@AdamMajer
Copy link
Contributor Author

The patch is now rebased on top of latest main and I've added DB migration for larger hash fields that I've tested with postgres. Next step is to finally add gogit support.

@lunny lunny mentioned this pull request Sep 9, 2023
@6543
Copy link
Member

6543 commented Sep 11, 2023

is this you @AdamMajer -> https://sha2git.com ?!?

@AdamMajer
Copy link
Contributor Author

is this you @AdamMajer -> https://sha2git.com ?!?

No, I'm not related to this.

Aside, now I have a little more time to work on this so hopefully we can have this merged soon ™️ 🤞

@github-actions github-actions bot added the modifies/api This PR adds API routes or modifies them label Sep 21, 2023
@AdamMajer
Copy link
Contributor Author

@6543 50c5488 should be reverted, it probably will break things

@6543
Copy link
Member

6543 commented Jan 19, 2024

@6543 50c5488 should be reverted, it probably will break things

well the insert yes ... did add a count check

as for the other one ... see #23894 (comment)

we either have to update it first or we dont

@6543
Copy link
Member

6543 commented Jan 19, 2024

ok I'll put my hands away again ;)

#23894 (comment)
Y/N?

@6543
Copy link
Member

6543 commented Jan 19, 2024

Well avter a 360° commit session we are back ... to start diff 😆

could you add at least a codecomment that the update statement only exist to catch strange edgecases if they exist ... ?

@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Jan 19, 2024
@AdamMajer
Copy link
Contributor Author

Thanks for all the work in helping this patch to finally land. Extra credits go to @lunny for all the help!

@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Jan 19, 2024
@6543 6543 merged commit d68a613 into go-gitea:main Jan 19, 2024
25 checks passed
@6543
Copy link
Member

6543 commented Jan 19, 2024

wow thanks for the hard work to all - we have initiall support ... now we can build ontop that e.g. for go-git ...

@denyskon
Copy link
Member

Great that this finally landed! Did you by any chance check the label in repo heeder on mobile? It could be broken as we display icons instead of labels there. If not, I'll check later...

@6543
Copy link
Member

6543 commented Jan 19, 2024

no i did not checked the mobile ui 😅 - that's then for a followup if it's broken :)

@AdamMajer
Copy link
Contributor Author

wow thanks for the hard work to all - we have initiall support ... now we can build ontop that e.g. for go-git ...

GoGit needs to implement new API for ability to support different hashes. Currently they only support sha256 if compiled with sha256 buildtag, and at same time this drops sha1 support. This is a no-go (pun kind of intended). Proper support needs to be implemented for their next major version since API changes are required. Then we can add that here, which should be easy. See: go-git/go-git#706

Any issues here with this patch, please open a new issue.

@go-gitea go-gitea locked as resolved and limited conversation to collaborators Jan 19, 2024
@6543 6543 added the type/changelog Adds the changelog for a new Gitea version label Jan 20, 2024
@AdamMajer AdamMajer deleted the sha256 branch February 19, 2024 10:06
techknowlogick pushed a commit that referenced this pull request Apr 24, 2024
DennisRasey pushed a commit to DennisRasey/forgejo that referenced this pull request Apr 30, 2024
Fix go-gitea/gitea#23894 (comment)

(cherry picked from commit 2ad9ef4984f0b68ef38241fd6b557d8427d851d8)

Conflicts:
	models/migrations/v1_16/v210.go
	models/migrations/v1_22/v286.go
	trivial conflicts because MSSQL is no longer supported
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. modifies/api This PR adds API routes or modifies them modifies/migrations modifies/translation size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. type/changelog Adds the changelog for a new Gitea version type/feature Completely new functionality. Can only be merged if feature freeze is not active.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

GIT sha256 support
8 participants