Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve migrations to support migrating milestones/labels/issues/comments/pullrequests #6290

Merged
merged 33 commits into from May 7, 2019

Conversation

@lunny
Copy link
Member

commented Mar 9, 2019

This PR will improve the migrations process. It will define an interface to migrate data from external git service and at first implement from github to gitea. Below is the TODO list:

  • Define the migration interface
  • Migrate labels from github
  • Migrate milestones from github
  • Migrate issues and comments from github
  • Migrate pull requests and comments from github: remote deleted branch will made commits lost.
  • Migrate releases from github
  • Tests for download from github
  • Tests for upload to gitea
  • UI options to check wiki, milestones, labels, issues(with comments), prs(with comments), releases

Below will be ignored

  • Migrate issues or PRs' user from github: It requires we creates all the users automatically but cannot determine how to match the users between two sites since cannot retrieve users' email.
  • Migrate issues or PRs' reactions from github: reactions on gitea requires all the users IDs which cannot retrieves.

The new migration UI looks like:

image

Next steps after this merged

  • Implement GiteaDownloadV1 so that repositories could be transfer cross gitea instances.
  • Implement GitlabDownload
  • Implement BitBucketDownload
  • Implement GogsDownloadV1

@lunny lunny added the kind/feature label Mar 9, 2019

@lunny lunny changed the title WIP: add migrations WIP: improve migrations to support migrating milestones/labels/issues/reactions/comments/pullrequests Mar 9, 2019

@lunny lunny force-pushed the lunny:lunny/migrations branch from 4633d92 to ed8a614 Mar 16, 2019

@lunny lunny referenced this pull request Mar 16, 2019

Open

Gitea hosted Gitea #1029

9 of 10 tasks complete

@lunny lunny force-pushed the lunny:lunny/migrations branch from c9e58dd to 37e0757 Mar 16, 2019

@codecov-io

This comment has been minimized.

Copy link

commented Mar 16, 2019

Codecov Report

❗️ No coverage uploaded for pull request base (master@1c7c739). Click here to learn what that means.
The diff coverage is 39.23%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master    #6290   +/-   ##
=========================================
  Coverage          ?   41.13%           
=========================================
  Files             ?      432           
  Lines             ?    59504           
  Branches          ?        0           
=========================================
  Hits              ?    24475           
  Misses            ?    31817           
  Partials          ?     3212
Impacted Files Coverage Δ
modules/auth/repo_form.go 41.73% <ø> (ø)
models/migrate.go 0% <0%> (ø)
modules/migrations/base/pullrequest.go 0% <0%> (ø)
modules/migrations/error.go 0% <0%> (ø)
modules/migrations/migrate.go 23.88% <23.88%> (ø)
routers/repo/repo.go 22.96% <38.09%> (ø)
routers/api/v1/repo/repo.go 55.59% <44.23%> (ø)
modules/migrations/git.go 52% <52%> (ø)
models/repo.go 47.82% <64.28%> (ø)
modules/migrations/gitea.go 8.3% <8.3%> (ø)
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1c7c739...5436028. Read the comment docs.

@lunny lunny added this to the 1.9.0 milestone Mar 16, 2019

@lunny lunny force-pushed the lunny:lunny/migrations branch 4 times, most recently from cfa1233 to a7c2fe0 Mar 18, 2019

@lunny lunny force-pushed the lunny:lunny/migrations branch 2 times, most recently from c5bc0ed to 4daa5c7 Mar 29, 2019

@lunny lunny force-pushed the lunny:lunny/migrations branch 2 times, most recently from 7ceb4de to c0878f8 Apr 12, 2019

@lunny lunny changed the title WIP: improve migrations to support migrating milestones/labels/issues/reactions/comments/pullrequests Improve migrations to support migrating milestones/labels/issues/reactions/comments/pullrequests Apr 12, 2019

@lunny

This comment has been minimized.

Copy link
Member Author

commented Apr 12, 2019

I think this is ready for review.

@kolaente
Copy link
Member

left a comment

Also, what happens if the user creates the repo to be a mirror? Will issues/prs/wiki etc still be synced or only on the initial creation of the repo?

import "github.com/go-xorm/xorm"

// InsertIssue insert one issue to database
func InsertIssue(issue *Issue, labelIDs []int64) error {

This comment has been minimized.

Copy link
@kolaente

kolaente Apr 12, 2019

Member

How is this different to createIssue?

This comment has been minimized.

Copy link
@lunny

lunny Apr 13, 2019

Author Member

This will not trigger notifications.

This comment has been minimized.

Copy link
@kolaente

kolaente Apr 14, 2019

Member

But then I'd prefer to have createIssue call this function and trigger notifications to avoid having duplicated code.

This comment has been minimized.

Copy link
@lunny

lunny Apr 15, 2019

Author Member

I want to do that, but it's so different between theses two functions I would like let it duplicated currently. Maybe send a refactor PR after this PR.

This comment has been minimized.

Copy link
@kolaente

kolaente Apr 15, 2019

Member

Fair enough, sounds good.

@@ -877,6 +877,7 @@ type MigrateRepoOptions struct {
IsPrivate bool
IsMirror bool
RemoteAddr string
NoWiki bool

This comment has been minimized.

Copy link
@kolaente

kolaente Apr 12, 2019

Member

Please call this UsesWiki or just Wiki or something like that, then check if opts.Wiki instead of if !opts.NoWiki to avoid double negation.

This comment has been minimized.

Copy link
@lunny

lunny Apr 14, 2019

Author Member

done.

if err := os.RemoveAll(wikiPath); err != nil {
return repo, fmt.Errorf("Failed to remove %s: %v", wikiPath, err)
}

if err = git.Clone(wikiRemotePath, wikiPath, git.CloneRepoOptions{
Mirror: true,

This comment has been minimized.

Copy link
@kolaente

kolaente Apr 12, 2019

Member

Why is this always a mirror?

This comment has been minimized.

Copy link
@lunny

lunny Apr 13, 2019

Author Member

I think the Mirror means a full clone but not a mirror repository on Gitea.

This comment has been minimized.

Copy link
@lunny

lunny Apr 14, 2019

Author Member

These codes just moved from above because I added opts.Wiki.

This comment has been minimized.

Copy link
@zeripath

zeripath Apr 14, 2019

Contributor

I think this is probably ok and we certainly want at least part of what it's doing regarding copying the remote tracking branches. We should check what happens when we remove the upstream at the end of the migration and check that the refspec configuration is cleaned up though.

)

// GiteaLocalUploader implements an Uploader to gitea sites
type GiteaLocalUploader struct {

This comment has been minimized.

Copy link
@kolaente

kolaente Apr 12, 2019

Member

IMHO since this is already on the local Gitea instance, the word Local is double here.

The name should also make it more clear this is a migrator from github (is it?) and not just something "to move local git repos around".

This comment has been minimized.

Copy link
@lunny

lunny Apr 13, 2019

Author Member

@kolaente there are uploaders and downloaders here. GiteaLocalUploader will save data via gitea's local operations. Currently we have GithubV3Downloader, in future, we could implement GitLab/Gitea/Gogs/... downloaders. Furthermore we could have GithubV3Uploader so that we can migrate gitea's data to github.

This comment has been minimized.

Copy link
@kolaente

kolaente Apr 14, 2019

Member

So the flow is GithubV3Uploader -> GiteaLocalUploader?

This comment has been minimized.

Copy link
@lunny

lunny Apr 14, 2019

Author Member

Current flow is GithubV3Downloader - > GiteaLocalUploader. Always from a downloader interface implementation to an uploader interface implementation.

}

// CreateRepo creates a repository
func (g *GiteaLocalUploader) CreateRepo(repo *base.Repository, includeWiki bool) error {

This comment has been minimized.

Copy link
@kolaente

kolaente Apr 12, 2019

Member

How is this (and a lot of other functions in this file) different from the implementation in models/migrate.go?

This comment has been minimized.

Copy link
@lunny

lunny Apr 13, 2019

Author Member

This file will invoke models/migrate.go when migrate git/wiki data.

labels = append(labels, convertGithubLabel(l))
}
// FIXME: This API missing reactions, we may need another extra request to get reactions
/*var reactions *Reactions

This comment has been minimized.

Copy link
@kolaente

kolaente Apr 12, 2019

Member

Please remove the commented code.

This comment has been minimized.

Copy link
@lunny

lunny Apr 13, 2019

Author Member

done.


if opts.PullRequests {
log.Trace("migrating pull requests and comments")
prs, err := downloader.GetPullRequests(0, 1000000)

This comment has been minimized.

Copy link
@kolaente

kolaente Apr 12, 2019

Member

What does this magic number do? I'd guess it's a limit of some kind, but what if there are > 1000000? Will it fail? Will it just not downlod anything > 1000000?

This comment has been minimized.

Copy link
@lunny

lunny Apr 13, 2019

Author Member

resolved.

@@ -979,6 +979,23 @@ function initRepository() {
}
}

var toggleMigrations = function(){
console.log("toggleMigrations")

This comment has been minimized.

Copy link
@kolaente

kolaente Apr 12, 2019

Member

Please remove this console.log

This comment has been minimized.

Copy link
@lunny

lunny Apr 13, 2019

Author Member

done.

@@ -979,6 +979,23 @@ function initRepository() {
}
}

var toggleMigrations = function(){
console.log("toggleMigrations")
if (!$('#mirror').is(":checked") && $('#auth_username').val().length > 0 && ($('#clone_addr').val().startsWith("https://github.com") || $('#clone_addr').val().startsWith("http://github.com"))) {

This comment has been minimized.

Copy link
@kolaente

kolaente Apr 12, 2019

Member

Won't this only be triggered when a user loads the page and the clone url is already in the clone field? IMHO this should listen on input.

@@ -263,11 +290,12 @@ func MigratePost(ctx *context.Context, form auth.MigrateRepoForm) {
// remoteAddr may contain credentials, so we sanitize it
err = util.URLSanitizedError(err, remoteAddr)

if repo != nil {
// TODO: remove
/*if repo != nil {

This comment has been minimized.

Copy link
@kolaente

kolaente Apr 12, 2019

Member

Please remove this commented code.

This comment has been minimized.

Copy link
@lunny

lunny Apr 13, 2019

Author Member

done.

@ashimokawa

This comment has been minimized.

Copy link
Contributor

commented Apr 12, 2019

@lunny
It seems to work, nice work. 👍

One minor issue I noticed is that wrong github passwords gives just a big error 500 screen, that should not happen.

@lunny

This comment has been minimized.

Copy link
Member Author

commented Apr 13, 2019

@kolaente When you check mirror on the UI, all the migration items will be hidden. That means issues/milestones/... will not be mirrored.

@lunny

This comment has been minimized.

Copy link
Member Author

commented Apr 13, 2019

@ashimokawa yes, I will fix that.

@lunny lunny force-pushed the lunny:lunny/migrations branch from ca0777c to 71881e6 Apr 13, 2019

@lunny

This comment has been minimized.

Copy link
Member Author

commented Apr 13, 2019

@ashimokawa auth error handled.

@ashimokawa

This comment has been minimized.

Copy link
Contributor

commented Apr 13, 2019

@lunny
Great progress!

What about introducing a new database column for comments "original_author" or something where "github/ashimokawa" is stored for example?

Then if it is confirmed later that ashimokawa signed up on the gitea instance where the repo was moved to, the admin could reassign the comments to the real local user ashimokawa.
This would also eliminate the need to store the original author as text inside the issue comment text itself, and it would be possible to put some nice "imported from github" widget above or below the comment instead, which is clickable and leads the the github user page as long as it has not been reassigned by the local admin to the local user.

@lunny

This comment has been minimized.

Copy link
Member Author

commented Apr 14, 2019

@ashimokawa I will not do that change since it's not possible to let all repository related users register in this Gitea instance and you cannot mapping them in fact. It will not result in difficult on maintain all the issues and pull requests even if the posters are not the original authors.

@ashimokawa

This comment has been minimized.

Copy link
Contributor

commented Apr 14, 2019

@lunny
I get your point, still I think my proposal makes sense - not to propery assign all of the comments, but it would be great for a core team of a project, which moved to its own gitea instance where the admin knows who is who to be able to reassign authors properly. That is what my proposal was for.

@lunny

This comment has been minimized.

Copy link
Member Author

commented Apr 14, 2019

@ashimokawa that could be next step for a private repository because I think there are many work to do. Currently I'll focus on public repositories.

@lunny lunny force-pushed the lunny:lunny/migrations branch from ab1a9d3 to 148226a Apr 14, 2019

@lunny lunny added the changelog label Apr 14, 2019

lunny added some commits Mar 29, 2019

@lunny lunny force-pushed the lunny:lunny/migrations branch from bc33856 to d798b0d May 5, 2019

@lunny

This comment has been minimized.

Copy link
Member Author

commented May 5, 2019

@jolheiser done.

@techknowlogick
Copy link
Member

left a comment

Looks good, my one concern is that with a large project migration API calls might hit rate limit quota, but I think adding in a GitHub token for auth to increase API rate limit can be done in another PR.

@GiteaBot GiteaBot added lgtm/done and removed lgtm/need 1 labels May 6, 2019

@lunny

This comment has been minimized.

Copy link
Member Author

commented May 7, 2019

@techknowlogick that's already supported on this PR. you could put your token on username inputbox and then that's it.

@lunny lunny merged commit 08069dc into go-gitea:master May 7, 2019

2 checks passed

approvals/lgtm this commit looks good
continuous-integration/drone/pr Build is passing
Details
@techknowlogick

This comment has been minimized.

Copy link
Member

commented May 7, 2019

@lunny oh that's perfect! I missed that. Thanks for telling me :)

@lunny lunny deleted the lunny:lunny/migrations branch May 7, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.