Improve migrations to support migrating milestones/labels/issues/comments/pullrequests #6290

lunny · 2019-03-09T14:01:52Z

This PR will improve the migrations process. It will define an interface to migrate data from external git service and at first implement from github to gitea. Below is the TODO list:

Define the migration interface
Migrate labels from github
Migrate milestones from github
Migrate issues and comments from github
Migrate pull requests and comments from github: remote deleted branch will made commits lost.
Migrate releases from github
Tests for download from github
Tests for upload to gitea
UI options to check wiki, milestones, labels, issues(with comments), prs(with comments), releases

Below will be ignored

Migrate issues or PRs' user from github: It requires we creates all the users automatically but cannot determine how to match the users between two sites since cannot retrieve users' email.
Migrate issues or PRs' reactions from github: reactions on gitea requires all the users IDs which cannot retrieves.

The new migration UI looks like:

Next steps after this merged

Implement GiteaDownloadV1 so that repositories could be transfer cross gitea instances.
Implement GitlabDownload
Implement BitBucketDownload
Implement GogsDownloadV1

codecov-io · 2019-03-16T04:22:38Z

Codecov Report

❗ No coverage uploaded for pull request base (master@1c7c739). Click here to learn what that means.
The diff coverage is 39.23%.

@@            Coverage Diff            @@
##             master    #6290   +/-   ##
=========================================
  Coverage          ?   41.13%           
=========================================
  Files             ?      432           
  Lines             ?    59504           
  Branches          ?        0           
=========================================
  Hits              ?    24475           
  Misses            ?    31817           
  Partials          ?     3212

Impacted Files	Coverage Δ
modules/auth/repo_form.go	`41.73% <ø> (ø)`
models/migrate.go	`0% <0%> (ø)`
modules/migrations/base/pullrequest.go	`0% <0%> (ø)`
modules/migrations/error.go	`0% <0%> (ø)`
modules/migrations/migrate.go	`23.88% <23.88%> (ø)`
routers/repo/repo.go	`22.96% <38.09%> (ø)`
routers/api/v1/repo/repo.go	`55.59% <44.23%> (ø)`
modules/migrations/git.go	`52% <52%> (ø)`
models/repo.go	`47.82% <64.28%> (ø)`
modules/migrations/gitea.go	`8.3% <8.3%> (ø)`
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1c7c739...5436028. Read the comment docs.

lunny · 2019-04-12T14:48:50Z

I think this is ready for review.

kolaente

Also, what happens if the user creates the repo to be a mirror? Will issues/prs/wiki etc still be synced or only on the initial creation of the repo?

kolaente · 2019-04-12T16:27:28Z

models/migrate.go

+import "github.com/go-xorm/xorm"
+
+// InsertIssue insert one issue to database
+func InsertIssue(issue *Issue, labelIDs []int64) error {


How is this different to createIssue?

This will not trigger notifications.

But then I'd prefer to have createIssue call this function and trigger notifications to avoid having duplicated code.

I want to do that, but it's so different between theses two functions I would like let it duplicated currently. Maybe send a refactor PR after this PR.

Fair enough, sounds good.

kolaente · 2019-04-12T16:45:36Z

models/repo.go

@@ -877,6 +877,7 @@ type MigrateRepoOptions struct {
 	IsPrivate   bool
 	IsMirror    bool
 	RemoteAddr  string
+	NoWiki      bool


Please call this UsesWiki or just Wiki or something like that, then check if opts.Wiki instead of if !opts.NoWiki to avoid double negation.

kolaente · 2019-04-12T16:46:45Z

models/repo.go

 			if err := os.RemoveAll(wikiPath); err != nil {
 				return repo, fmt.Errorf("Failed to remove %s: %v", wikiPath, err)
 			}
+
+			if err = git.Clone(wikiRemotePath, wikiPath, git.CloneRepoOptions{
+				Mirror:  true,


Why is this always a mirror?

I think the Mirror means a full clone but not a mirror repository on Gitea.

These codes just moved from above because I added opts.Wiki.

I think this is probably ok and we certainly want at least part of what it's doing regarding copying the remote tracking branches. We should check what happens when we remove the upstream at the end of the migration and check that the refspec configuration is cleaned up though.

kolaente · 2019-04-12T16:50:08Z

modules/migrations/gitea.go

+)
+
+// GiteaLocalUploader implements an Uploader to gitea sites
+type GiteaLocalUploader struct {


IMHO since this is already on the local Gitea instance, the word Local is double here.

The name should also make it more clear this is a migrator from github (is it?) and not just something "to move local git repos around".

@kolaente there are uploaders and downloaders here. GiteaLocalUploader will save data via gitea's local operations. Currently we have GithubV3Downloader, in future, we could implement GitLab/Gitea/Gogs/... downloaders. Furthermore we could have GithubV3Uploader so that we can migrate gitea's data to github.

So the flow is GithubV3Uploader -> GiteaLocalUploader?

Current flow is GithubV3Downloader - > GiteaLocalUploader. Always from a downloader interface implementation to an uploader interface implementation.

kolaente · 2019-04-12T16:53:09Z

modules/migrations/gitea.go

+}
+
+// CreateRepo creates a repository
+func (g *GiteaLocalUploader) CreateRepo(repo *base.Repository, includeWiki bool) error {


How is this (and a lot of other functions in this file) different from the implementation in models/migrate.go?

This file will invoke models/migrate.go when migrate git/wiki data.

kolaente · 2019-04-12T16:55:49Z

modules/migrations/github.go

+				labels = append(labels, convertGithubLabel(l))
+			}
+			// FIXME: This API missing reactions, we may need another extra request to get reactions
+			/*var reactions *Reactions


Please remove the commented code.

kolaente · 2019-04-12T16:58:05Z

modules/migrations/migrate.go

+
+	if opts.PullRequests {
+		log.Trace("migrating pull requests and comments")
+		prs, err := downloader.GetPullRequests(0, 1000000)


What does this magic number do? I'd guess it's a limit of some kind, but what if there are > 1000000? Will it fail? Will it just not downlod anything > 1000000?

kolaente · 2019-04-12T16:58:44Z

public/js/index.js

@@ -979,6 +979,23 @@ function initRepository() {
    }
 }

+var toggleMigrations = function(){
+    console.log("toggleMigrations")


Please remove this console.log

kolaente · 2019-04-12T17:00:02Z

public/js/index.js

@@ -979,6 +979,23 @@ function initRepository() {
    }
 }

+var toggleMigrations = function(){
+    console.log("toggleMigrations")
+    if (!$('#mirror').is(":checked") && $('#auth_username').val().length > 0 && ($('#clone_addr').val().startsWith("https://github.com") || $('#clone_addr').val().startsWith("http://github.com"))) {


Won't this only be triggered when a user loads the page and the clone url is already in the clone field? IMHO this should listen on input.

kolaente · 2019-04-12T17:00:45Z

routers/repo/repo.go

@@ -263,11 +290,12 @@ func MigratePost(ctx *context.Context, form auth.MigrateRepoForm) {
 	// remoteAddr may contain credentials, so we sanitize it
 	err = util.URLSanitizedError(err, remoteAddr)

-	if repo != nil {
+	// TODO: remove
+	/*if repo != nil {


Please remove this commented code.

ashimokawa · 2019-04-12T18:42:09Z

@lunny
It seems to work, nice work. 👍

One minor issue I noticed is that wrong github passwords gives just a big error 500 screen, that should not happen.

lunny · 2019-04-13T02:45:11Z

@kolaente When you check mirror on the UI, all the migration items will be hidden. That means issues/milestones/... will not be mirrored.

lunny · 2019-04-13T02:45:37Z

@ashimokawa yes, I will fix that.

lunny · 2019-04-13T13:27:44Z

@ashimokawa auth error handled.

ashimokawa · 2019-04-13T16:24:56Z

@lunny
Great progress!

What about introducing a new database column for comments "original_author" or something where "github/ashimokawa" is stored for example?

Then if it is confirmed later that ashimokawa signed up on the gitea instance where the repo was moved to, the admin could reassign the comments to the real local user ashimokawa.
This would also eliminate the need to store the original author as text inside the issue comment text itself, and it would be possible to put some nice "imported from github" widget above or below the comment instead, which is clickable and leads the the github user page as long as it has not been reassigned by the local admin to the local user.

lunny · 2019-04-14T06:03:50Z

@ashimokawa I will not do that change since it's not possible to let all repository related users register in this Gitea instance and you cannot mapping them in fact. It will not result in difficult on maintain all the issues and pull requests even if the posters are not the original authors.

ashimokawa · 2019-04-14T08:18:26Z

@lunny
I get your point, still I think my proposal makes sense - not to propery assign all of the comments, but it would be great for a core team of a project, which moved to its own gitea instance where the admin knows who is who to be able to reassign authors properly. That is what my proposal was for.

lunny · 2019-04-14T08:49:54Z

@ashimokawa that could be next step for a private repository because I think there are many work to do. Currently I'll focus on public repositories.

…releases

lunny · 2019-05-05T13:43:44Z

@jolheiser done.

techknowlogick

Looks good, my one concern is that with a large project migration API calls might hit rate limit quota, but I think adding in a GitHub token for auth to increase API rate limit can be done in another PR.

lunny · 2019-05-07T01:11:30Z

@techknowlogick that's already supported on this PR. you could put your token on username inputbox and then that's it.

techknowlogick · 2019-05-07T01:13:09Z

@lunny oh that's perfect! I missed that. Thanks for telling me :)

strk · 2019-08-07T12:30:01Z

The token-on-username thing is not documented anywhere, or did I miss it ?

lunny · 2019-08-07T14:00:41Z

It is mentioned on the tip above the input boxes.

strk · 2019-08-07T14:06:38Z

Also, from gitea.com when selecting "This repo will be a mirror" the options to also sync issues/labels/releases etc. disappears. Is that not supported ?

strk · 2019-08-07T14:09:21Z

The migration target page now became a 500 error on https://gitea.com/QGIS/QGIS ...

lunny · 2019-08-07T14:41:21Z

mirroring is not supported now. When it's migrating, it maybe failed on UI but it may still running on backend. That's what my new PR wants to resolve. See #6200

lunny added the type/feature Completely new functionality. Can only be merged if feature freeze is not active. label Mar 9, 2019

lunny changed the title ~~WIP: add migrations~~ WIP: improve migrations to support migrating milestones/labels/issues/reactions/comments/pullrequests Mar 9, 2019

lunny force-pushed the lunny/migrations branch from 4633d92 to ed8a614 Compare March 16, 2019 03:16

lunny mentioned this pull request Mar 16, 2019

Gitea hosted Gitea #1029

Open

20 tasks

lunny force-pushed the lunny/migrations branch from c9e58dd to 37e0757 Compare March 16, 2019 04:10

GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Mar 16, 2019

lunny added this to the 1.9.0 milestone Mar 16, 2019

lunny force-pushed the lunny/migrations branch 4 times, most recently from cfa1233 to a7c2fe0 Compare March 24, 2019 08:21

lunny force-pushed the lunny/migrations branch 2 times, most recently from c5bc0ed to 4daa5c7 Compare March 30, 2019 05:05

lunny force-pushed the lunny/migrations branch 2 times, most recently from 7ceb4de to c0878f8 Compare April 12, 2019 08:35

lunny changed the title ~~WIP: improve migrations to support migrating milestones/labels/issues/reactions/comments/pullrequests~~ Improve migrations to support migrating milestones/labels/issues/reactions/comments/pullrequests Apr 12, 2019

kolaente suggested changes Apr 12, 2019

View reviewed changes

lunny force-pushed the lunny/migrations branch from ca0777c to 71881e6 Compare April 13, 2019 04:46

lunny force-pushed the lunny/migrations branch from ab1a9d3 to 148226a Compare April 14, 2019 08:53

lunny added the type/changelog Adds the changelog for a new Gitea version label Apr 14, 2019

lunny added 7 commits May 5, 2019 20:19

migrate API will also support migrate milestones/labels/issues/pulls/…

3997e95

…releases

fix tests and remove unused codes

7040908

add DownloaderFactory and docs about how to create a new Downloader

9dd95f5

fix misspell

f14a09f

fix migration docs

d27bffc

Add hints about migrate options on migration page

6d06ae4

fix tests

d798b0d

lunny force-pushed the lunny/migrations branch from bc33856 to d798b0d Compare May 5, 2019 12:20

Merge branch 'master' into lunny/migrations

5436028

techknowlogick approved these changes May 6, 2019

View reviewed changes

GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels May 6, 2019

lunny merged commit 08069dc into go-gitea:master May 7, 2019

lunny deleted the lunny/migrations branch May 7, 2019 01:14

This was referenced May 23, 2019

Migration Tweaks #6260

Merged

No error message when trying to migrate repository with incorrect name #6254

Closed

Eloston mentioned this pull request Aug 9, 2019

leave Github ungoogled-software/ungoogled-chromium#795

Closed

6543 mentioned this pull request Oct 25, 2019

Migration Summary #8689

Open

24 tasks

alexanderadam mentioned this pull request Jun 10, 2020

GitHub Migrations for Enterprise GitHub servers #11815

Closed

TheAssassin mentioned this pull request Oct 24, 2020

Set up official mirrors? TeamNewPipe/NewPipe#4619

Open

go-gitea locked and limited conversation to collaborators Nov 24, 2020

delvh removed the type/changelog Adds the changelog for a new Gitea version label Oct 7, 2023

Improve migrations to support migrating milestones/labels/issues/comments/pullrequests #6290

Improve migrations to support migrating milestones/labels/issues/comments/pullrequests #6290

Conversation

lunny commented Mar 9, 2019 • edited

codecov-io commented Mar 16, 2019 • edited

Codecov Report

lunny commented Apr 12, 2019

kolaente left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lunny Apr 13, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashimokawa commented Apr 12, 2019

lunny commented Apr 13, 2019

lunny commented Apr 13, 2019

lunny commented Apr 13, 2019

ashimokawa commented Apr 13, 2019

lunny commented Apr 14, 2019

ashimokawa commented Apr 14, 2019

lunny commented Apr 14, 2019

lunny commented May 5, 2019

techknowlogick left a comment

Choose a reason for hiding this comment

lunny commented May 7, 2019

techknowlogick commented May 7, 2019

strk commented Aug 7, 2019

lunny commented Aug 7, 2019

strk commented Aug 7, 2019

strk commented Aug 7, 2019

lunny commented Aug 7, 2019

lunny commented Mar 9, 2019 •

edited

codecov-io commented Mar 16, 2019 •

edited

lunny Apr 13, 2019 •

edited