Make a detailed migration plan

[mocobeta]

It will take at least a few days and there will be some moratorium where GitHub issue is not lifted yet but a Jira snapshot is already taken. We need a detailed migration plan to avoid possible conflicts/confusion.

A draft plan would be (bold texts indicate things that all should know):

The plan was fixed as follows.

• [Full snapshot] Run the download script to take a snapshot of the whole Lucene Jira.
  • This should take ~8 hours
• Make Jira read-only to everyone other than my account.
• [Delta snapshot] Run the download script again to take a snapshot for updated issues since the full snapshot.
• Commit all attachments to lucene-jira-archive (this repository).
• Run the conversion script that generates GitHub importable data from the Jira dump.
  • This should take one or two hours.
• [First pass] Run the import script to initialize all issues and comments.
  • This should take ~24 hours (needs intervals between GitHub API calls)
  • This is done by an admin account https://github.com/asfimport
• Announce that the migration is started to the mail list.
  • New issues should be created on GitHub after that
• Run the issue number re-mapping script to preserve cross-issue links.
  • This should take ~12 hours (needs intervals between GitHub API calls)
• [Second pass] Run the update script to create (re-mapped) cross-issue links.
  • This should take ~10 hours (needs intervals between GitHub API calls)
  • This is done by the same admin account as importing.
• Manual checks and post-processing.
  • Set colors/descriptions to issue labels.
• Annonce that the migration is completed to the mail list.
  • GitHub issue is available in full at this point.
• Disable notifications to issues@ from Jira
• Add comments to each Jira issue that say "Moved to GitHub <link>".
• Recover migration errors manually
  • Manually recover Jira labels that are not migrated by the import script. (Skip errorcausing labels #84)
  • Manually recover migration errors.
    • Investigate import failure of LUCENE-1498 #14
    • Inlined patches don't migrate correctly #106

[dweiss]

bq. Add comments to each Jira issue that say "Moved to GitHub ".

I would love if these old URLs could simply redirect to github, unless a special token is passed ("noredirect") was used in the URL - this is what spring folks did.

[mocobeta]

I would love if these old URLs could simply redirect to github, unless a special token is passed ("noredirect") was used in the URL - this is what spring folks did.

While this would be nice to have, I don't think it'd be a must-have thing. Improvements like this can come later if there are people who are willing to work on it (with infra people?). I don't mean we should not include nice-to-have things in the first scope of our migration, just wanted to mention that my hands are full with dealing with our minimum requirements.

[dweiss]

Sure, I agree this can be postponed - I just wanted to signal that it'd be a nice thing to have. We wouldn't be able to do it anyway - I think it'd have to be a request to infra to set up such a redirect (and we'd need to produce a mapping file of old-new urls).

[mocobeta]

Just to make certain, I tried importing issues to an ASF repo (this repo) if we can do the whole steps ourselves.
I saw an error - still, we have to ask infra to run the long-running batches.

[mocobeta]

#13 confirmed that my account was able to update issues in an ASF repo without modifying the author. We can do step 7 (second pass - the most time-consuming and involving lots of REST API calls via internet) ourselves. "edited by mocobeta" was noted in the headers though.

[mikemccand]

Sure, I agree this can be postponed - I just wanted to signal that it'd be a nice thing to have. We wouldn't be able to do it anyway - I think it'd have to be a request to infra to set up such a redirect (and we'd need to produce a mapping file of old-new urls).

Let's at least open an issue for this (lower priority) future task. It would be great if we could work with Infra to eventually achieve this. But I agree it need not block the initial migration.

[mikemccand]

Could we separate step 2) into:

2a) Announce X days in advance that on Y date, Jira will go read-only and we will migrate to GitHub issues, taking estimated Z time to complete.
2b) On Y date, send email saying migration will start shortly, reminding that Jira will be read-only for the duration.
2c) Then, make Jira read-only ([this seems like a good hint](https://confluence.atlassian.com/jirakb/how-to-make-jira-issues-read-only-using-workflow-properties-1063568771.html)).

[mocobeta]

2a) Announce X days in advance that on Y date, Jira will go read-only and we will migrate to GitHub issues, taking estimated Z time to complete.
2b) On Y date, send email saying migration will start shortly, reminding that Jira will be read-only for the duration.
2c) Then, make Jira read-only (this seems like a good hint).

As for "making JIra read-only", can you please explicitly gain consensus among devs in dev@ list, as I wrote in #15?

[mocobeta]

I understand it may not fulfill everyone's request/hope though, I already have too much on my plate. Please feel free to work on further improvements on Jira side, if you think it should be done for the move to GitHub (making Jira read-only, redirecting to GitHub from Jira, and so on).

I really don't want to say this but - just to be clear, I'm working on this migration project as an individual without support/returns from any organizations.

[mikemccand]

For step 9, maybe we can ask/plan for volunteers in the dev community to help check for anything that went wrong during the import process. We of course should test as much as possible in this development phase (thank you @mocobeta!) before we migrate as well.

[mikemccand]

I would love if these old URLs could simply redirect to github, unless a special token is passed ("noredirect") was used in the URL - this is what spring folks did.

I opened #28 for this. Thanks @dweiss!

[mikemccand]

I really don't want to say this but - just to be clear, I'm working on this migration project as an individual without support/returns from any organizations.

I totally understand @mocobeta and I am in absolute awe of the energy you've put into this. I'm sorry I have failed to make enough time to help. I know it is an insanely difficult project. I'm sorry if I am causing more stress! We don't have to rush this migration. Please don't burn out :(

[mikemccand]

As for "making JIra read-only", can you please explicitly gain consensus among devs in dev@ list, as I wrote in #15?

I opened #29 to make progress on this.

[mocobeta]

For step 9, maybe we can ask/plan for volunteers in the dev community to help check for anything that went wrong during the import process. We of course should test as much as possible in this development phase (thank you @mocobeta!) before we migrate as well.

Thanks. Instead of manually checking after the actual migration I'd ask the dev community to check if there are major conversion bugs when I finish the final test migration. For small/minor errors, committers can update/fix any issues/comments anytime, after migration.

We don't have to rush this migration.

While there is no harry for Lucene TLP, I cannot prolong this project for a long period due to my situation. Sorry but let me set the timeline someday this August (or September, at the latest). If this takes more time, I think I will have to leave this in the middle hoping someone takes over this.

[mocobeta]

Once the migration is started, issues opened in Jira have to be manually migrated to GitHub by the authors afterward and it'd be bothersome.

I wanted to add some texts that say,

We are switching from Jira to GitHub issues, and data migration is now in progress.
Although you can still open a Jira issue, you may want to wait until the migration is finished
and open a GitHub issue after that, if you are not in a hurry.
Migration will be completed within a few days.

to the Jira issue template (wording could be refined).

But it looks like I don't have permission to browse/edit the issue templates... Could someone who is able to edit the issue template help me with it?

Thank you.

[mocobeta]

As a note, here is the INFRA issue.
https://issues.apache.org/jira/browse/INFRA-23563

[mocobeta]

I talked about the date with infra.
The migration will start on Monday, August 22th at 8:00 UTC.

Note: infra gave me an API token with admin access. I'll do the whole migration by using the token.

[mocobeta]

All tasks including necessary post-processing/cleanup have been completed. Eventually it took around one week.