Backup Issues/PRs for projects hosted in https://github.com/sagemath/

[mkoeppe]

(from https://groups.google.com/g/sage-devel/c/hX6ojxlNwOU/m/r88PtBMTBQAJ)

We set up such backups for the projects that are already hosted at https://github.com/sagemath/:
In particular:

• https://github.com/sagemath/cypari2
• https://github.com/sagemath/sagetex
• https://github.com/sagemath/pplpy
• https://github.com/sagemath/cysignals
• https://github.com/sagemath/memory_allocator

Available technologies:

• https://github-backup.branchable.com/ (mmezzarobba reports: fairly complete but getting timeouts with it),
• dimpase suggests: set up a GitHub actions to produce jsons for created/updated issues and PRs
• gh-tobiasdiez suggests: one can download a "migration archive" from github should include all data
• gh-tobiasdiez suggests: use one of the third-party solutions that keep backups per repo
• https://github.com/josegonzalez/python-github-backup
  • used for example by Cython: https://github.com/cython/cython-issues

We should have the backup running before making the proposed migration of sage development from Trac to GH (#30363). We should also refrain from using features of github not supported by our backup tool.

CC: @dimpase @tobiasdiez @kliem @mezzarobba @videlec @jhpalmieri @nbruin @sagetrac-tmonteil

Component: website/wiki

Issue created by migration from https://trac.sagemath.org/ticket/34624

[mkoeppe]

comment:1

​https://github.com/josegonzalez/python-github-backup looks fine to me and would be trivial to use in a cron job.

Would a friendly academic hosting volunteer please step up and take care of this?

[mezzarobba]

comment:2

I just played a bit with ​https://github.com/josegonzalez/python-github-backup and it stops after failing to download hooks for the first repository. After a quick and dirty patch

diff --git a/github_backup/github_backup.py b/github_backup/github_backup.py
index 873a40c..e2dcd6e 100644
--- a/github_backup/github_backup.py
+++ b/github_backup/github_backup.py
@@ -1012,7 +1012,7 @@ def backup_hooks(args, repo_cwd, repository, repos_template):
                      template,
                      output_file,
                      hook_cwd)
-    except SystemExit:
+    except (SystemExit, Exception):
         log_info("Unable to read hooks, skipping")

it works better, but a look at the issues list suggests it still does not backup everything. In am concerned in particular about

• All pull request comments not backed up josegonzalez/python-github-backup#150
• Incremental backup skips some new events josegonzalez/python-github-backup#168
• A way to include the actual commits of backupped PRs? josegonzalez/python-github-backup#193
  So: not a bad staring point, but we will have to patch it, and to be safe we should regularly download migration archives as well.

[mezzarobba]

comment:3

Regardubg joeyh's github-backup, I missed that: https://joeyh.name/blog/entry/Withrawing_github-backup/

[mezzarobba]

comment:4

I now have disk space on a system on which I could set up incremental backups of the github organization using python-github-backup. However, I can provide absolutely no guarantee that the backups will be reliable or complete, nor can I easily give access to that system to other people. (I suppose I could serve the backups by http if necessary, but that's about the best I can do.)

So it would be nice if someone else would independently set up backups using the other method discussed above (the github migration API). Note that one apparently needs “owner” permissions on the organization to use the migration API.

[mkoeppe]

comment:5

Replying to Marc Mezzarobba:

it would be nice if someone else would independently set up backups using the other method discussed above (the github migration API). Note that one apparently needs “owner” permissions on the organization to use the migration API.

I'll do this in sagemath/trac-to-github#82