Tracking issue for rewriting the history of `main` (git-fsck fails)

[mkroening]

Issue

Running

git clone https://github.com/hermit-os/kernel.git
cd kernel
git fsck

results in

Checking ref database: 100% (1/1), done.
Checking object directories: 100% (256/256), done.
error in blob 41e8993ab02e76aac1adb2680e00ae026eef62a1: gitmodulesUrl: disallowed submodule url: https://github.com:RWTH-OS/linux.git
error in blob 41e8993ab02e76aac1adb2680e00ae026eef62a1: gitmodulesUrl: disallowed submodule url: https://github.com:RWTH-OS/gcc.git
error in blob 41e8993ab02e76aac1adb2680e00ae026eef62a1: gitmodulesUrl: disallowed submodule url: https://github.com:RWTH-OS/binutils.git
error in blob 41e8993ab02e76aac1adb2680e00ae026eef62a1: gitmodulesUrl: disallowed submodule url: https://github.com:RWTH-OS/pthread-embeded.git
error in blob 41e8993ab02e76aac1adb2680e00ae026eef62a1: gitmodulesUrl: disallowed submodule url: https://github.com:RWTH-OS/newlib.git
error in blob 41e8993ab02e76aac1adb2680e00ae026eef62a1: gitmodulesUrl: disallowed submodule url: https://github.com:RWTH-OS/LwIP.git
error in blob 41e8993ab02e76aac1adb2680e00ae026eef62a1: gitmodulesUrl: disallowed submodule url: https://github.com:RWTH-OS/libomp_oss.git
Checking objects: 100% (57844/57844), done.

with exit code 4.

This happens since git 2.44.0, released Feb 2024, more specifically since git-scm/git@8430b43.

On the Hermit side, this is caused by c1c6a1a in 2016. It was resolved four minutes later in db6e997.

On its own, this is fine. You can still clone the repo locally and on GitHub. However, you cannot create a new repo any more on:

• git.nrw
• git.rwth-aachen.de
• git-ce.rwth-aachen.de
• gitlab.com
• Any GitLab instance that uses GitLab 17.0 or newer.
• codeberg.org

This is quite unfortunate, and we would like to make this repo work on those instances.

There are two troubleshooting pages for GitLab:

• Resolving sync failure: gitmodulesUrl: disallowed submodule url – GitLab, Inc.
• Troubleshooting - Direct Transfer Migration Fails with "gitmodulesUrl: disallowed submodule url" Error – GitLab, Inc.

Options

The least invasive option would be to just disable the check by setting fsck.gitmodulesUrl to warn or ignore. This works great locally, but unfortunately, there is no way to configure this in a way that automatically gets applied when creating a new clone for this specific repo. There are ways to make this work by changing the global git config of the GitLab instance as an administrator, but it is unlikely that we can persuade all relevant instances to make this change.

Instead, the only option I can see is to do a ten-year history rewrite. :(

The flag day will be painful, but instead of just force-pushing main and causing mayhem, we might be able to do this more smoothly:

1. Create a main-fsck branch, which does not contain the faulty submodule state.
2. Rename main to main-pre-fsck. GitHub has decent functionality for dealing with renaming the main branch, as far as I know.
3. Make main-fsck the new default branch. GitHub also has decent functionality for this.
4. Rename main-fsck to main.

There should be at least six weeks between each of the steps to allow local clones of active contributors to adjust and feedback to arrive.

Rewriting history

Rewriting history is done best via git filter-repo. Since we are forced to rewrite history, we might as well purge LFS from the new branch's history, which makes pushing to GitLab even easier. Additionally, we can go ahead and remove a few binary files and large, generated files. They are described in more detail in the next section.

Warning

Always start rewriting from a fresh repo, if you want to try this yourself.

remove-from-history.txt:

# LFS files
img/demo.gif
img/hermitcore_logo.png

# Binaries
config/bzImage
config/initrd.cpio
hermit/usr/examples/hello++
hermit/usr/libgomp/libgomp.a
usr/xray/xray.odt
hermit/usr/xray/xray.odt
librs/src/arch/x86_64/boot.bin

# PCI ID database
pci_ids_parser/pci.ids
target/pci.ids
src/arch/x86_64/kernel/pci.ids
src/arch/x86_64/kernel/pci_ids.rs

# XRay report
usr/xray/tools/report.xray
hermit/usr/xray/tools/report.xray

# asciinema JSON files
tools/hermit_ircce.json
hermit/tools/hermit_ircce.json
tools/hermit_kvm.json
hermit/tools/hermit_kvm.json
tools/hermit_openmp.json
hermit/tools/hermit_openmp.json

filter-kernel.sh

#!/usr/bin/env bash

set -euo pipefail
IFS=$'\n\t'

set -x

git clone https://github.com/hermit-os/kernel.git kernel-fsck
cd kernel-fsck

git switch -c main-fsck

# Squash this commit into it's parent
FIXUP_REF=db6e99711ca23fd1e109598b82cdb78ef9d52328
git switch --detach $FIXUP_REF
git commit --amend --no-edit --fixup HEAD^
GIT_COMMITTER_DATE=$(git show -s --format=%cI $FIXUP_REF) git rebase --autosquash HEAD^^
NEW_REF=$(git rev-parse HEAD)
git switch main-fsck

# Replace the old commit with the new one
git replace $FIXUP_REF^ "$NEW_REF"
git replace -l

# Bake replacement into history and remove some files
git filter-repo --invert-paths --paths-from-file ../remove-from-history.txt --refs main-fsck --force

Results

Repository size according to GitLab when only pushing main-fsck:

1. Squashing the offending commit: 52.1 MiB repo + 11.5 MiB LFS = 63.6 MiB total
2. Filtering out all files that were in LFS at one point: 33.6 MiB total
   • img/demo.gif (uses LFS and non-LFS storage)
   • img/hermitcore_logo.png (uses LFS and non-LFS storage)
3. Filtering out binary objects: 11.3 MiB total
   • config/bzImage
   • config/initrd.cpio
   • hermit/usr/examples/hello++
   • hermit/usr/libgomp/libgomp.a
   • usr/xray/xray.odt, hermit/usr/xray/xray.odt
   • librs/src/arch/x86_64/boot.bin
4. Filtering out the PCI ID database and the corresponding build-time generated Rust file: 10.8 MiB total
   • pci_ids_parser/pci.ids, src/arch/x86_64/kernel/pci.ids, target/pci.ids
   • src/arch/x86_64/kernel/pci_ids.rs
5. Filtering out an example XRay report: 10.8 MiB total
   • usr/xray/tools/report.xray, hermit/usr/xray/tools/report.xray
6. Filtering out asciinema JSON files: 10.7 MiB total
   • tools/hermit_ircce.json, hermit/tools/hermit_ircce.json
     
   • tools/hermit_kvm.json, hermit/tools/hermit_kvm.json
   • tools/hermit_openmp.json, hermit/tools/hermit_openmp.json

At the end, git filter-repo --analyze reports the following statistics on a repo containing only main-fsck:

== Overall Statistics ==
  Number of commits: 8541
  Number of filenames: 1300
  Number of directories: 185
  Number of file extensions: 39

  Total unpacked size (bytes): 218742593
  Total packed size (bytes): 7473231

For reference, this is the report on the current repo, without main-fsck:

== Overall Statistics ==
  Number of commits: 11783
  Number of filenames: 1340
  Number of directories: 188
  Number of file extensions: 47

  Total unpacked size (bytes): 280252014
  Total packed size (bytes): 51589333

This is the report on the current repo, with main-fsck:

== Overall Statistics ==
  Number of commits: 20150
  Number of filenames: 1340
  Number of directories: 188
  Number of file extensions: 47

  Total unpacked size (bytes): 280252014
  Total packed size (bytes): 51590289

Note that the size of the full repo will not get smaller, even after fully migrating to main-fsck. This is because we need to keep the old main-pre-fsck around forever to keep tags and references to specific commits working. The size benefit will come to all clones who only clone main-fsck with --single-branch, though.

Most importantly, though, this now completes successfully:

git fsck
    --unreachable \
    --dangling \
    --tags \
    --root \
    --cache \
    --reflogs \
    --full \
    --strict \
    --lost-found \
    --progress \
    --name-objects \
    --references

Status

Before pushing main-fsck to the main repo, I would like to gather feedback on the files filtered out of the branch history. Should we filter out more files? Should we filter out fewer files? I'd like to avoid pushing a history-rewritten branch more than once, since this causes explosions in the number of overall commits in the repository.

History

There was an even worse force-push back in February 2021 with histories that don't share a root commit (v0.3.48..v0.3.49, GitHub comparison):

$ git diff v0.3.48..7ac172ff054dab4cfde26e3e74a6a058a475107d
diff --git a/.gitattributes b/.gitattributes
index 5f9b1ec38..af9d07100 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -1,2 +1,3 @@
 *.gif filter=lfs diff=lfs merge=lfs -text
 *.png filter=lfs diff=lfs merge=lfs -text
+. text !filter !merge !diff
diff --git a/img/demo.gif b/img/demo.gif
index d5ed4269a..83d900526 100644
Binary files a/img/demo.gif and b/img/demo.gif differ
diff --git a/img/hermitcore_logo.png b/img/hermitcore_logo.png
index 123ab3cd4..7d578afb4 100755
Binary files a/img/hermitcore_logo.png and b/img/hermitcore_logo.png differ

Rebasing feature branches

Rebasing is surprisingly easy. To rebase a feature branch from main-pre-fsck onto main-fsck, just run:

git rebase --onto main-fsck main-pre-fsck <branch>

For details, see RECOVERING FROM UPSTREAM REBASE (the hard case).