'Unknown object xxx' when running 'keybase git cleanup'

[kg4zow]

Describe the bug

$ keybase git gc REPO
This may take several minutes...
▶ ERROR Unknown object 39326639643863333632393930373834666538303164323465393636616234383764636663313938 tag *object.Tag

To Reproduce

1. Run a git push command which shows the "Tip: this repo could be sped up with some garbage collection." message and shows the appropriate keybase git gc command.
2. Run the suggested keybase git gc command.
3. After a few seconds the error message shown above appears.

Expected behavior

Garbage collection finishes normally.

Additional numbers

The number 345567

[kg4zow]

✅ FIXED IT.

$ keybase git gc REPO
This may take several minutes...
Success!

Troubleshooting process

Out of curiosity I tried fixing it "by hand", i.e...

cd /keybase/private/USER/.kbfs_git/REPO/
git gc

At first this looked like it was going to work, but after a few seconds it failed with a long (7800+) series of this message, over and over:

error: non-monotonic index ./objects/pack/._.tmp-65968-pack-7a80af4165bf765cda76090ada4744c0a3b58997.idx

... followed by three of this message:

error: non-monotonic index ./objects/pack/._pack-7a80af4165bf765cda76090ada4744c0a3b58997.idx

The filenames involved start with ._, which suggested the problem might be macOS resource fork files.

Under the native macOS filesystems (HFS, HFS+, and APFS), every file has two "forks". The "data fork" is the normal "stream of bytes" that you would expect a file to contain, while the "resource fork" contains a series of "resources", including metadata created by macOS itself. Other filesystems support the same concept, but macOS is the only operating system which makes regular use of them. (NTFS calls them "Alternate Data Streams", and allows an arbitrary number of "streams" in each file.)

When macOS stores files on a filesystem like FAT, ExFAT, KBFS, etc. which doesn't support forks, the contents of the file's resource fork are stored in a sidecar file whose name is ._ plus the original filename.

This is why, if you use a FAT-formatted USB stick on a Mac, then look at it using Linux (or windows, I guess?), you'll see these ._ files. They are the "resource forks" of any file that macOS interacted with. In 99% of cases, these files can be removed without hurting anything.

My guess about what happened

At some point in the past, I was poking around in the KBFS directory which exposes the repo's back-end storage while a git gc command was running.

• git gc created a .tmp-65968-pack-7a80af4165bf765cda76090ada4744c0a3b58997.idx file.
• Something in macOS saw that file, decided to store something in its resource fork, and created a ._.tmp-65968-pack-7a80af4165bf765cda76090ada4744c0a3b58997.idx file.
• When git gc finished, it deleted or renamed the .tmp-65968-pack-7a80af4165bf765cda76090ada4744c0a3b58997.idx file.
• The ._.tmp-65968-pack-7a80af4165bf765cda76090ada4744c0a3b58997.idx file wasn't deleted or renamed, and has been sitting there ever since.

I don't know if keybase git gc works by running an external git gc command behind the scenes, or if it uses a Golang library to do the garbage collection directly. Whatever it was, it tried to read the old ._.tmp-65968-pack-7a80af4165bf765cda76090ada4744c0a3b58997.idx file as if it were any other .idx file. It wasn't able to parse the file's contents, which caused the Unknown object errors.

Commands

For a personal repo:

dot_clean -v /keybase/private/USER/.kbfs_git/REPO && keybase git gc REPO

For a team repo:

dot_clean -v /keybase/team/TEAM/.kbfs_git/REPO && keybase git gc --team TEAM REPO