How to remove the objects reference according to the hash?

[aaaa-j]

Describe the issue
Sorry for opening an issue but I really need some help.

We already have a large repository(LFS>300G), but a lot large files are not tracked in LFS. And I am trying to git lfs migrate import those files.
However, I got errors when I tried to run git lfs fetch -all:
$ git lfs fetch --all fetch: 35658 object(s) found, done. fetch: Fetching all references... [5f3c4cb4bcfe815ceaff849ffdb19f85c2b067608fde04a98f037eeaf4ceea08] Object '5f3c4cb4bcfe815ceaff849ffdb19f85c2b067608fde04a98f037eeaf4ceea08' not found: [404] Object '5f3c4cb4bcfe815ceaff849ffdb19f85c2b067608fde04a98f037eeaf4ceea08' not found .......(more than 200 objects) error: failed to fetch some objects from 'http://xxxx.git/info/lfs'

When I pushed to another repository, I got same errors. I can't continue my work. T_T

I guess I need to remove the missing objects reference from history.

I have read some similar issues' solution, such as issue3245. But I'm still confused about what to do.

Wish for some advice!
Thanks a lot.

Output of git lfs env
git-lfs/2.9.2 (GitHub; windows amd64; go 1.12.7; git 0274d85)
git version 2.21.0.windows.1

[chrisd8088]

Hey, I'm sorry to hear you're having trouble. You're also welcome to start a thread in our new discussions forum, where other LFS users might have tips to provide.

The comments and scripts in #3245 primarily look at resolving a problem where a single LFS file is missing, but could be adapted to work for more files. However, given that it sounds like you're missing more than 200 objects, I'm not sure that would be the safest approach, as you could end up removing a lot of history. If you definitely want to remove them in the manner outlined there, though, we could work up the appropriate script commands.

My own instinct in a situation like this would be to try to understand better what happened with the git lfs migrate import command and why it left the repository in this state. Do you have to a way to reproduce that sequence of steps?

My other thought is that this error usually means your Git LFS client isn't able to locate a large file blob on the LFS server, which implies the server doesn't have it. You might try git lfs push --all, which tries to push all LFS objects up to the server.

[aaaa-j]

I'm sorry that I didn't introduce clearly my needs.
Though our team has being using LFS for a long time, there are still many big files uploaded in repository. I was going to add those files to LFS, thus updating history and making a new repository.

My mate taught me to run git lfs migrate info to get big files list, then git lfs migrate import and finally git push -f --all to get new repo. So first of all, I need to download whole repository. But after git lfs fetch -all, I got errors above.

If I continue running git lfs migrate import, I will get same errors when I run git push -f --all . So it seems the planned route will not work. I don't know what to do. I guess if I remove those "missing" reference from history, the problem may disappear. That's why I open this issue. (Hope you won't laugh at me >.<)

I doubt there is a problem on our gitlab server result in losing files. But technical supporter didn't help.

Can any hero give me some help? Thanks a lot!!!

[chrisd8088]

OK, thanks for the additional context; I think perhaps I understand better what your situation is. If I follow, you have an existing repo with some LFS files, and when you try to fetch, not every LFS blob is available on the server.

To quote from this #4099 (comment), one thing you might try is to see if anyone else who has been working on the repo has the missing objects and can push them from their development machines with git lfs push --all.

I suppose a first step is to get a list of all the missing Git LFS object IDs. If you don't have that from your git lfs fetch --all tries, you might try something like the following to get a list of LFS OIDs and corresponding file paths for which you are missing LFS object files in your local repository:

$ git lfs fetch --all
$ git lfs ls-files --all -l | \
  perl -n -e 'chomp; @x = split; next if $x[1] eq "*"; $o = $x[0]; $p = substr($o, 0, 2) . "/" . substr($o, 2, 2) . "/$o"; print "$o $x[2]\n" unless -f ".git/lfs/objects/$p";'

That might help identify at least some of the file paths associated with the missing LFS objects (but not the Git commits they are in); perhaps some of your colleagues will recognize them and be able to supply those files to the server with git lfs push --all or a more specific push. (Note that git lfs ls-files may not report all file paths for a given LFS object, because objects will be shared when the same LFS-tracked file is committed under different file paths. But this should at least give you some idea of what you're missing.)

If the missing LFS objects are all in a particular branch (perhaps a working branch, and not the main one), you might be able to avoid trying to rewrite history by just ignoring that branch when fetching, since LFS normally only tries to fetch what is needed for the current working tree.

---

If you really want to remove these from your history, I would first get a list of the LFS object IDs you want to remove into a simple file with one OID per line and nothing else, e.g.:

5f3c4cb4bcfe815ceaff849ffdb19f85c2b067608fde04a98f037eeaf4ceea08
dabbdb94977298ff5a8d06a200ad7ec9539e0644a15b38143ad03f9fa2a7b772
...

You could do that with a modified version of the command above which only outputs the OID SHA256s (replace print "$o $x[2]\n" with just print "$o\n", or pipe the whole thing into a cut -d' ' -f1 or awk '{print $1}', for instance). Perhaps something like:

$ git lfs fetch --all
$ git lfs ls-files --all -l | \
  perl -n -e 'chomp; @x = split; next if $x[1] eq "*"; $o = $x[0]; $p = substr($o, 0, 2) . "/" . substr($o, 2, 2) . "/$o"; print "$o\n" unless -f ".git/lfs/objects/$p";' \
  >/path/to/lfs-sha256s-to-remove.txt

Note that the git lfs fetch --all is important here because otherwise you'll get a list of all the LFS objects you don't have locally, which may be much longer than the list of LFS objects which are missing from the server. So you want to start by getting as many as the server has down to your local system.

Once you are sure (really sure!) you have the list of LFS object ID SHA256s you want to remove from your history in a file, let's say named lfs-sha256s-to-remove.txt, you could run a modified version of the command from #3245 (comment) to get the corresponding Git SHA1s:

$ git rev-list --all --objects -- | \
  cut -d' ' -f1 | \
  while read sha1; do
    if git cat-file -p "$sha1" | grep -q -F -f /path/to/lfs-sha256s-to-remove.txt; then
      echo "$sha1";
    fi;
  done \
  >/path/to/git-sha1s-to-remove.txt

Then you can take that list of Git SHA1s which you want to remove from your history and use a modified version of the second command from #3245 (comment), adding in a lot of detail from this helpful blog post:

$ git filter-branch --index-filter '
  git ls-files --stage |
  grep -v -F -f /path/to/git-sha1s-to-remove.txt |
  GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info &&
  mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE
' --prune-empty --tag-name-filter cat -- --all

$ git for-each-ref --format='delete %(refname)' refs/original | git update-ref --stdin
$ git reflog expire --expire-unreachable=now --all
$ git gc --prune=now

At this point your local repo should, hopefully, have none of the missing objects anywhere in its history, refs, etc. (and any commits which created only these missing objects will be removed, thanks to the --prune-empty flag). But that's only your local repository; you've still got the headache that you will now need to push these changes and overwrite the history elsewhere, like on your origin remote, with git push origin --force --all and so forth. CAUTION: That will be destructive, so please be certain you want to do this!

I would strongly suggest you to try to find some other workaround if possible -- see the warnings for git filter-branch -- because among other things, these changes may rewrite a lot of your history, including the SHAs, so you may have many new conflicts if you try to push afterwards (which is why the git push ... --force flag is used above):

WARNING! The rewritten history will have different object names for all
the objects and will not converge with the original branch. You will
not be able to easily push and distribute the rewritten branch on top
of the original branch. Please do not use this command if you do not
know the full implications....

If the git filter-branch commands don't seem to work, but you do want to rewrite your history, you might be better off trying a tool like BFG or git filter-repo (maybe with the --strip-blobs-with-ids <blob_id_filename> option, passing the git-sha1s-to-remove.txt file?) for this job.

[aaaa-j]

Thanke you very much for your detailed comment!!! I've learned a lot.