Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After migrating my Azure repository from one project to another, checking out LFS files within a build pipeline results in a 404 object not found error #3809

Open
edgariscoding opened this issue Sep 5, 2019 · 7 comments

Comments

@edgariscoding
Copy link

commented Sep 5, 2019

Error for all files: LFS object not found: [404] LFS object not found

I assume this is a result of the migration. How can I remedy this? I have all the files locally on my workstation.

@bk2204

This comment has been minimized.

Copy link
Member

commented Sep 5, 2019

Hey,

If you have all the LFS files locally, you can run git lfs push --all origin (assuming your remote is origin) to push the objects to the new repository. If you're lacking some of the objects, you can try to fetch them using git lfs fetch --all old-remote and then do a git lfs push --all to the new remote.

If you're still missing objects, you'll want to reach out to Azure support, since they probably still have them available somewhere.

@bk2204 bk2204 closed this Sep 5, 2019

@edgariscoding

This comment has been minimized.

Copy link
Author

commented Sep 5, 2019

Hey,
If you have all the LFS files locally, you can run git lfs push --all origin (assuming your remote is origin) to push the objects to the new repository. If you're lacking some of the objects, you can try to fetch them using git lfs fetch --all old-remote and then do a git lfs push --all to the new remote.
If you're still missing objects, you'll want to reach out to Azure support, since they probably still have them available somewhere.

I tried doing a git lfs push --all origin but it seems to get stuck uploading at the same exact place even after cancelling the operation. It's been stuck for more than 30 mins now.

I researched this issue and other people seemed to run into this when their credentials timed out. I'm using git credential manager for windows if that makes a difference. (credential.helper manager)

Another weird thing is that when I run the lfs push command, the locking support detected line is displayed 12 times. See below:

image

I also ran this command with git_trace and git_curl_verbose:

https://pastebin.com/3rcpw1Va

Here's my git lfs env:

git-lfs/2.8.0 (GitHub; windows amd64; go 1.12.2; git 30af66bb)
git version 2.23.0.windows.1

Endpoint=https://CiraConnect@dev.azure.com/CiraConnect/CiraConnect/_git/CiraConnect%20CiraNet%20ASPNET.git/info/lfs (auth=basic)
LocalWorkingDir=C:\Users\edgar.sanchez\source\repos\CiraConnect CiraNet ASPNET
LocalGitDir=C:\Users\edgar.sanchez\source\repos\CiraConnect CiraNet ASPNET\.git
LocalGitStorageDir=C:\Users\edgar.sanchez\source\repos\CiraConnect CiraNet ASPNET\.git
LocalMediaDir=C:\Users\edgar.sanchez\source\repos\CiraConnect CiraNet ASPNET\.git\lfs\objects
LocalReferenceDirs=
TempDir=C:\Users\edgar.sanchez\source\repos\CiraConnect CiraNet ASPNET\.git\lfs\tmp
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=C:\Users\edgar.sanchez\source\repos\CiraConnect CiraNet ASPNET\.git\lfs
AccessDownload=basic
AccessUpload=basic
DownloadTransfers=basic
UploadTransfers=basic
GIT_EXEC_PATH=C:/Program Files/Git/mingw64/libexec/git-core
GIT_LFS_PATH=C:\Program Files\Git LFS
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"
@bk2204

This comment has been minimized.

Copy link
Member

commented Sep 6, 2019

Hmmm. We don't usually see hangs at this point when pushing. We do have a known but unreproducible issue that occurs near xfer: adapter "basic" worker 1 waiting for Auth, but I'm not seeing that at the end of your output. I'm not seeing any of the things that usually show up during hanging pushes.

Could you try another run, but this one with all of GIT_TRACE=1, GIT_CURL_VERBOSE=1, and GIT_TRANSFER_TRACE=1? The latter sometimes provides interesting output that the middle variable misses.

@edgariscoding

This comment has been minimized.

Copy link
Author

commented Sep 6, 2019

Hmmm. We don't usually see hangs at this point when pushing. We do have a known but unreproducible issue that occurs near xfer: adapter "basic" worker 1 waiting for Auth, but I'm not seeing that at the end of your output. I'm not seeing any of the things that usually show up during hanging pushes.
Could you try another run, but this one with all of GIT_TRACE=1, GIT_CURL_VERBOSE=1, and GIT_TRANSFER_TRACE=1? The latter sometimes provides interesting output that the middle variable misses.

Sure thing: https://pastebin.com/Q4KDfRGj

@bk2204

This comment has been minimized.

Copy link
Member

commented Sep 6, 2019

It looks like that's producing the same output. We have just fixed a potentially related issue in #3806. Could you try that version and see if it fixes things for you?

Since you're on Windows, I've attached some (unsigned) binaries that you can use to test if you want. You may need to invoke the binary directly (i.e., via git-lfs), since Git Bash can make it hard to override the existing Git LFS binary that's shipped with it.

git-lfs-windows-386-v2.8.0-63-gf730705d.zip
git-lfs-windows-amd64-v2.8.0-63-gf730705d.zip

@bk2204 bk2204 reopened this Sep 6, 2019

@edgariscoding

This comment has been minimized.

Copy link
Author

commented Sep 6, 2019

It looks like that's producing the same output. We have just fixed a potentially related issue in #3806. Could you try that version and see if it fixes things for you?
Since you're on Windows, I've attached some (unsigned) binaries that you can use to test if you want. You may need to invoke the binary directly (i.e., via git-lfs), since Git Bash can make it hard to override the existing Git LFS binary that's shipped with it.
git-lfs-windows-386-v2.8.0-63-gf730705d.zip
git-lfs-windows-amd64-v2.8.0-63-gf730705d.zip

In an attempt to resolve this issue I created a new empty orphaned branch. I then manually copied all files from my existing branch and pushed this new branch. Everything worked perfectly and it uploaded the LFS files, I also tested our build pipeline and it was able to successfully check out all LFS files.

Since we don't yet need any history on this branch I just went ahead and deleted master and renamed this new branch to master.

I assume this issue came about from the repository migration but either way it's working now. Thank you for your help!

@bk2204

This comment has been minimized.

Copy link
Member

commented Sep 6, 2019

Okay, then it looks like we may be hitting a very slow traversal case, which is certainly possible. I'll keep this open in case anyone else sees this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.