Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Files get skipped, repository or object not found. #1151

Closed
StefanLundmark opened this issue Apr 16, 2016 · 12 comments
Closed

Files get skipped, repository or object not found. #1151

StefanLundmark opened this issue Apr 16, 2016 · 12 comments

Comments

@StefanLundmark
Copy link

Upon pushing to a remote, LFS skipped all of my files:
Git LFS: (16 of 11213 files, 11205 skipped) 1.22 MB / 1.42 GB, 1019.20 MB skipped

And then stops with the following error:

Repository or object not found: Check that it exists and that you have proper access to it

I'm now blocked from pushing to the repository. Any idea what's going on?

@technoweenie
Copy link
Contributor

Skipped files means the server already has it, and you don't need to upload it again. I need more details to troubleshoot this though:

  • What version of LFS are you on? (run git lfs env)
  • Where is your git repository hosted?
  • What is the object that's missing?

The easiest way to find this is by running:

$ git remote -v
$ GIT_TRACE=1 git push ....

If this is for a GitHub repository, you can send me the details privately to https://github.com/contact. If it's hosted elsewhere, you may be better off contacting their support. If you're just completely stuck, you can add the output to this issue, or send it to me privately, and I'll do my best.

@StefanLundmark
Copy link
Author

That makes sense, but why is it treated like an error? I'm using SmartGit and SourceTree and both stop the push when the files get ignored and print it as an error. Could the issue be at the server? It's a GitLab server.

Git LFS version is 1.1.2 but I'm not sure if SourceTree uses the locally installed version or it's own. There are thousands of skipped objects. I guess it's due to an aborted push earlier, but I still don't understand why it doesn't just ignore the warnings and keep pushing..

I'm not completely stuck as we still have the repository at Perforce left. The reason we tried to move to GitLab is because GitHub doesn't support such a large repository and we have a lot textures and sounds. It's a shame but I understand why the limitation is there.

Thanks for your time Rick!

@technoweenie
Copy link
Contributor

That makes sense, but why is it treated like an error? I'm using SmartGit and SourceTree and both stop the push when the files get ignored and print it as an error. Could the issue be at the server? It's a GitLab server.

Skipped files aren't an error. That's totally normal. I wonder if we should even show that info :) The response "Repository or object not found" has to do with an http response from the lfs server. You'll have to work with Gitlab on that. If you want to post the info that I requested, I can help nail down what specific questions to ask them.

@StefanLundmark
Copy link
Author

I really appreciate the offer, thanks a lot! I managed to track this down.. we're not using HTTPS which LFS defaults to. I tried to force HTTP by issuing the following command:

git config --add lfs.url "http://host.com/group/repo.git/info/lfs/objects/batch"

And now it's pushing! :)

One question though: are you sure skipped files always mean the files exist on the server? What about when the connection is down? These files couldn't possibly have existed on the server to begin with because there was no connection?

In any case, thanks again!

@technoweenie
Copy link
Contributor

Wow, you should not be using HTTP :) Glad you got it to work at least.

If the connection is down, the git pre-push hook will fail. What version of LFS were you using at the time you posted this issue? And what kind of git push is it? Is it the initial push with a bunch of refs to a brand new git and lfs server?

@StefanLundmark
Copy link
Author

StefanLundmark commented Apr 20, 2016

Just wanted to get something up and running so I can test it. =)

LFS is at 1.1.2. It's the initial push.

The push failed with a huge stacktrace:

fatal error: runtime: failed to release pages

runtime stack:
runtime.throw(0x831cc0, 0x20)
/usr/local/Cellar/go/1.5.3/libexec/src/runtime/panic.go:527 +0x7f
runtime.sysFree(0x0, 0x10000000, 0x334dfe04)
/usr/local/Cellar/go/1.5.3/libexec/src/runtime/mem_windows.go:86 +0x5b
runtime.mHeap_SysAlloc(0x9dde00, 0x100000, 0x0)
/usr/local/Cellar/go/1.5.3/libexec/src/runtime/malloc.go:418 +0x4a1
runtime.mHeap_Grow(0x9dde00, 0x80, 0x0)
/usr/local/Cellar/go/1.5.3/libexec/src/runtime/mheap.go:628 +0x52
runtime.mHeap_AllocSpanLocked(0x9dde00, 0x80, 0x44f9d5)
/usr/local/Cellar/go/1.5.3/libexec/src/runtime/mheap.go:532 +0x574
runtime.mHeap_Alloc_m(0x9dde00, 0x80, 0x0, 0x1, 0x18e092c0)
/usr/local/Cellar/go/1.5.3/libexec/src/runtime/mheap.go:425 +0x281
runtime.mHeap_Alloc.func1()
/usr/local/Cellar/go/1.5.3/libexec/src/runtime/mheap.go:484 +0x3d
runtime.systemstack(0x334dff00)
/usr/local/Cellar/go/1.5.3/libexec/src/runtime/asm_386.s:283 +0x81
runtime.mHeap_Alloc(0x9dde00, 0x80, 0x0, 0x12920101, 0x42e1bd)
/usr/local/Cellar/go/1.5.3/libexec/src/runtime/mheap.go:485 +0x5a
runtime.largeAlloc(0x100000, 0x1, 0x4472c0)
/usr/local/Cellar/go/1.5.3/libexec/src/runtime/malloc.go:748 +0xa0
runtime.mallocgc.func3()
/usr/local/Cellar/go/1.5.3/libexec/src/runtime/malloc.go:637 +0x30
runtime.systemstack(0x128fca00)
/usr/local/Cellar/go/1.5.3/libexec/src/runtime/asm_386.s:267 +0x57
runtime.mstart()

/usr/local/Cellar/go/1.5.3/libexec/src/runtime/proc1.go:668

goroutine 100 [running]:

runtime.systemstack_switch()

/usr/local/Cellar/go/1.5.3/libexec/src/runtime/asm_386.s:222 fp=0x13169a68 sp=0x13169a64

runtime.mallocgc(0x100000, 0x718c20, 0x1, 0x4025c5)

/usr/local/Cellar/go/1.5.3/libexec/src/runtime/malloc.go:638 +0x77f fp=0x13169ad0 sp=0x13169a68

runtime.newarray(0x718c20, 0x100000, 0x40da7f)

/usr/local/Cellar/go/1.5.3/libexec/src/runtime/malloc.go:780 +0xbc fp=0x13169af4 sp=0x13169ad0

runtime.makeslice(0x7100e0, 0x100000, 0x0, 0x100000, 0x0, 0x0, 0x0, 0x0)

github.com/github/git-lfs/lfs.Stack(0x0, 0x0, 0x0)

/Users/rick/go/src/github.com/github/git-lfs/lfs/errors.go:556 +0x53 fp=0x13169b64 sp=0x13169b34

github.com/github/git-lfs/lfs.newWrappedError(0x328f0018, 0x1b071de8, 0x47d9bc80, 0xb5, 0x0, 0x0)

/Users/rick/go/src/github.com/github/git-lfs/lfs/errors.go:304 +0x13a fp=0x13169bdc sp=0x13169b64

github.com/github/git-lfs/lfs.Errorf(0x328f0018, 0x1b071de8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)

/Users/rick/go/src/github.com/github/git-lfs/lfs/errors.go:222 +0x10f fp=0x13169c28 sp=0x13169bdc

github.com/github/git-lfs/lfs.Error(0x328f0018, 0x1b071de8, 0x0, 0x0)

/Users/rick/go/src/github.com/github/git-lfs/lfs/errors.go:208 +0x4b fp=0x13169c50 sp=0x13169c28
github.com/github/git-lfs/lfs.defaultError(0x43811e00, 0x0, 0x0)

/Users/rick/go/src/github.com/github/git-lfs/lfs/client.go:645 +0x148 fp=0x13169d04 sp=0x13169c50

github.com/github/git-lfs/lfs.handleResponse(0x43811e00, 0x4aab77e0, 0x0, 0x0)

/Users/rick/go/src/github.com/github/git-lfs/lfs/client.go:600 +0xf5 fp=0x13169d3c sp=0x13169d04

github.com/github/git-lfs/lfs.doHttpRequest(0x45d1dab0, 0x4aab77e0, 0x0, 0x0, 0x0)
/Users/rick/go/src/github.com/github/git-lfs/lfs/client.go:513 +0x408 fp=0x13169dcc sp=0x13169d3c

github.com/github/git-lfs/lfs.doApiRequestWithRedirects(0x45d1dab0, 0x13169e88, 0x0, 0x4, 0x2c2edd01, 0x65d394, 0x0, 0x0)

/Users/rick/go/src/github.com/github/git-lfs/lfs/client.go:537 +0x89 fp=0x13169e44 sp=0x13169dcc

github.com/github/git-lfs/lfs.doLegacyApiRequest(0x45d1dab0, 0x13, 0x13169f08, 0x0, 0x0)

/Users/rick/go/src/github.com/github/git-lfs/lfs/client.go:413 +0x72 fp=0x13169e9c sp=0x13169e44

github.com/github/git-lfs/lfs.UploadCheck(0x1336f380, 0x79, 0x13169f9c, 0x0, 0x0)

/Users/rick/go/src/github.com/github/git-lfs/lfs/client.go:299 +0x471 fp=0x13169f48 sp=0x13169e9c

github.com/github/git-lfs/lfs.(*Uploadable).Check(0x132cc990, 0x131e8d40, 0x0, 0x0)
/Users/rick/go/src/github.com/github/git-lfs/lfs/upload_queue.go:41 +0x3d fp=0x13169f60 sp=0x13169f48

github.com/github/git-lfs/lfs.(*TransferQueue).individualApiRoutine(0x13586000, 0x131e8980)

/Users/rick/go/src/github.com/github/git-lfs/lfs/transfer_queue.go:132 +0x73 fp=0x13169fc8 sp=0x13169f60

runtime.goexit()

/usr/local/Cellar/go/1.5.3/libexec/src/runtime/asm_386.s:1662 +0x1 fp=0x13169fcc sp=0x13169fc8

created by github.com/github/git-lfs/lfs.(*TransferQueue).launchIndividualApiRoutines.func1

/Users/rick/go/src/github.com/github/git-lfs/lfs/transfer_queue.go:303 +0x5e

goroutine 1 [chan send]:
github.com/github/git-lfs/lfs.(_Batcher).Add(0x1299a4b0, 0x32d70850, 0x12c6c8a0)
/Users/rick/go/src/github.com/github/git-lfs/lfs/batcher.go:38 +0xce
github.com/github/git-lfs/lfs.(_TransferQueue).Add(0x13586000, 0x32d70850, 0x12c6c8a0)

/Users/rick/go/src/github.com/github/git-lfs/lfs/transfer_queue.go:71 +0xba

github.com/github/git-lfs/lfs.(*TransferQueue).Wait(0x13586000)
/Users/rick/go/src/github.com/github/git-lfs/lfs/transfer_queue.go:96 +0x1b5

github.com/github/git-lfs/commands.prePushRef(0x128f2912, 0x28, 0x128e94d0, 0x29)
/Users/rick/go/src/github.com/github/git-lfs/commands/command_pre_push.go:129 +0x46b

github.com/github/git-lfs/commands.prePushCommand(0x9d4f20, 0x128edad0, 0x2, 0x2)

/Users/rick/go/src/github.com/github/git-lfs/commands/command_pre_push.go:72 +0x36d
github.com/github/git-lfs/vendor/_nuts/github.com/spf13/cobra.(*Command).execute(0x9d4f20, 0x128ed9f0, 0x2, 0x2, 0x0, 0x0)

/Users/rick/go/src/github.com/github/git-lfs/vendor/_nuts/github.com/spf13/cobra/command.go:477 +0x2d8
github.com/github/git-lfs/vendor/_nuts/github.com/spf13/cobra.(*Command).Execute(0x9d5b60, 0x0, 0x0)

/Users/rick/go/src/github.com/github/git-lfs/vendor/_nuts/github.com/spf13/cobra/command.go:551 +0x44f

github.com/github/git-lfs/commands.Run()
/Users/rick/go/src/github.com/github/git-lfs/commands/commands.go:88 +0x23

main.main()
/Users/rick/go/src/github.com/github/git-lfs/git-lfs.go:34 +0x107

goroutine 5 [syscall, 50 minutes]:
os/signal.loop()

/usr/local/Cellar/go/1.5.3/libexec/src/os/signal/signal_unix.go:22 +0x1a

created by os/signal.init.1
/usr/local/Cellar/go/1.5.3/libexec/src/os/signal/signal_unix.go:28 +0x36

goroutine 28359 [chan send]:

github.com/github/git-lfs/lfs.(*Batcher).acceptInput(0x1299a4b0)

/Users/rick/go/src/github.com/github/git-lfs/lfs/batcher.go:72 +0xee

created by github.com/github/git-lfs/lfs.(*Batcher).Add

/Users/rick/go/src/github.com/github/git-lfs/lfs/batcher.go:35 +0x9f

goroutine 9 [chan receive, 50 minutes]:

main.main.func1(0x129a5ac0, 0x128ed8f0)

/Users/rick/go/src/github.com/github/git-lfs/git-lfs.go:22 +0x49

created by main.main

/Users/rick/go/src/github.com/github/git-lfs/git-lfs.go:32 +0x102

goroutine 97 [select]:

net/http.(*persistConn).readLoop(0x128e3740)

/usr/local/Cellar/go/1.5.3/libexec/src/net/http/transport.go:976 +0x8e8

created by net/http.(*Transport).dialConn
/usr/local/Cellar/go/1.5.3/libexec/src/net/http/transport.go:685 +0xabc

goroutine 65 [chan receive]:

github.com/github/git-lfs/lfs.(*TransferQueue).errorCollector(0x13586000)
/Users/rick/go/src/github.com/github/git-lfs/lfs/transfer_queue.go:256 +0x4a

created by github.com/github/git-lfs/lfs.(*TransferQueue).run
/Users/rick/go/src/github.com/github/git-lfs/lfs/transfer_queue.go:317 +0x36

goroutine 67 [chan receive, 50 minutes]:
github.com/github/git-lfs/lfs.(*TransferQueue).transferWorker(0x13586000)

/Users/rick/go/src/github.com/github/git-lfs/lfs/transfer_queue.go:270 +0x72

created by github.com/github/git-lfs/lfs.(*TransferQueue).run

/Users/rick/go/src/github.com/github/git-lfs/lfs/transfer_queue.go:322 +0x127

goroutine 68 [chan receive, 50 minutes]:

github.com/github/git-lfs/lfs.(*TransferQueue).transferWorker(0x13586000)

/Users/rick/go/src/github.com/github/git-lfs/lfs/transfer_queue.go:270 +0x72

created by github.com/github/git-lfs/lfs.(*TransferQueue).run
/Users/rick/go/src/github.com/github/git-lfs/lfs/transfer_queue.go:322 +0x127

goroutine 69 [chan receive, 50 minutes]:

github.com/github/git-lfs/lfs.(*TransferQueue).transferWorker(0x13586000)
/Users/rick/go/src/github.com/github/git-lfs/lfs/transfer_queue.go:270 +0x72

created by github.com/github/git-lfs/lfs.(*TransferQueue).run

/Users/rick/go/src/github.com/github/git-lfs/lfs/transfer_queue.go:322 +0x127

goroutine 76 [chan receive, 50 minutes]:

github.com/github/git-lfs/lfs.(*TransferQueue).launchIndividualApiRoutines.func1(0x13586000)

/Users/rick/go/src/github.com/github/git-lfs/lfs/transfer_queue.go:305 +0x7a

created by github.com/github/git-lfs/lfs.(*TransferQueue).launchIndividualApiRoutines
/Users/rick/go/src/github.com/github/git-lfs/lfs/transfer_queue.go:310 +0x32

goroutine 98 [select]:
net/http.(_persistConn).writeLoop(0x128e3740)
/usr/local/Cellar/go/1.5.3/libexec/src/net/http/transport.go:1009 +0x303
created by net/http.(_Transport).dialConn
/usr/local/Cellar/go/1.5.3/libexec/src/net/http/transport.go:686 +0xadc

goroutine 75 [chan send]:
github.com/github/git-lfs/lfs.(_TransferQueue).legacyFallback(0x13586000, 0x13568000, 0x64, 0x64)
/Users/rick/go/src/github.com/github/git-lfs/lfs/transfer_queue.go:181 +0x1a6
created by github.com/github/git-lfs/lfs.(_TransferQueue).batchApiRoutine
/Users/rick/go/src/github.com/github/git-lfs/lfs/transfer_queue.go:210 +0x48b

Wow that's huge, sorry. Let me know if I should trim it down and put up a link instead.

@technoweenie
Copy link
Contributor

Yikes, that looks like a memory issue on your system. How's your memory utilization while the push is running?

Initial pushes typically scan the entire history of each branch to push. Since your branches all have common ancestors. You can see this when you push by looking for multiple "Git LFS" prompts:

Git LFS: (11197 of 11197 files, 0 skipped)
Git LFS: (16 of 11213 files, 11205 skipped)
Git LFS: (0 of 11213 files, 11213 skipped)

In my completely fabricated scenario above, the same 11197 objects are being uploaded 3 times (for 3 different branches). Git LFS v1.2 has a major optimization on initial pushes here, where those duplicate objects in the same git command are ignored. The same push then might look more like:

Git LFS: (11197 of 11197 files, 0 skipped)
Git LFS: (16 of 16 files, 0 skipped)
Git LFS: (0 of 0 files, 0 skipped)

@StefanLundmark
Copy link
Author

StefanLundmark commented Apr 20, 2016

Makes sense, thanks! I guess I'm just confused why those warnings are gone after the change to HTTP. I'm happy with the additional progress though. :)

Memory is 78% free on the server and 39% free on the client. I'll try with 1.2 tomorrow incase that helps.

Edit: Same result. Memory usage is pretty stable and never goes above 500MB.

@technoweenie
Copy link
Contributor

I'm just confused why those warnings are gone after the change to HTTP

You mean this error from your initial post?

Repository or object not found: Check that it exists and that you have proper access to it

That depends on how your server is configured. If looks like it just doesn't support LFS properly.

Memory is 78% free on the server and 39% free on the client.
Same result. Memory usage is pretty stable and never goes above 500MB.

Is that the git-lfs process, or the entire server? Can you estimate how many objects total it's processing based on the Git LFS: prompts?

@StefanLundmark
Copy link
Author

It just dies and shows the runtime stack without showing any info about the objects processed.

Oh and those numbers are for the client and the server, not the individual processes.

Would love to try this out on GitHub but my repository is too large, hehe!

@StefanLundmark
Copy link
Author

On BitBucket the same repository pushes just fine.

@technoweenie
Copy link
Contributor

Cool, glad you found a workaround. Sorry we couldn't help you, nothing you posted points to a server side issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants