Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a New Repository fails #3036

Closed
rudifa opened this issue Oct 12, 2017 · 9 comments
Closed

Create a New Repository fails #3036

rudifa opened this issue Oct 12, 2017 · 9 comments
Labels
more-info-needed The submitter needs to provide more information about the issue

Comments

@rudifa
Copy link

rudifa commented Oct 12, 2017 •
edited

Description

a. The dialog "Create a New Repository" seems to hang.

b. It displays "This directory appears to be a Git repository. Would you like to add this repository instead?" whether the directory contains a git repository or not (see 1st screenshot).

c. Button "Create repository" never changes the color to dark blue (I suppose that it should do so when ready to create).

d. Click on "Cancel" sometimes produces an Error dialog that suggests to increase the options.maxBuffer (which I did not try; see the 2nd screenshot).

e. Attached logfile 23:23 2017-10-12.desktop.production.log grew to 11.5 MB and some sequences suggest infinite loops.

Version

GitHub Desktop version: 1.0.3

OS version: OS X 10.11.6

Steps to Reproduce

  1. Click on menu item "File : New Repository" and try to fill in the dialog "Create a New Repository", selecting a directory not containing a git repository.

Expected behavior: Desktop should create an empty repository and select it as Current.

Actual misbehavior: See above Description.

Reproduces how often: 100%

Logs

2017-10-12.desktop.production.log

Additional Information

Workaround: $ git init

screen shot 2017-10-12 at jeu 12 oct 15 00 57

screen shot 2017-10-12 at jeu 12 oct 15 08 51

@rudifa update 2017-10-15

After I submitted this bug report, I made several observations on my computer, as follows:

The root cause of behaviors that I observed was the presence of a .git directory in my home directory /Users/rudifarkas. This .git was forgotten there since 2013, and I don't know why and how I copied it there, probably by mistake. It contains 2032 items, 73.4 MB.

For this reason, an attempt to Create a New Repository in any of my subdirectories produces the message ""This directory appears to be a Git repository. Would you like to add this repository instead?" with the button "Create repository" disabled.

In this situation, a click on the "add this repository" adds the "repository" rudifarkas, which is the intended behavior.

The additional buggy behavior (Error dialog shown above, as well as presence, after a while, of tens of process instances of git on the computer) was 100% reproducible when I wrote the bug report (2017-10-12).

Today (after I rebooted my computer 2 days ago), I tried to reproduce these errors, by repeatedly trying to Create a New Repository (with that ~/.gitstill present), and clicking on Cancel or on "add this repository". I did get the above Error box a few times, and the log below does contain a git lfs track exited with an unexpected code: 2., and a sequence of goroutine logs, within a 5 second interval.

2017-10-15.desktop.production.log
@shiftkey
Copy link
Member

@rudifa thanks for the log file!

The options.maxBuffer message is a NodeJS limitation, and related to how we invoke commands. We need to switch these over to using a streaming API, but we haven't nailed down which commands are failing so we can port those first.

Can you tell me about the folder you are using to initialize the repository? We also only do a git init on the folder, and I don't think git init generates significant output, so I think something else is failing in this flow.

The log file shows this error that is very interesting:

2017-10-12T12:56:58.277Z - info: [ui] Executing isUsingLFS: git lfs track (took 137.960s)
2017-10-12T12:56:58.487Z - error: [ui] `git lfs track` exited with an unexpected code: 2.
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7fff9c943f06 m=2 sigcode=0

goroutine 0 [idle]:

goroutine 1 [runnable]:
github.com/git-lfs/git-lfs/tools.FastWalkGitRepo(0xc420138640, 0x11, 0xc420051390)
	/Users/rick/go/src/github.com/git-lfs/git-lfs/tools/filetools.go:152 +0x25d
github.com/git-lfs/git-lfs/git.findAttributeFiles(0xc420138640, 0x11, 0xc420138600, 0x16, 0x0, 0x0, 0xc420142dd8)
	/Users/rick/go/src/github.com/git-lfs/git-lfs/git/attribs.go:145 +0x13c
github.com/git-lfs/git-lfs/git.GetAttributePaths(0xc420138640, 0x11, 0xc420138600, 0x16, 0x1160b80, 0xc4200423b0, 0xc420051638)
	/Users/rick/go/src/github.com/git-lfs/git-lfs/git/attribs.go:44 +0xcb
github.com/git-lfs/git-lfs/commands.listPatterns()
	/Users/rick/go/src/github.com/git-lfs/git-lfs/commands/command_track.go:218 +0x63
github.com/git-lfs/git-lfs/commands.trackCommand(0xc42013d520, 0x1796cd8, 0x0, 0x0)
	/Users/rick/go/src/github.com/git-lfs/git-lfs/commands/command_track.go:50 +0x1a43
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).execute(0xc42013d520, 0x1796cd8, 0x0, 0x0, 0xc42013d520, 0x1796cd8)
	/Users/rick/go/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:477 +0x17a
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute(0xc42011c000, 0xc420051f08, 0x1)
	/Users/rick/go/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:551 +0x2ee
github.com/git-lfs/git-lfs/commands.Run()
	/Users/rick/go/src/github.com/git-lfs/git-lfs/commands/run.go:68 +0x193
main.main()
	/Users/rick/go/src/github.com/git-lfs/git-lfs/git-lfs.go:35 +0x103

...

Two things stand out:

  • the git lfs track operation (without any arguments) is just to see if any patterns are listed in a .gitattributes file. Do you have any of these in your repository?
  • the action takes over 2 minutes before failing. This is probably due to the state of files on disk, which is why details about the folder structure and .gitattributes files would be handy.

cc @technoweenie for what else we might need to troubleshoot

@shiftkey
Copy link
Member

b. It displays "This directory appears to be a Git repository. Would you like to add this repository instead?" whether the directory contains a git repository or not (see 1st screenshot).

On this form we check whether the folder is part of a repository by asking Git:

git rev-parse --show-cdup [path]

For example, when I run this against a path under the Desktop repo on machine, it tells me the root of the repository:

$ git rev-parse --show-cdup /Users/shiftkey/src/desktop/app/src/ui
../../../
/Users/shiftkey/src/desktop/app/src/ui

If you run that with the folder you provided to the form (it's hard to say from here what you were seeing), what does it return?

@shiftkey shiftkey added bug Confirmed bugs or reports that are very likely to be bugs more-info-needed The submitter needs to provide more information about the issue labels Oct 12, 2017
@technoweenie
Copy link

The track command walks the repository, looking for .gitattributes files since they can technically live anywhere.

That error and stack trace points to this:

https://github.com/git-lfs/git-lfs/blob/150edd19ea727308e5b448b50d63f0c77967d3cd/tools/filetools.go#L151-L154

It's just sequentially calling a function callback for any file entries. However, if you dive into some of the private funcs, it is spawning goroutines to scan subdirectories:

https://github.com/git-lfs/git-lfs/blob/150edd19ea727308e5b448b50d63f0c77967d3cd/tools/filetools.go#L256-L266

My hunch is that the directory tree is so wide and/or deep that it's spinning up too many goroutines. There should probably be a limit.

If I created a custom lfs build for debugging, how easy would it be to run it and send me the log? I'd be really curious to see how big the tree is.

@technoweenie
Copy link

technoweenie commented Oct 13, 2017 •
edited

Hey @rudifa, would you be willing to run this custom build in your terminal? I updated the file walker to limit how many concurrent goroutines it spawns.

git-lfs-darwin-amd64-2.3.3.tar.gz

You can just unzip this anywhere and run it directly. This won't impact your current git/lfs/Desktop installs.

First, run with a limit of 1. This will be the slowest, but it shouldn't crash:

~p/linux git:(master) $ time WG_LIMIT=1 GIT_TRACE=1 ~p/git-lfs/bin/releases/darwin-amd64/git-lfs-2.3.3/git-lfs track
12:38:08.901286 trace git-lfs: run_command: 'git' version
12:38:08.911844 trace git-lfs: run_command: 'git' config -l
12:38:08.917185 trace git-lfs: Install hook: pre-push, force=false, path=/Users/rick/p/linux/.git/hooks/pre-push, upgrading...
12:38:08.917490 trace git-lfs: Install hook: post-checkout, force=false, path=/Users/rick/p/linux/.git/hooks/post-checkout, upgrading...
12:38:08.917772 trace git-lfs: Install hook: post-commit, force=false, path=/Users/rick/p/linux/.git/hooks/post-commit, upgrading...
12:38:08.918043 trace git-lfs: Install hook: post-merge, force=false, path=/Users/rick/p/linux/.git/hooks/post-merge, upgrading...
12:38:08.918230 trace git-lfs: START: buffer=1
12:38:09.847900 trace git-lfs: END: queued=1, run=4424
WG_LIMIT=1 GIT_TRACE=1  track  0.86s user 0.45s system 137% cpu 0.957 total

Running with a limit of 2 should be faster:

~p/linux git:(master) $ time WG_LIMIT=2 GIT_TRACE=1 ~p/git-lfs/bin/releases/darwin-amd64/git-lfs-2.3.3/git-lfs track
12:39:34.488115 trace git-lfs: run_command: 'git' version
12:39:34.498840 trace git-lfs: run_command: 'git' config -l
12:39:34.503783 trace git-lfs: Install hook: pre-push, force=false, path=/Users/rick/p/linux/.git/hooks/pre-push, upgrading...
12:39:34.506362 trace git-lfs: Install hook: post-checkout, force=false, path=/Users/rick/p/linux/.git/hooks/post-checkout, upgrading...
12:39:34.507333 trace git-lfs: Install hook: post-commit, force=false, path=/Users/rick/p/linux/.git/hooks/post-commit, upgrading...
12:39:34.508247 trace git-lfs: Install hook: post-merge, force=false, path=/Users/rick/p/linux/.git/hooks/post-merge, upgrading...
12:39:34.508911 trace git-lfs: START: buffer=2
12:39:35.006166 trace git-lfs: END: queued=236, run=4189
WG_LIMIT=2 GIT_TRACE=1  track  0.85s user 0.40s system 228% cpu 0.545 total

The new default in the custom build is based on how many CPUs you have:

~p/linux git:(master) $ time WG_LIMIT=0 GIT_TRACE=1 ~p/git-lfs/bin/releases/darwin-amd64/git-lfs-2.3.3/git-lfs track
12:40:19.310186 trace git-lfs: run_command: 'git' version
12:40:19.329865 trace git-lfs: run_command: 'git' config -l
12:40:19.335427 trace git-lfs: Install hook: pre-push, force=false, path=/Users/rick/p/linux/.git/hooks/pre-push, upgrading...
12:40:19.336050 trace git-lfs: Install hook: post-checkout, force=false, path=/Users/rick/p/linux/.git/hooks/post-checkout, upgrading...
12:40:19.336519 trace git-lfs: Install hook: post-commit, force=false, path=/Users/rick/p/linux/.git/hooks/post-commit, upgrading...
12:40:19.336984 trace git-lfs: Install hook: post-merge, force=false, path=/Users/rick/p/linux/.git/hooks/post-merge, upgrading...
12:40:19.337418 trace git-lfs: START: buffer=8
12:40:19.584892 trace git-lfs: END: queued=1943, run=2482
WG_LIMIT=0 GIT_TRACE=1  track  1.31s user 0.41s system 601% cpu 0.286 total

You can even try a higher limit. It doesn't impact the speed in my tests, but hopefully it still doesn't crash:

~p/linux git:(master) ✗$ time WG_LIMIT=256 GIT_TRACE=1 ~p/git-lfs/bin/releases/darwin-amd64/git-lfs-2.3.3/git-lfs track
12:42:51.054553 trace git-lfs: run_command: 'git' version
12:42:51.064480 trace git-lfs: run_command: 'git' config -l
12:42:51.070579 trace git-lfs: Install hook: pre-push, force=false, path=/Users/rick/p/linux/.git/hooks/pre-push, upgrading...
12:42:51.070882 trace git-lfs: Install hook: post-checkout, force=false, path=/Users/rick/p/linux/.git/hooks/post-checkout, upgrading...
12:42:51.071097 trace git-lfs: Install hook: post-commit, force=false, path=/Users/rick/p/linux/.git/hooks/post-commit, upgrading...
12:42:51.071321 trace git-lfs: Install hook: post-merge, force=false, path=/Users/rick/p/linux/.git/hooks/post-merge, upgrading...
12:42:51.071490 trace git-lfs: START: buffer=256
12:42:51.301909 trace git-lfs: END: queued=4051, run=374
WG_LIMIT=256 GIT_TRACE=1  track  1.25s user 0.37s system 622% cpu 0.260 total

The queued and run values track how many times it spawns a go routine, vs running inline. The sum of the two values should be the number of directories and subdirectories in your project.

If you could run that, it'd really help me out :)

EDIT: Feel free to sanitize the output too. The only lines I care about are the ones with START: or END:, and the time output. The trace messages above START: are irrelevant here, and I don't need to see what file patterns you're tracking with LFS.

@rudifa
Copy link
Author

rudifa commented Oct 15, 2017 •
edited

Hi @shiftkey & @technoweenie

I did not find the time yet to try out the modified file walker.

However, I could add several observations to clarify what I reported in Create a New Repository fails #3036, see @rudifa update 2017-10-15.

I also intend to submit suggestions for making the dialog Create a New Repository more informative.

@shiftkey shiftkey removed the bug Confirmed bugs or reports that are very likely to be bugs label Oct 16, 2017
@technoweenie
Copy link

@rudifa: Here's a posted build, based on an official PR for LFS: git-lfs/git-lfs#2672.

git-lfs-darwin-amd64-2.3.3.tar.gz

If you still haven't run the debug build above, I'd appreciate trying the above build on your repository. No special env vars or debug values necessary:

~p/linux git:(master)$ time ~p/git-lfs/bin/releases/darwin-amd64/git-lfs-2.3.3/git-lfs track
git lfs track  1.32s user 0.39s system 641% cpu 0.267 total

If you don't have time to run this, I totally understand :)

@rudifa
Copy link
Author

rudifa commented Oct 17, 2017 via email

@technoweenie
Copy link

@rudifa Thanks a lot! To explain the output:

START: buffer=2
END: queued=20, run=315

This run processed 20 groups of files in the 2 parallel goroutines. 315 groups of files were run sequentially so that it doesn't go over the max of 2 parallel goroutines.

13:46:29.529654 trace git-lfs: START: buffer=256
13:46:29.595044 trace git-lfs: END: queued=335, run=0

With a max limit of 256 parallel goroutines, it spawned new goroutines for 335 groups of files, never reaching the max.

Did you run this on the same repository? Those numbers don't indicate a huge repository, so I may be wrong about the original error that @shiftkey found.

runtime/cgo: pthread_create failed: Resource temporarily unavailable

Based on more googling, you could have run out of file descriptors, or hit some bug with some "cgo" code. LFS doesn't use cgo, but the Go runtime can use it for some things like DNS resolution.

At any rate, thanks so much for running my experimental code on your repository. I just merged a fix for this in git-lfs/git-lfs#2672, and will prepare a new release for the Desktop team this week. I hope it solves your issues :)

@shiftkey
Copy link
Member

#3078 (comment) also talks about using a folder with a large amount of files, which feels like the first issue you encountered. Once LFS is updated I'll close this out and address that error with the repro steps provided in #3078.

@shiftkey shiftkey mentioned this issue Oct 20, 2017
Merged
@lock lock bot locked as resolved and limited conversation to collaborators Jul 19, 2018
@desktop desktop unlocked this conversation Sep 27, 2018
@BenjamimCS BenjamimCS mentioned this issue Nov 23, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
more-info-needed The submitter needs to provide more information about the issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@technoweenie @rudifa @shiftkey