listing local repositories is very slow since recent changes

[thomasf]

It take about 5 seconds now:

real    0m5.194s
user    0m2.436s
sys 0m2.172s

360 entries listed

I quickly tried using https://github.com/MichaelTJones/walk which helps a bit:

real    0m1.755s
user    0m3.208s
sys 0m2.064s

360 entries listed

Reverting back to 53c4cae (bump version to 0.7.3) the speed is up again:

real    0m0.028s
user    0m0.004s
sys 0m0.012s

355 entries listed

For most situations I rather go back to the old behaviour in most situations if it isn't possible to solve this somewhere else.. Maybe adding an option to stop traversal is a good idea?

I also tried using MichaelTJones/walk on 53c4cae but that only listed one repository so it's not working well there.. I havent had time to really read through the changes now so I'll be using 53c4cae as a temporary workaround.

[thomasf]

I'm runnig these tests on a normal speed SSD disk.

Heres an additional timed iteration using find:

time find . | wc -l

657086

real    0m1.080s
user    0m0.388s
sys 0m0.736s

[stefan-pdx]

I find that this also extends to the ghq look command.

---

Edit: Oh, this is because this command kicks off a new shell, loading my lengthy .zshrc config file. Disregard!

[motemen]

I guess that the PR #66 could make ghq list slower, but you are saying reverting to 0.7.3 does speed up, which should include #66... I cannot see the changes affecting speed between 0.7.3 and 0.7.4 (if you don't set GHQ_ROOT env variable).

• What version of ghq are you using and how do you build it?
• Could you give any information about the extra 5 directories in the output of the newer version? (where they are located, you are expecting them to be appeared or not, etc.)

[thomasf]

• My local repo is still parked at revision 53c4cae.
• I build it using the makefile
• By looking at the commit log it looks like the problem is related to Revert "Merge pull request #54 from maoe/skip-non-vcs-dirs" #66.

The extra directories in the newer version is because some repositories has other repositories in them which were skipped before #66 and I don't care for them since those are "owned" by their parent repository anyway.

[thomasf]

New numbers, recompiled binaries with go162 and now running latest ubuntu 16.04 kernel.

All the files:

$ time find . | wc -l
971769

real    0m1.787s
user    0m0.640s
sys 0m1.212s

recompiled 53c4cae

$ time ghq list | wc -l
DEPRECATED Action signature.  Must be `cli.ActionFunc`.  This is an error in the application.  Please contact the distributor of this application if this is not you.  See https://github.com/urfave/cli/blob/master/CHANGELOG.md#deprecated-cli-app-action-signature
490

real    0m0.205s
user    0m0.084s
sys 0m0.104s

compiled from master

$ time ghq list | wc -l
496

real    0m11.759s
user    0m4.488s
sys 0m5.896s

Compiled from 3464624 (the 0.7.4 tag) which also works great

$ time ghq list | wc -l
DEPRECATED Action signature.  Must be `cli.ActionFunc`.  This is an error in the application.  Please contact the distributor of this application if this is not you.  See https://github.com/urfave/cli/blob/master/CHANGELOG.md#deprecated-cli-app-action-signature
490

real    0m0.219s
user    0m0.084s
sys 0m0.112s

[motemen]

Thank you for the detailed information. Now I'm wondering if I should add an option to use the old behavior reverted by #66.

Let me make things clearer, as for "some repositories has other repositories in them which were skipped", you mean the layout look like below, where you don't need the "bar" directory? I don't think this could happen since if the parent directory was found to be a repository, then its children won't be visited (https://github.com/motemen/ghq/blob/c087d82/local_repository.go#L169). Are there anything I am missing?

GHQ_ROOT/
|-- baz
|   `-- .git
`-- foo
    |-- .git
    `-- bar
        `-- .git

[rcoedo]

I use ghq with peco and in it's current state it is unusable. I have 84 repositories and it takes ~10 seconds to list them. What's the last version that I should install which doesn't include the new behavior?

Edit: downgrading to 0.7.0 and now it takes 0.10 seconds. yay!

[thomasf]

I guess that this might have been fixed now? Unless I'm unaware that I use my own patched version of ghq it probably is fixed or just much better now.