Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools: unify directory scanning code between gorename, goimports, etc? #16427

Open
bradfitz opened this Issue Jul 19, 2016 · 2 comments

Comments

Projects
None yet
3 participants
@bradfitz
Copy link
Member

bradfitz commented Jul 19, 2016

goimports has its own optimized directory scanning code.

Reportedly, gorename is quite a bit slower.

They should probably share code.

/cc @alandonovan

@thomasf

This comment has been minimized.

Copy link

thomasf commented Jul 19, 2016

It does take up to a minute for me to run gorename and the default output does
complain about some things as well. This is maybe not the same issue as the
general slowness one but it's maybe related.

I have a bunch of gcc sources and builds here and there under my src/GOPATH
which seems to confuse gorename a bit, maybe it has something todo with the
fact that the gcc sources includes the go compiler and tests (?)

A normal run of gorename spits out somewhere around a thousand lines like the
ones below this while running, there are also some errors about the tests of my
Go repository checkout which I have under src for reference. The gorename -v
argument doesn't seem to add any interesting additional information to let me
know what's going on.

Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/complex.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/complex.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/parallel": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/parallel/algo.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/ext/pb_ds/detail/bin_search_tree_": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/ext/pb_ds/detail/bin_search_tree_/bin_search_tree_.hpp: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/parallel": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/parallel/algo.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/backward": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/backward/auto_ptr.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/arm-none-eabi/bits": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/arm-none-eabi/bits/atomic_word.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/armv6-m/libstdc++-v3/include/arm-none-eabi/bits": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/armv6-m/libstdc++-v3/include/arm-none-eabi/bits/atomic_word.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/ext/pb_ds/detail/binary_heap_": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/ext/pb_ds/detail/binary_heap_/binary_heap_.hpp: no such file or directory.
...

EDIT: I temporarily moved the three gcc's which gorename was complaining about
out of my src directory and gorename finished a good portion of seconds earlier.

My ~/src has 1800000 files in total with each GCC installation having
up to ~19000 files each so they do represent a sizable portion of all my files in ~src (~30%).

(I'll revisit these numbers after I've had some sleep, a bit too tired for this atm)

@haya14busa

This comment has been minimized.

Copy link
Contributor

haya14busa commented Sep 6, 2016

Hi! Thank you for maintaining go and making goimports fast. I really like it 💓

For what it's worth, I created https://github.com/haya14busa/gopkgs
which just list all packages by using the same implementation as goimports.

In my environment ($GOPATH/src has 725045 files), go list ... takes about 10 seconds to list packages, but
gopkgs command with .goimportsignore takes under 1 sec.

go list ... is slow but gopkgs is fast enough for daily use and can use it with filtering tool or something (if you are interested in some usages, please see README).
I'm very happy with it.

I implemented gopkgs by coping and modifing https://godoc.org/golang.org/x/tools/imports to export some interface,
but it's not clever way.

So I'm really +1 for this issue.

I think it might be great to use the scanning code in go list as well if possible and
it's also great to provide the code as a public library.

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.