x/tools: unify directory scanning code between gorename, goimports, etc? #16427

Open
bradfitz opened this Issue Jul 19, 2016 · 2 comments

Projects

None yet

3 participants

@bradfitz
Member
bradfitz commented Jul 19, 2016 edited

goimports has its own optimized directory scanning code.

Reportedly, gorename is quite a bit slower.

They should probably share code.

/cc @alandonovan

@bradfitz bradfitz added this to the Unreleased milestone Jul 19, 2016
@thomasf
thomasf commented Jul 19, 2016 edited

It does take up to a minute for me to run gorename and the default output does
complain about some things as well. This is maybe not the same issue as the
general slowness one but it's maybe related.

I have a bunch of gcc sources and builds here and there under my src/GOPATH
which seems to confuse gorename a bit, maybe it has something todo with the
fact that the gcc sources includes the go compiler and tests (?)

A normal run of gorename spits out somewhere around a thousand lines like the
ones below this while running, there are also some errors about the tests of my
Go repository checkout which I have under src for reference. The gorename -v
argument doesn't seem to add any interesting additional information to let me
know what's going on.

Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/complex.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/complex.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/parallel": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/parallel/algo.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/ext/pb_ds/detail/bin_search_tree_": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/ext/pb_ds/detail/bin_search_tree_/bin_search_tree_.hpp: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/parallel": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/parallel/algo.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/backward": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/backward/auto_ptr.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/arm-none-eabi/bits": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/arm-none-eabi/bits/atomic_word.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/armv6-m/libstdc++-v3/include/arm-none-eabi/bits": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/armv6-m/libstdc++-v3/include/arm-none-eabi/bits/atomic_word.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/ext/pb_ds/detail/binary_heap_": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/ext/pb_ds/detail/binary_heap_/binary_heap_.hpp: no such file or directory.
...

EDIT: I temporarily moved the three gcc's which gorename was complaining about
out of my src directory and gorename finished a good portion of seconds earlier.

My ~/src has 1800000 files in total with each GCC installation having
up to ~19000 files each so they do represent a sizable portion of all my files in ~src (~30%).

(I'll revisit these numbers after I've had some sleep, a bit too tired for this atm)

@haya14busa
haya14busa commented Sep 6, 2016 edited

Hi! Thank you for maintaining go and making goimports fast. I really like it 💓

For what it's worth, I created https://github.com/haya14busa/gopkgs
which just list all packages by using the same implementation as goimports.

In my environment ($GOPATH/src has 725045 files), go list ... takes about 10 seconds to list packages, but
gopkgs command with .goimportsignore takes under 1 sec.

go list ... is slow but gopkgs is fast enough for daily use and can use it with filtering tool or something (if you are interested in some usages, please see README).
I'm very happy with it.

I implemented gopkgs by coping and modifing https://godoc.org/golang.org/x/tools/imports to export some interface,
but it's not clever way.

So I'm really +1 for this issue.

I think it might be great to use the scanning code in go list as well if possible and
it's also great to provide the code as a public library.

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment