grepping for the word "semaphore" reveals five separate I/O semaphores in x/tools. Some of them could easily be used in the same executable, e.g. those in
I suggest we add a single exported semaphore to x/tools, which all x/tools packages can import and use, thus providing an actual process-wide limit on I/O concurrency. It could be in an internal package or not; I don't feel strongly.
The text was updated successfully, but these errors were encountered:
Great, will do. Anyone want to bikeshed API details before I send a CL? For lack of better names, perhaps:
Allowed concurrency would be fixed at 20.
Usage would look like:
iosem.Acquire() // do work iosem.Release()
This change parallelizes the outer loop in globalReferrersPkgLevel, which loops over packages to inspect. There is also an easily parallelizable inner loop. However, parallelizing it adds complication (deffiles needs a mutex, inQueryPackage requires a wait group) and offers only a 2% speed-up. Benchmarks for this change, looking for encoding/json.MarshalIndent: name old time/op new time/op delta Referrers 5.31s ± 2% 4.67s ± 3% -11.95% (p=0.000 n=10+10) name old user-time/op new user-time/op delta Referrers 15.9s ± 2% 16.5s ± 3% +3.71% (p=0.000 n=10+10) name old sys-time/op new sys-time/op delta Referrers 15.7s ± 3% 16.1s ± 3% +2.73% (p=0.011 n=10+10) Fixes golang/go#24272 Updates golang/go#25017 This work supported by Sourcegraph. Change-Id: I5dcda9017103cdff59d0ffdf5e87d2c2c955a33a Reviewed-on: https://go-review.googlesource.com/108878 Run-TryBot: Josh Bleecher Snyder <email@example.com> TryBot-Result: Gobot Gobot <firstname.lastname@example.org> Reviewed-by: Alan Donovan <email@example.com>