Reduce the amount of goroutines used for builds#126
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.50.
| Benchmark suite | Current: 6271adf | Previous: d803417 | Ratio |
|---|---|---|---|
BenchmarkWorkspaceCycle (typefox.dev/fastbelt/examples/statemachine) - MB/s |
13.28 MB/s |
5.21 MB/s |
2.55 |
This comment was automatically generated by workflow using github-action-benchmark.
|
Ignore comment above by the git-bot. It doesn't understand that 11MB/s is better than 5MB/s 🙄 |
Lotes
left a comment
Lotes
left a comment
There was a problem hiding this comment.
Can you adjust the performance check in that regard that improvements of the speed are not flagged?
On the other hand this can also be an indicator that important code was skipped. But actually then the tests should also fail...
I added some comments about the parallelization module.
sailingKieler
left a comment
sailingKieler
left a comment
There was a problem hiding this comment.
Nice improvement @msujew 👍
I have some remarks and suggestions, see below.
|
FYI I've also introduced a |
sailingKieler
left a comment
sailingKieler
left a comment
There was a problem hiding this comment.
Great, good to go from my point of view! 🚀
|
... don't forget to update the title/final commit msg 😉 |
3 participants
Massively reduces the amount of spawned goroutines required for a workspace build. I noticed that during large workspace builds, we spawn an unreasonable number of goroutines, which (a) drags down our memory profile and (b) slows down runtime performance when multithreading (but also when single-threading!).
This change adds a new
parallel.ForEachIter(iter.Seq[T]) andparallel.ForEach([]T) for efficient chunking of data for goroutines. This is now used for both the linking and general workspace build phase (essentially all steps where we usesync.WaitGroup), which results in a total speedup of roughly 40%, and a 20% memory usage decrease.