Recycle goroutine doing compaction

[cyriltovena]

While working on #118 I realised that we create a go-routine per compaction for each granule.

I think this should be bounded but also we should probably recycle go-routine since those can grow pretty large in term of memory

I suggest we create 8 go-routines that will take care of compactions, then send granule to compact via a buffered channel of 32, meaning it can block on the write path.

[brancz]

I think it's important that we keep the write path non-blocking/lock-free, so I wonder if we could find another way. Maybe we could just mark the granule to need compaction and have the compaction go routines just cycle through the granules to pick up the ones that need compaction? Happy for other suggestions but not spawning a go routine at every insert is obviously not a long term option (I think we always knew this but just punted on that when we initially wrote this).

[thorfour]

Yea, when I initially wrote that I played around with buffered channels as well as a mark and sweep style. This one performed the best, and until we better understood the characteristics of how it performed decided to defer solving this.

[brancz]

Potentially wild idea, what if we used a sync pool for reusing goroutines?

[brancz]

Never mind we would end up leaking goroutines.