[CONTRIB] Add PopenWorker process recycling

[petersalas]

Background processes (e.g. builds during autotuning) can sometimes accumulate state or leak memory. This adds the ability for PopenWorker processes to optionally be recycled after a specified number of uses.

[AndrewZhaoLuo]

Haha, I think this is a parsimonious (but good) workaround to the issue of memory leaks in some uses of PopenWorker. It would be cool if we could open an issue with an example B) if you have time.

[AndrewZhaoLuo]

Wonder what others think though? @shingjan ?

[zxybazh]

Very interesting, we observe the same behaviour with PopenWorker during tuning and our workaround is also very similar. Would be happy to learn about how you guys figured out the problem and do you have any trace on the root cause or potential fix. A minimum reproducible example would also be appreciated!

[shingjan]

Thanks for sending in this fix! Memory leak around PopenPool has been an issue identified in #11074. Does this fix imply that PopenExecutor underlying process will be a zombie process if used too many times? If so what is the best heuristic number for max_number_use?

[junrushao]

That's exactly one issue that we recently identified and fixed in #11074. Thank you so much for this fix! Also CC @tqchen as the original author of PopenWorker

[AndrewZhaoLuo]

Do we have an existing issue for the memory leak stuff? I think based on some internal data we suspect it's in tvm.lower somewhere.

[zxybazh]

Just created one #11096.

[petersalas]

@shingjan to be clear this change alone isn't enough to mitigate the leaks because the default is "never recycle" unless operations timeout. But specifying a value when initializing PopenPoolExecutor ensures that the background processes are killed periodically which prevents any per-operation leaks from accumulating too much. I used 32 in my test, which worked fine.

Of course, we should also try to fix the leaks :)

[AndrewZhaoLuo]

Any objections? Otherwise will merge by eod

[junrushao]

LGTM too! BTW, if maximum_uses is set, while the process is run async (e.g. MetaSchedule RPCRunner), we might want some logic to make sure synchronization happens before recycling the child processes, so that running results are not lost, which is beyond the scope of this PR

[junrushao]

Thanks @petersalas for bringing this workaround, and @AndrewZhaoLuo @shingjan @zxybazh for review and discussion :-))