Slow build speed with punica after #1804

[zhuohan123]

After #1804, the build speed of vLLM becomes significantly slower. Most of the time was spent on /home/zhuohan/vllm/vllm/csrc/punica/bgmv/bgmv_all.cu. Should we turn off punica by default?

cc @Yard1

[Yard1]

Should we make it depend on an env var that can be set when calling setup.py/pip install?

[njhill]

Duplicated by #2604 and now resolved by #2605?

[Yard1]

Yes!