Repository navigation
Basalt PR (Cap Prefill Chunks at 2048 on 16GB GPUs)
Pre-release
Pre-release
Test build for capping prefill chunks at 2048 tokens on <=16 GiB GPUs to avoid prompt-read OOMs and driver paging.
Test build for capping prefill chunks at 2048 tokens on <=16 GiB GPUs to avoid prompt-read OOMs and driver paging.