Skip to content

Basalt PR (Cap Prefill Chunks at 2048 on 16GB GPUs)

Pre-release
Pre-release

Choose a tag to compare

@cfelicio cfelicio released this 11 Nov 23:07

Test build for capping prefill chunks at 2048 tokens on <=16 GiB GPUs to avoid prompt-read OOMs and driver paging.