We can further reduce quantization time (substantial) by minimizing the amount of `forward` we need to do.