Quantization produces non-deterministic weights

[MarkSchmidty]

Below is a segment of the 7B 4bit weights generated using the line in the same environment with two different video cards. An A4000 (on the left) and an A6000 (on the right).

Notice how every 20-40bytes there is a half byte difference? These differences are always off by one, a B becomes an A and a 5 becomes a 6 etc. This issue seems to persist across all model sizes when producing weights on different cards.

No idea what is causing it.

Without reproducible builds it is hard to say if we're actually producing the same weights.

[qwopqwop200]

related to this issue.(IST-DASLab/gptq#1)

[MarkSchmidty]

@qwopqwop200 is it possible the CUDA_VISIBLE_DEVICES variable is somehow being used somewhere in the quant code where it shouldn't be? I see no references to it. But the only difference between the two models above is one was generated with CUDA_VISIBLE_DEVICES=0 and the other with CUDA_VISIBLE_DEVICES=1

[qwopqwop200]

That doesn't seem to happen. Also, this seems to be caused by different GPUs.
The difference in performance caused by these differences is negligible.