Mixed Precision Merging

[sais-github]

I am curious about whether models used for merging are converted to the set dtype before merging, or if it is only the resulting merge being converted.
And if there is any loss when merging a fp16 and bf16 model?

[cg123]

When you specify a dtype everything is converted to the specified dtype before merging, yeah.

Merging between fp16 and bf16 is potentially lossy, as both data types are capable of representing values that the other is not. This generally isn't going to be a meaningful difference though. If you're working with something super numerically sensitive, I'd recommend upcasting to fp32. I haven't found a use case that needs that kind of precision yet personally.