Add PagedAdamW32bit

[xzuyn]

No description provided.

[kohya-ss]

Thank you for this! I didn't know PagedAdamW32bit.

[oliverban]

This is great, but isn't the 32bit part still expensive in terms of memoy? Or is this different for images? I really like this way of optimization but if we can get a PagedAdamW8Bit, wouldn't that be even better? I read that but they were talking about LLMs.

[xzuyn]

This is great, but isn't the 32bit part still expensive in terms of memory?

In terms of memory usage it's more expensive than using 8bit, but since its Paged that memory isn't eating into GPU VRAM, only system RAM. So if you have the RAM for it, using 32bit is higher precision. No idea how much that translates into real world quality though.

I really like this way of optimization but if we can get a PagedAdamW8Bit, wouldn't that be even better?

It's already implemented. I added PagedAdamW32bit here since only PagedAdamW8bit was added.

There may also be a 16bit paged version just named PagedAdamW but that may just be an alias for PagedAdamW32bit or PagedAdamW8bit. I'll have to check into this later.

edit: From what it seems, PagedAdamW is like AdamW so I will do another PR shortly after I confirm it runs.

[oliverban]

This is great, but isn't the 32bit part still expensive in terms of memory?

In terms of memory usage it's more expensive than using 8bit, but since its Paged that memory isn't eating into GPU VRAM, only system RAM. So if you have the RAM for it, using 32bit is higher precision. No idea how much that translates into real world quality though.

I really like this way of optimization but if we can get a PagedAdamW8Bit, wouldn't that be even better?

It's already implemented. I added PagedAdamW32bit here since only PagedAdamW8bit was added.

There may also be a 16bit paged version just named PagedAdamW but that may just be an alias for PagedAdamW32bit or PagedAdamW8bit. I'll have to check into this later.

edit: From what it seems, PagedAdamW is like AdamW so I will do another PR shortly after I confirm it runs.

Great news! I have 64 GB system RAM so should easily fit! Thanks for looking into this! <3