[Bug]: When EMA is enabled it doesn't learn anything is this expected? SDXL Base model

[FurkanGozukara]

What happened?

with EMA on 0 learning. Absolutely same settings except EMA on and off

EMA off

150 epoch working as expected

EMA on GPU
Decay 0.999
Update interval 1

150 epoch

200 epoch

[dxqb]

How are you judging that it learned nothing, by external sampling of a checkpoint or by OT sampling?

If external:
you can enable non-EMA sampling in OT and see the same sample with EMA and without EMA applied. If the non-EMA sample does learn but your EMA-sample does not, your EMA parameters are too slow.

[FurkanGozukara]

@dxqbYD I am checking from saved checkpoints. I am using SwarmUI (ComfyUI backend), I compared both saved checkpoints and also final checkpoint

As you can see 150 epoch and 200 epoch are literally same and character is nothing of my training dataset

Unless there is no additional option that I have to check than these 2, this is the current status of a fresh new OneTrainer installation

The only different option between regular working training and EMA is below setting nothing else

GPU vs off

[FurkanGozukara]

@dxqbYD here my EMA preset

config.json

[dxqb]

as I said above I recommend using OT sampling because you can see both, EMA and non-EMA samples
if you train on a single subject I don't see the point in using EMA at all. it'll just slow down your training.

[FurkanGozukara]

as I said above I recommend using OT sampling because you can see both, EMA and non-EMA samples if you train on a single subject I don't see the point in using EMA at all. it'll just slow down your training.

well if it doesnt work at the end generated product (saved safetensors checkpoints) i dont see how i can count it working even if working with samples during generation

also i had tested EMA previously on only a single subject and it was slightly improving but currently it didnt train at all like before, it is not like it didnt improve or benefited

[dxqb]

well if it doesnt work at the end generated product (saved safetensors checkpoints) i dont see how i can count it working even if working with samples during generation

it will not.
it will show you whether the base model is not learning, or just the EMA interpolation is not working or has the wrong parameters - by generating EMA samples and non-EMA samples which you can compare.

[FurkanGozukara]

well if it doesnt work at the end generated product (saved safetensors checkpoints) i dont see how i can count it working even if working with samples during generation

it will not. it will show you whether the base model is not learning, or just the EMA interpolation is not working or has the wrong parameters - by generating EMA samples and non-EMA samples which you can compare.

ok doing a fresh training right now will post here

[FurkanGozukara]

@dxqbYD yes proven it is not working

these 3 below are with EMA

non ema samples are accurate and learning

epoch 0

20

30

now non EMA samples

0

20

30

[Heasterian]

Isn't your EMA decay of 0.999 just way to high for dataset of just 28 images? Use 0.99 or even 0.9 or you will see results much later than after 30 epoch...

[dxqb]

@dxqbYD yes proven it is not working

I think you might have shown that it is working. Model learns, it's just not transferred to the EMA model.

• your EMA is slow. 0.999 means each step 1/1000th of the current difference is added to the EMA model. It takes you thousands of steps to affect the EMA model even at perfect precision
• you are using BF16 weights, which is very bad at adding up small numbers. you might even run into precision issues with 0.999 and really learn nothing at all in the EMA model

I'll change this into a github Discussion because this is likely a matter of parameter tuning.

[FurkanGozukara]

just to let you know model was trained 5600+5600 (subject + reg images) steps and it still didnt have any impact

but i am testing with EMA decay 0.9 and EMA 0.99 right now to test

[HarborCoke]

@FurkanGozukara can you share your loss graph, smooth loss and loss.