Wan 2.2 training and general discussion -- advice, ideas, questions #455
Replies: 6 comments 27 replies
|
I just started with Wan 2.2 myself! As for your questions, the only one I really have an answer for is 3 - and in my opinion it's a solid no. The final size of a LoRA is a function of it's rank/dimension and the number of parameters the base model has. Wan 2.2 is much larger than SDXL so a given rank LoRA is MUCH bigger and more capable. I personally almost always train Wan at rank 16/alpha 16, sometimes even less for simple character LoRA. A smaller rank is not only easier to share but plays nicer with other LoRA and can be less prone to learn parts of the training data you aren't targetting. With smaller models like SDXL or SD, bumping the rank was beneficial to increase the learning capacity but it's just not really necessary here. This applies to all the "larger" diffusion transformers like Wan 2.1/2.2, Hunyuan, Flux, etc. If you were using the 5B Wan 2.2 or the 1.3B Wan 2.1, then it can be beneficial, but not for the biggies! Of course, feel free to experiment! So anyway, I have 16GB of VRAM on my 4070 ti Super. My first foray last night, I used the same exact settings I'd been using for Wan 2.1 to train Wan 2.2 high: The training ran for about 8 hours, the optimized compile I'm working on really helps! Loss looks great, initial results are quite promising though unrefined without the low noise LoRA. I need to train the low noise model tonight to fully see how it turns out. A few of those option are unique to my Blissful Tuner(https://github.com/Sarania/blissful-tuner/), specifically I used multiple resolutions on the same video dataset, e.g.: For the low noise model tonight, I was thinking of removing the lower res 480x272x65f bucket and adding an 848x480x21f bucket to show a little more detail to it to. The idea is to play to the model's way of working, the high noise model decides overall structure and layout, the low noise model refines it into something aesthetically pleasing and adds fine details. I don't know if this is beneficial but I'm just following my intuition as always. That's where I am currently! I'll definitely come back to this thread and share more when I have more results, cheers! |
|
I'm actually kind of confused on how uniform frame extraction works. I read the docs on it repeatedly but never really understood it. If a clip is 81 frames long, captioned as one complete clip, and you have Does it break it into three chunks of ~33 frames? does it just train on the first 33 frames? If it's splitting it into sub-clips of 33 frames, would that not make the captioning incorrect for those chunks? |
|
Hello, I'm new to musubi tuner - installed it for WAN and still learning. So a newbie questions here: you use "--save_state_on_train_end". Have you sucessfuly resumed training for WAN? I did a training for 40 epochs and then I try to continue training with --resume option pointing to the "state" folder for another 40 epochs, however the final LoRAs generated almost identical results with the one trained for half the epochs. So either I'm doing something wrong or the script is not resuming training, but restarts. Will try to do some testing, but any info on resuming will be welcome. Also for other models if they resume sucessfuly. EDIT: I should have messed up something first time as the second and third attempts to resume were successful and produced the expected results. |
I think resumed training is currently bugged just based on observing loss and LR scheduling in original run vs resumed. It seems to not properly account for the total number of steps and thus assigns an incorrect (or in my case even inverse) LR schedule. I've swapped to simply starting from existing weights with no state using --network_weights "path/to/lora.safetensor" instead and manually setting the LR/optimizer params when I continue training. FWIW this may also be the case with sd_scripts backend in general and not only Wan2.2, as I observed similar issues when resuming from state for SDXL lora training. I however never dived deep enough into the issue to provide a good idea of what exactly is going wrong there. |
If you add --debug_mode video when preparing the cache you can see what it generates, unfortunately Uniform seems to just generate single frames so the videos are all a single extracted frame. I was hoping it would take a video and calculate how many frames to skip based on frame_sample and source_fps, instead it generates sequences of still frames which doesn't seem to work well for training motion. The good news is Musubi does turn the clips into 16fps which I didn't know before checking with debug_mode. If anyone can shed some light on achieving what I just mentioned please let me know. Otherwise I'm back to using Full as it doesn't seem like the other options are good for training anything but really simple motions and style & character training. |
|
So that run didn't really work out, I tried a new run last night where I did 3e-5 with loraplus of 4 for 1600 steps with shift of 12. This allows me to train high and low back to back in a single night, took about 12 hours! It's looking pretty good too, I did 360x360x65 and 512x512x33 for the high noise model, and for the low noise model I added in 640x640x21 to show some more detail. The rest was the same as above. It's looking promising and I'm especially pleased that I can train both high and low noise in a single night on 16GB! |
Hi there! Any news a possible fix of --optimized_compile :D BTW also schedulefree.RAdamScheduleFree give me terrible results, dont know if I should use this scheduler... |
|
Firstly I apologize, I didn't get a notification for when you posted the traceback somewhy but I did for this one so here I am! Secondly that's... weird. Seems above my pay grade so I asked AI to help me craft a fix. What version of Pytorch are you on fwiw? Anyway I've pushed this hopeful fix so please give it a try and let me know! Edit: FWIW this is because I want to compile a function for inference but not for training because it changes shape with the input shape so during training if you have more than one bucket it will cause the compiler to hit it's limit quickly. I guess I was trying to treat a method like a function but they aren't exactly the same, compiler can't set it's attrib on the method. |
Hi there ! No problem haha, Im using Pytorch 2.8, seems the error is now InductorError: AttributeError: type object 'CompiledKernel' has no attribute 'launch_enter_hook' . Also unfortunately I really have hard time to get good result with this config : accelerate launch --num_cpu_threads_per_process 1 --mixed_precision fp16 src/musubi_tuner/wan_train_network.py Dont know if the issue come from the fork or I just need to push to 40-50 epochs ( low noise config is same ) |
|
As far as your error, I'll be honest I have no idea. ChatGPT5 says it comes from a mismatch between your ondisk cached compiled modules and the version of pytorch you're using but I don't know if I buy that and it's misled me a lot lately(4 and o3/4 were a lot better IMO), it suggests clearing caches from As far as training Wan 2.2... I've never gotten completely satisfactory results in honesty, I've gotten like 90% of the way there. I don't think the issue should be from my fork but of course you're free to try the base repo and see. Wow I feel like my reply isn't very helpful, sorry! |
Thanks a lot, I think the issue also can come from the rtx 6000 pro Im using, it's a new gpu so it's maybe not compatible yet with torch compile.. anyway if you know others optimization with 96GB vram let me know. Thanks again. |
|
I don’t know what a typical configuration example for Schedule Free Optimizer is. I kind of understand that the other settings aren’t necessary, but what should I do about --learning_rate? |
|
Depends on the optimizer, the model, and other factors. There are many schedulefree optimizers. Which one specifically are you using? |
|
everyone training in fp16 but normally bf16 is much more accurate and working better anyone tested bf16? wan original models are fp32 so can be converted into bf16 i think |
|
Kinda butchered wan yesterday.... Settings: INFO:musubi_tuner.modules.fp8_optimization_utils:Number of optimized Linear layers: 300 Training time 30 minutes on a 5060ti, just a quick test to see we actually train anything. Result t2v ComfyUI_00436_.mp4ComfyUI_00437_.mp4Result i2v, 3/3 [00:17<00:00, 6.00s/it] Prompt executed in 82.83 seconds ComfyUI_00450_.mp4Used about 12gb ram and 9.8gb vram, so anyone know what settings i should actually be using? because i don't know what i'm doing lol. |
|
Regarding dual training vs. separate runs of low/high: Am I right in thinking that for some loras, you are better served with separate loras for each? If I use dual training with a flow shift of 1.0 (or sigmoid), it's amazing for detail/likeness of an object or character, but the high noise model is not really trained at all. On the other hand, if I choose a shift value of 3.0, the high noise model does get trained alongside the low, but the detail takes a noticeable hit. Even at 2.0 the detail is noticeably worsened. Character loras are fine at a shift of 1.0/sigmoid but it seems you'd get an even more refined lora if the high noise model is doing something too, especially if you're training with videos and teaching expressions & movements. So I think for loras where I need both detail and motion it probably makes more sense to train separate loras with the low noise model getting 70-80% of the training and then doing 20% or so of epochs on the high noise model. I have a lora in mind that is mostly motion, but also has to teach likeness of things the base model doesn't know and to get the best detail a shift of 1.0/sigmoid is needed, but I also want the high noise model to help with the motion and initial structure. It seems like I could also use dual training with a shift of 1.0 (high noise model almost ignored) then do another training with a much higher shift value targeting the high noise model (or manually specify timesteps) and I'd still get a single lora, but training will be slower due to the block swap I need to use for dual training. Anyone else got any thoughts about this or experience? Edit: Early experiments seem to indicate the separate training idea is indeed a better option for some concepts, for a character lora training low noise at 1.0 (for 1800 steps) and high noise at 5.0 (600 steps) resulted in the person's body type and face shape being slightly more faithfully represented, verified by toggling only the high noise lora on and off. A motion lora I just made turned out significantly better with the separate approach while requiring less training time vs. dual training. |
|
discrete flow shift set to 3.0 is significantly better for the high noise model with two motion loras I made, I'm not going to do dual training anymore! I also realised that single training having separate loras means I can probably drop rank, as dual training is using the same single rank lora for training both models which in theory would use capacity up quicker. For a t2v motion and identity lora, 1250 steps at 32/32 1e-4 cosine showed remarkable likeness and motion, I didn't think the high noise model would even be able to show such good likeness given the narrow stretch of timesteps. It's really impressive how fast it can represent both motion and detail despite still being so noisy. I probably don't need to let it finish its 2500 step run but I'll see how the final samples compare. It's also really nice not having to use any block swap on my 4090 24GB. Training is at least 30% faster per model. |
|
To my surprise I tried dropping from rank 32 to rank 8 for the same motion lora and I did not expect rank 8 to not only learn so quickly but deliver an exceptional result. I guess this is because WAN is such a good (and large) base model that it often only needs nudging so low ranks can be enough. I'm still changing parameters a lot, right now I'm liking shift 2.0 for low and 5.0 for high. |
|
Yeah I found early on with these large video diffusion models that rank 32 is unnecessary. The total amount of parameters in the LoRA is a function of it's Rank and the base model's number of parameters, so the bigger a model gets, the larger number of parameters a given rank of LoRA will contain. I usually start with Rank 16 which has been plenty for everything I've tried and lower ranks will play better when using multiple LoRA. I never got around to running tests for Ranks < 16 but it seems that even Rank 8 is enough in some cases based on your findings and that doesn't really surprise me and I imagine Rank 8 LoRA will play even nicer with each other than 16. |
The other thing I didn't think about until last week is that these ranks should be even more capable when you're creating a lora for each model, as the lora's capacity isn't being used for both models at once. The next thing I want to try is doing 1 pass at a higher flow shift and then a refinement pass at a lower one (for the low noise model). |
|
Absolutely yeah - in the same way that it works for the base model. Each one will specialize on it's part of the diffusion process instead of the full thing. I never tried training high/low with different flow shifts though, that's a good discovery. I discovered recently that if you do mixed image/video training it's beneficial to set different shifts for images versus video and added support for it to my fork. In that case it's for a different reason but I can definitely see how different values for low/high could be beneficial since they specialize on different ends of the timestep schedule. Especially in the case of video gen, the high noise model is responsible for core structure of both geometry AND motion, but by the time the low noise model is in play the motion and structure has largely been decided so it's focusing on mostly fine spatial detail, with a small amount of fine temporal detail too. Makes me suspect the high noise model might target removing lower frequency noise components and the low noise model might target removing higher frequency noise components. But that's a speculation XD |
Uh oh!
There was an error while loading. Please reload this page.
Advice and general discussion is appreciated, both for the config provided and for general discussion! Hopefully we can share knowledge here to improve everyone's runs.
A few questions to start a discussion:
--timestep_sampling shift --discrete_flow_shift 12.0?I'll start by sharing my current config, just doing a test run on images only on t2v 14B high noise. This is with a 24gb VRAM card, running on Linux mint. The images are batch size 4 with resolutions [768,768]. I managed to squeeze it all to fit into VRAM by running
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Truein the terminal (only works on Linux I think).All reactions