-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Appeal for the Separation of SD 1.5 from SDXL #1401
Comments
Let's agree on the last good commit and I will create a SD1.5 branch from it. Then we can work out why it will not run properly as in theory it should. It might come down to some gradio version for the UI. That release will not see any further development but will allow to keep what used to work in SD1.5 functional. Obviously if people contribute PR for the 1.5 branch I will merge them if it make sense... |
I agree with you on this point. The problem is to determine which version was functioning correctly. I will give you feedback tomorrow. My colleague and I will verify the logs and the journals where everything went perfectly fine. We will confirm with you if the version that we believe is indeed the version: https://github.com/bmaltais/kohya_ss/releases/tag/v21.7.6 I will come back tomorrow to inform you about this without fail. Thank you for your response. |
Hello @bmaltais , I'm reaching out to you again, as indicated, to provide you with the version that has given me slight results so far. I've tested the following versions: from version 21.6.5 to version v21.7.8. The only one that yielded a result of about 60% is this version. https://github.com/bmaltais/kohya_ss/releases/tag/v21.7.8 However, there are clearly numerous illogical anomalies. I conducted the following tests: 20 identical images. Thank you for your feedback. I'm available for any tests or assistance that I can provide. Best regards. |
Well, if the version work to generate the models I can't really do much about the results. If you find a version that produce the results you used to have I can create a branch from it... but personally aI have never had issues with the models produced from any of the versions... so it is hard to troubleshoot. :-( But for now I can create the sd2.5 branch from the 21.7.8 release so it can be used as a fundation. |
@bmaltais how its a SD 2.5 branch? |
Typo. I have published the code in the sd15 branch. I also updated the gradio release so it does not cause issue with the browsers. |
@bmaltais |
If if find the release that work let me know. The code in the release is locked and should produce consistent results. Drivers update on the other hand have been known to cause training variations. It might be possible that new drivers are now used vs the ones that were a few months ago? |
@bmaltais |
Version v21.5.11 can you please tell me how to modify this version to carry out tests on it please? there are 2 different gratio, thank you: version ftfy==6.1.1 |
I modified the versions so that it is installed which and ok, bitsandbytes==0.35.0; sys_platform == 'win32' for BLIP captioningfairscale==0.4.13 tensorflow<2.11huggingface-hub>=0.14.0; sys_platform != 'darwin' For locon supportlycoris_lora==0.1.4 for kohya_ss library |
@XT-404 Let me know how this config goes. |
@bmaltais I must be at around 150 to 200 tests carried out since yesterday currently I have 4 versions that stand out I try to obtain a 75 / 80% image fidelity with different parameters let it be 20 frames / 30 / 40 / 50 / 100 / 1000 I do my best to see the best targeted among the 4 remaining |
After 2 whole days of hard testing of configuration ect , I finally found a version that reaches 80% with parameters not too hardcore worked, however Cudnn should not be installed as bytsandbit, , without it's two there the program works. accelerate==0.18.0 remove the * bitsandbytes==0.35.0; sys_platform == 'win32' This time I can confirm that this version is viable at 80% let it be under 20 frames, 50, 100, 1000. all that remains is to work on the code to obtain a success rate of 95/100% and all will be perfect;) PS: the graphics drivers had no impact on training, I used the old driver versions and the new ones, it did not change the training percentage. |
Now the community has hot talents or Python developers to contribute to the building to perfect our dear friend Kohya_ss 21.5.11 to be perfect on this training ^^. |
Adding a sidenote here that I also experienced complete breakage of anything 1.5 training when SDXL stuff was added but I managed to solve it by re-installing CUDA 11.8 (also noting that current display driver is 531.61). Non-functioning or wobbly LoRAs were a problem for weeks but this was the thing that 'repaired' the training. Took several hours to debug, that for some (whatever) reason, xformers was borking the entire training - if I trained without it, everything was more or less correct. I traced it back to CUDA not 'connecting' to the training session at all (if that is even a good way to describe it). After re-installing CUDA 11.8, training speed increased tremendously (going from ~4.5s/it ---> ~2.13s/it), so that was a further indication that something was borked badly. Not sure if it will help anyone else, just thought I'd mention this. 07:23:01-369448 INFO Version: v21.8.7 07:23:01-390391 INFO nVidia toolkit detected |
Thank you @Loadus for sharing your experience... SO insummary folks with issues should re-install CUDA 11.8 and make sure thay use NVidia drivers 531.61. |
Direct link to 531.61 drivers: Game Ready Driver Download Link: https://us.download.nvidia.com/Windows/531.61/531.61-desktop-win10-win11-64bit-international-dch-whql.exe Studio Driver Download Link: https://us.download.nvidia.com/Windows/531.61/531.61-notebook-win10-win11-64bit-international-nsd-dch-whql.exe |
@bmaltais I of course use the indicated driver CUDA 11.8 and NVidia 531.61 drivers. |
great findings by @XT-404 and @Loadus. Have you figured out why the steps have been increased to 3000 from 1000 and why the realistic images are hard to achieve? |
"After several weeks of intensive testing, Training on realism does not work at all. For a reason I can't understand, training on manga/comics/BD/drawing/3D/2D works perfectly in LORA. Realism, on the other hand, is completely shattered. Why are training steps above 1000 and jump to 3000? No idea. I tried to get closer to the most functional with 20 images, and only 3000 steps work. If I'm below that, I get under-training and if I go above, it burns the training (I'm under 4090). I don't have a slowness problem and the training remains at 2.5 or 2 without significant loss. However, regarding the design of Checkpoints, it's not even worth mentioning: nothing works. I can put any type, all the CPs made from 3000 steps to 10K steps and others come out in confetti mode or completely blown up. The only thing currently working on my side, whether on this version or version 21.8.8, is the creation of Lora manga, comics, bd, and nothing else. I tried a series over several days of parameter modification, installation of old drivers, etc., to no avail. Many people have given up on the idea of designing Lora or CP given the disastrous results obtained. Personally, I'm not giving up, but I'm also tired of these utterly disastrous results and that nothing is found to set things right. Being forced to use SDXL while many people refuse this version is really a punishment for us." |
For information, the latest version that correctly teaches on realism for AMD 6000 cards line -is -21.5.8, everything above is horror -) |
Totally agree it's literally impossible to get consistent LoRA results with recent versions of kohya_ss. The only version that has been working pretty good for me is the collab version https://colab.research.google.com/github/hollowstrawberry/kohya-colab/blob/main/Lora_Trainer.ipynb which in turn uses https://github.com/kohya-ss/sd-scripts. Both commit e6ad3cb and 9a67e0d seems to work fine. Just sharing if anyone needs to try. |
You can try this method, it works for me regardless of the version: Please note that due to version updates, the line numbers may not be consistent, but the modified code is consistent |
Sorry, I have been away... and last time I was here, missed this entire issue somehow. Previously, I spent a lot of time looking into this issue and managed to find a way to reliably reproduce the "old" LORA behavior as described here: #1291 (comment) - though the results are not exactly the same as I got previously. I can't agree with some of the comments about this only being a problem for realism. I've had all these issues trying to train and retrain a cartoony LORA, but perhaps that is because I am using NAI. Maybe I don't know what I am doing and have less margin of error. Another observation I had about this matter is that the newer Kohya gives more reliable, but worse results. The old one gives much less stable results, but some of them are high quality. Put another way: 'new' is almost all poor quality images, 'old' is mostly awful but some great images. |
Is Kohya_ss effectively over for non SDXL creators? has there been a fork or a dedicated project since then? I am curious how people are making loras now |
Kohya_ss the author of the sd-scripts code base I use in this repo is not maintaining an sd1.5 branch… so I guess this is pretty much the end of the sd1.5 only code base. His code should support both sd1.5 and SDXL but some of the new modules required may not produce the same sd1.5 results it used to. I suggest you raise this concern directly with him on his sd-scripts repo. |
Hello everyone, It's been a while since I've posted in this topic, which I created due to multiple anomalies related to the Kohya_ss script. I'd like to clarify, as bmaltais mentioned, that he is not the original author of this script. Instead, he uses the independently developed Kohya_ss script. Since my last post, I have achieved a lot. After several months of testing, I've noticed significant changes with the integration of SDXL into the SD1.5 Kohya_ss script. I've chosen to focus on version v21.8.10, which allows me, in 90% of cases, to create various types of Lora, in terms of style or concept. However, one issue persists since the addition of SDXL: the realism of characters, known or not, with any training checkpoint model. To overcome this challenge, I developed a specific Checkpoint that excludes images of the Cartoon/2D/3D/ANIME/MANGA/2.5D type. By training the lora with realistic images, they are transformed into BD/COMICS/Cartoon versions, etc. The Checkpoint I created then transforms these 2D/anime images back into realistic versions. To date, this is the only method I've found to achieve pure realism with functional lora images. I've also experimented with other training systems that have yielded similar results to Bmaltais's code. These systems all seem to be based on the same developer, the creator of the Kohya_ss script. Currently, I am not aware of any ongoing project aiming to develop a script similar to Bmaltais's for SD1.5 users who prefer to stay on this version. Unless a talented developer like Bmaltais embarks on such a project, it seems that the only alternative is to stick to functional older versions and block updates. Best regards, |
Have you tried the modification method I mentioned before? The latest version of Kohya_ss has basically fixed the SD1.5 problem, and the only remaining issue is the reproducibility of the loss function. Because Kohya_ss corrected the SD1.5 problem, but at the same time modified some references and loaded VAE into xformess, this caused subsequent versions to still be trainable but the loss function is different from before. To restore the exact same loss function, just follow my modification method. The evidence lies in the fact that I trained with the old version of Kohya_ss, trained with the latest version of Kohya_ss, and made the code modifications I mentioned. The lora trained by both under the same seed are almost identical in action and appearance. |
Hello @AIEXAAA After applying the suggested method in the comment on the GitHub topic (kohya-ss/sd-scripts#855 (comment)), I encountered several technical difficulties.
In conclusion, despite following the instructions scrupulously and checking for potential manipulation errors, the modified script does not function correctly. Reinstalling the script in its original version restored its operation, but the problem of realism remains unresolved. I remain open to any further suggestions or assistance to rectify these issues. |
It might be a translation issue, I’m somewhat unclear about your response. Are you saying that when you make modifications according to the second point, your program throws an error? If so, the most likely reason is that your version of Kohya_ss is not up-to-date. In one version of Kohya_ss, when the aforementioned two lines of code are removed, the GPU’s RAM usage becomes huge, leading to an error. The latest version of Kohya_ss has already fixed this. If it’s not a program error, but you still can’t reproduce lora after the modification, then this is beyond what I can explain. |
@AIEXAAA
launching the training crashes automatically unless of course I reinstall it as originally |
I dare not make a definitive statement here, but as you can see from the code, if the PyTorch version is too low, it will not load. Therefore, even if you remove this section of code, there should be no problem. Because after removal, it’s as if your PyTorch version is too low. So, I’m puzzled by your results. Additionally, changing
to
actually has the same effect. This way, you don’t need to reinstall it, and if it can’t run, you can directly change it back. |
Spent 3 days trying to train a 1.5 checkpoint, only to find out it doesn't work on 1.5 3 days wasted, |
Greeting @OriginLive |
I was running latest. What version are you suggesting to use? What needs to be done? I've tried a 27 release before sdxl was mentioned but i can't use ui there with the current python version |
For old versions before the insertion of SDXL there are modifications to be made in a Python script file that @bmaltais the 0.5% absent is linked to direct Realism which does not work on any type of training and configuration or checkpoint. |
What changes, there's drivers mentioned and all sorts of stuff like a different branch? Could you help out a bit more, i'm trying to get 1.5 working |
Make it very professional and detailed in explanations: Simply install the version without doing any updates, In the settings section, I proceed as follows:
That's a setting I've done on my side and it works perfectly, except as said for realism where the training fails at 99.8%. |
@OriginLive |
Go DL and install : https://developer.nvidia.com/cuda-11-6-0-download-archive |
`============================================================= Modules installed outside the virtual environment were found. You can uninstall all local modules with: ============================================================= 20:25:07-691269 INFO Version: v21.8.10 20:25:07-694779 INFO nVidia toolkit detected Leme try w/ an old driver |
I installed the old drives, but it still says 11.8 for cuda :/ even though 11.6 was installed and the old drivers were installed as well |
@OriginLive |
I cannot, i do not see it in the list of available programs. Maybe it's part of pytoch? edit: https://discord.gg/ySHHDKkhat |
Ok, i think i figured it out, i was using 1e-4 for learning rate of Dreambooth, but 1e-6 or 1e-5 works better and doesn't produce the mosaic above, as finetuning would want a lower LR than LoRA since there’s a lot more weights that need adjusting and a higher LR would be much more destructive. I think 🤔 It works fine for me on latest cuda, drivers and latest kohya ss |
Hello, @bmaltais
Considering the critical situation of SD 1.5 content creators, which has been severely impacted since the SDXL update, shattering any feasible Lora or CP designs,
We are requesting that SD 1.5 be separated from SDXL in order to continue designing and creating our CPs or Loras.
Many of us, including myself, have invested significant amounts of money to passionately create quality Checkpoints and Loras.
I now find myself completely handicapped and unable to design even a functional and worthy Lora or Checkpoint.
Numerous individuals in the community, whether in France or the United States, are suffering due to the forced installation of SDXL, which has destroyed our ability to design and enjoy our creations.
The fact that we cannot roll back since all the commits are obsolete further exacerbates the situation.
I appeal to intelligence, logic, and reason to rescue SD 1.5 from this SDXL nightmare, in the interest of the community that supports SD 1.5 and has no interest in SDXL.
I understand that it will require effort, undoubtedly, but please realize that people like me, who have invested over €6,000 in equipment for significant projects, are now stuck and technically unemployed due to this SDXL implementation.
Thank you for considering my request. I also urge the entire community to support this message so that SD 1.5 can be revived and no longer remain in its current state.
Also, thank you for the effort and work invested, but please, separate SD 1.5 from SDXL, for the sake of all those who support you, believe in you, and hope for a repaired and functional SD 1.5 to return.
Thank you in advance.
Best regards,
The text was updated successfully, but these errors were encountered: