Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

This Scripts is Using too much RAM than Original Kohya script #116

Closed
SatyamSSJ10 opened this issue Feb 6, 2023 · 2 comments
Closed

This Scripts is Using too much RAM than Original Kohya script #116

SatyamSSJ10 opened this issue Feb 6, 2023 · 2 comments

Comments

@SatyamSSJ10
Copy link

SatyamSSJ10 commented Feb 6, 2023

Continuing Previous Issue: [Errno 2] No such file or directory: '\kohya_ss\venv\Scripts\accelerate.exe\main.py'
Issue 38
I compared it with original Kohya script and found that somehow this is consuming lots of RAMs and I guess it doesn't transfer the CKPT to VRAM, The Original Kohya script is working well, so I literally used the same VENV to run this.
This is my Task Manager RAM graph, we can see that there's sudden increase in RAM and then it crashes, 16GB RAM, graph time previous 30 seconds.
image
image
This doesn't happen in the original training script. this resulting in the Issue 38.
When using the original Training Script, the CKPT is transferred to VRAM thus resulting in successful training.
image

@ShaguarWKL
Copy link

Is there a discrete GPU on your machine? The scroll bar on the taskmgr seems to end at GPU 0. If that's the case, your statements on vram is incorrect. Intel UHD "VRAM is actually pulling from system ram.

The last graph also seem to indicate eating more ram than the previous 2 graphs, could be a scale thing. Taskmgr graphs are woefully inadequate to make conclusions.

@SatyamSSJ10
Copy link
Author

Is there a discrete GPU on your machine? The scroll bar on the taskmgr seems to end at GPU 0. If that's the case, your statements on vram is incorrect. Intel UHD "VRAM is actually pulling from system ram.

The last graph also seem to indicate eating more ram than the previous 2 graphs, could be a scale thing. Taskmgr graphs are woefully inadequate to make conclusions.

Yes, I have RTX 3080TI.
In the second graph it loads the ckpt and then transfers it to GPU vram. In the first graph it doesn't happen.

DoubleCake pushed a commit to DoubleCake/kohya_ss that referenced this issue Aug 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants