Memory Requirements

[aisagarw]

I have been running the notebooks on a 16GB t4. In some cases, it runs well. But in some other cases (the observation is that loss values don't seem to go down much over iterations in many such cases), it throws up CUDA out of memory error in between the iterations itself. Why is it the case that it works well for the same number of iterations for certain prompts but not for others?

For instance, it works fine for a prompt like a cow and dog standing ​with a cat . But at the same time, for prompts like a boy cutting a birthday cake wearing a hat and an elephant with a crown standing next to a man​, it throws memory errors within around 10 iterations. (The italics represent the subjects)

Am I missing something? What can be the probable reasons behind something like this?

[Meatfucker]

On a 24gb 3090 I have seen it hit a bit over 18gb with complex prompts.

[AttendAndExcite]

Hi @aisagarw, thanks for your interest!
In all our experiments the memory consumption did not exceed 16gb.
Since each step optimizes a single neglected token, the number of optimized tokens should not impact the consumption.
Additionally, the text is always at a fixed length of 77 tokens so I’m not sure the length of the text should impact that either.
Could you please share additional details? Seeds etc., we will do our best to look into it :)

[aisagarw]

Sure! Following are the exact details:

prompt = "an elephant with a crown standing next to a man​"
seeds = [21]

I actually didn't change any of the default values in the code (explain.ipynb) apart from the prompt and corresponding subject tokens (elephant, crown and man in the above example) . Thanks much for your response!

[AttendAndExcite]

thanks @aisagarw, it seems that your prompt ("an elephant with a crown standing next to a man​") probably resides outside the distribution that SD has learned for textual descriptions (see the limitations section of our paper). This triggers the iterative refinement for the maximal number of steps which increases the memory consumption beyond 16gb.
in general, as mentioned in the paper, since we do not fine-tune the generative model, our results are limited by the expressive power of SD.

[aisagarw]

Thanks much for your insights and response. But one observation I had was, that for the same prompt as mentioned above, the training went fine when only elephant and crown were the two subject tokens. Whereas, in case where 3 subject tokens namely elephant, crown and man were given, the above mentioned memory issue occurred.

Is it maybe that the optimization becomes difficult when there are multiple neglected subject tokens?

[AttendAndExcite]

of course @aisagarw, happy to help. this actually supports my previous claim- elephant + crown is in distribution therefore the method works, and the iterative refinement is not even activated so the memory consumption is ok ("man" is not highlighted, i.e., not enforced by the method and can be ignored). when you add the demand to generate the man- the model struggles, iterative refinement is applied (to the token "man") and the memory consumption increases.
I hope this clarifies things :)
as mentioned above, since the optimization is applied to the most neglected tokens, the actual number of tokens to optimize should not make a difference (also see multiple examples on our website and in the HF space with 3 or even 4 subjects to optimize that did not require over 16gb to optimize).

[6Roy]

Hello,sorry to bother you.I have some puzzles:When the requirement to generate man is added, iterative refinement is applied (to the tag "man"), and memory consumption increases.So why will lead to this memory consumption increases and the generated image is as follows(not right). As you exampled,it did support 3-4 subjects, when will not result in memory consumption increases?