Possibility to use with byt5-small ?

[oxysoft]

Hi, XL model way too large for me! I tried byt5-small but crash:

Traceback (most recent call last):
  File "/home/nuck/discore/src_core/lib/corelib.py", line 79, in invoke_safe
    func(*kargs, **kwargs)
  File "/home/nuck/discore/src_core/classes/printlib.py", line 193, in wrapper
    return func(*args, **kwargs)
  File "/home/nuck/discore/src_core/rendering/renderer.py", line 106, in emit
    cb(v, name)
  File "/home/nuck/discore/sessions/pinchsky-dance/script.py", line 101, in callback
    paella.txt2img(seed=v.seed, steps=v.steps, chg=clamp(v.chg, 0.1, 1), w=v.w, h=v.h, cfg=v.cfg, p=v.prompt, negprompt=v.negprompt, sampler=v.sampler)
  File "/home/nuck/discore/src_plugins/paella/PaellaPlugin.py", line 47, in txt2img
    ret = colab.txt2img(prompt, steps=steps, size=size, cfg=cfg, seed=seed)
  File "/home/nuck/discore/src_plugins/paella/colab.py", line 219, in txt2img
    sampled_tokens, intermediate = sample(model,
  File "/home/nuck/discore/src_plugins/paella/colab.py", line 66, in sample
    logits = model(sampled, t, **model_inputs, attn_weights=attn_weights)
  File "/home/nuck/discore/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/nuck/discore/plug-repos/paella/Paella/utils/modules.py", line 273, in forward
    c_embed = self.gen_c_embeddings(byt5, clip, clip_image)
  File "/home/nuck/discore/plug-repos/paella/Paella/utils/modules.py", line 224, in gen_c_embeddings
    seq = self.byt5_mapper(byt5)
  File "/home/nuck/discore/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/nuck/discore/venv/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x1472 and 2560x1024)
mat1 and mat2 shapes cannot be multiplied (1x1472 and 2560x1024)

[dome272]

Unfortunately, Paella requires at least 30GB of memory since we trained on big conditoning models. We will change that in future models.

[oxysoft]

So not even on RTX 3090? That's a pretty huge restriction if I do say so

[dome272]

Definitely. Paella is not a finished research project and we are still working on improving on many things. One thing that you can do is just not load t5 at all and enable t5=False in the sampling script. Then you can also do del clip.visual and then it should work probably. Let me know if that works for you!

[oxysoft]

I tried but I think there are references to t5 stuff even outside the checks for t5, and there seem to be more complications in the Paella model with the byt5_mapper

[GaggiX]

I managed to use Paella on colab using ~10GB of VRAM by simply using the CLIP image model to create variations, and deleting the T5 model and the prior model (I could also delete the text clip encoder).

If you have a rtx 3090 I think you can try loading the byT5-XL model alone, save the embeddings, delete the model and load the rest (if it doesn't work, try loading the model in half precision).

[SinanGncgl]

I managed to use Paella on colab using ~10GB of VRAM by simply using the CLIP image model to create variations, and deleting the T5 model and the prior model (I could also delete the text clip encoder).

If you have a rtx 3090 I think you can try loading the byT5-XL model alone, save the embeddings, delete the model and load the rest (if it doesn't work, try loading the model in half precision).

Can you share your code for image variations? Thank you.