textual inversion and dreambooth #10
Replies: 19 comments 39 replies
-
|
Hello, thank you. |
Beta Was this translation helpful? Give feedback.
-
|
i tried it today on collab with 40Gb Vram but give me a lot of trouble after generating the class images. i got 2 24 gb vram cards back in romania . next week im going back home , maybe we can meet over the weekend and try to debug and work on something. ? I have collab plus subscription so we can test it there.. what do you say ? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, it's a good idea, we can try. |
Beta Was this translation helpful? Give feedback.
-
|
I have not asked him yet, stil trying to find my way around it. i'm same as you, trying to catch up and learn :)) |
Beta Was this translation helpful? Give feedback.
-
|
update : don't try the notebooks , they don't work. finally I've run the python train_textual_inversion.py and it works... it takes : 21.5 / 40.0 GB VRAM for textual inversion. so we ca meet during the next following weekend and run tests on my card I have enough VRAM to run it |
Beta Was this translation helpful? Give feedback.
-
|
Understood, thanks for checking. |
Beta Was this translation helpful? Give feedback.
-
|
https://drive.google.com/file/d/1-gJOyxqL1rJBnns9GpzoytJCwouUrfgw/view?usp=sharing , try this oil painting embedding , keyword : sks , trained on 10 images 100 epochs => almost 4 hours |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your efforts! Sorry for misinformation, inference actually works with 16 Gb Tesla T4, total VRAM usage does not exceed ~9.5 Gb. |
Beta Was this translation helpful? Give feedback.
-
|
They should come out as oil painting style with heavy strokes , paint dripping, something similar with the ones on playground.ai |
Beta Was this translation helpful? Give feedback.
-
|
For inference I used slightly adapted notebook from https://github.com/TheDenk/Kandinsky-2-textual-inversion. Output for model without embeddings
Output for model with your embeddings
Looks like it works ;) |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Yes, exactly. I used this link for direct download https://drive.google.com/uc?id=1-gJOyxqL1rJBnns9GpzoytJCwouUrfgw |
Beta Was this translation helpful? Give feedback.
-
|
These are the images I trained it with .. here’s the link :
https://drive.google.com/file/d/1-0lToiMrU7xS7Ib6sIRJi7suT08sKr-9/view?usp=drivesdk
On Tue 25. 4. 2023 at 17:49, neutron_hare ***@***.***> wrote:
Yes, exactly. I used this link for direct download
https://drive.google.com/uc?id=1-gJOyxqL1rJBnns9GpzoytJCwouUrfgw
What should the result look like? Can you give few examples of images you
used for training?
—
Reply to this email directly, view it on GitHub
<#10 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABNVRRR7DP4AS26YCCAE4DXC7XA5ANCNFSM6AAAAAAXJE66X4>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
Best regards,
George
|
Beta Was this translation helpful? Give feedback.
-
|
Wati , i sunt YOU The wrong link… try this one
https://drive.google.com/drive/folders/1CCUFrmwjmZQv30GDsHgOenPjzKMSzwZi
On Tue 25. 4. 2023 at 17:49, neutron_hare ***@***.***> wrote:
Yes, exactly. I used this link for direct download
https://drive.google.com/uc?id=1-gJOyxqL1rJBnns9GpzoytJCwouUrfgw
What should the result look like? Can you give few examples of images you
used for training?
—
Reply to this email directly, view it on GitHub
<#10 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABNVRRR7DP4AS26YCCAE4DXC7XA5ANCNFSM6AAAAAAXJE66X4>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
Best regards,
George
|
Beta Was this translation helpful? Give feedback.
-
|
What initializer_token did you use for training? |
Beta Was this translation helpful? Give feedback.
-
|
If I remember correctly I think it was : painting . Try with this one… I’ll
open the collar in a min and check
On Tue 25. 4. 2023 at 21:35, neutron_hare ***@***.***> wrote:
What *initializer_token* did you use for training?
—
Reply to this email directly, view it on GitHub
<#10 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABNVRUWZ5A27XKUPCU33Q3XDARQRANCNFSM6AAAAAAXJE66X4>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
Best regards,
George
|
Beta Was this translation helpful? Give feedback.
-
|
Tried another prompt: Without embeddings
With embeddings
|
Beta Was this translation helpful? Give feedback.
-
|
There’s a guy modifying it to diffusers and now that nvidia published
tensorrt code to huggingface diffusers , if he sees it , he might get it
speeded up about 4 5 times then usual …
On Tue 25. 4. 2023 at 21:37, Adrian Puiu ***@***.***> wrote:
If I remember correctly I think it was : painting . Try with this one…
I’ll open the collar in a min and check
On Tue 25. 4. 2023 at 21:35, neutron_hare ***@***.***>
wrote:
> What *initializer_token* did you use for training?
>
> —
> Reply to this email directly, view it on GitHub
> <#10 (reply in thread)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AABNVRUWZ5A27XKUPCU33Q3XDARQRANCNFSM6AAAAAAXJE66X4>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
--
Best regards,
George
--
Best regards,
George
|
Beta Was this translation helpful? Give feedback.
-
|
You mean huggingface/diffusers#2985 (comment)? Speedup would be nice, but it won't solve memory problem. There are two diffusion models in Kandinsky pipeline (prior and unet), together they do not fit into 8 Gb. And I was hoping to somehow find a way to optimize and make it "consumer-friendly", so it might be able to compete with Stable Diffusion. But for this it has to be accessible for users without high-end GPUs. |
Beta Was this translation helpful? Give feedback.
-
|
:)) those look weird :)) .. I made a few last night but got them on pc,
they came out really nice, don’t forget the text decoder works better then
in stable diffusion. Give it something like this: a beautiful painting of a
meadow covered with grass and poppies and tall pine lit by sun at golden
hour.
On Tue 25. 4. 2023 at 22:20, neutron_hare ***@***.***> wrote:
Tried another prompt: sks painting The results are weird:
Without embeddings
[image: 0]
<https://user-images.githubusercontent.com/26826215/234393214-570b4ec1-b78b-40ce-9dbd-2488e82fc6ce.png>
[image: 1]
<https://user-images.githubusercontent.com/26826215/234393265-2a3be88d-bfbc-45b3-aefd-048af82ea2e7.png>
With embeddings
[image: 0 (1)]
<https://user-images.githubusercontent.com/26826215/234393471-06ceecc0-ce47-49ef-af03-328e88819c76.png>
[image: 1 (1)]
<https://user-images.githubusercontent.com/26826215/234393540-66e759f2-c4a3-416b-8476-879a88a7275f.png>
—
Reply to this email directly, view it on GitHub
<#10 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABNVRUUZ2SPP4BEJXWAXLLXDAWZLANCNFSM6AAAAAAXJE66X4>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
Best regards,
George
|
Beta Was this translation helpful? Give feedback.
-
|
Thank you, I will try tomorrow. |
Beta Was this translation helpful? Give feedback.
-
|
Still not getting it right... Embedding OFF
Embedding ON
Tbf, I never trained a textual inversion so I have no idea what's wrong with your embeddings (if there is an error). I will add UI for training into my app, but right now I am working on extensions support, upscaler and image browser components, after that gonna move on to training widgets and hopefully do some additional research afterwards. Btw, I am very interested in approaches suggested in the paper referenced here, looks promising. |
Beta Was this translation helpful? Give feedback.
-
|
class CoreAttention(MegatronModule): https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/model/transformer.py#L203-L209 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, will take a look. |
Beta Was this translation helpful? Give feedback.
-
|
DeepFloyd-IF was just released and there is a blog post on 🤗 that contains some insights into optimizing model for low-end GPUs and its usage with diffusers (and probably some of them can be used for Kandinsky as well). Guess I found some reading for next weeks :) |
Beta Was this translation helpful? Give feedback.
-
|
hi buddy, hope you're ok. |
Beta Was this translation helpful? Give feedback.
-
|
Hi. Well, actually there is already a built-in extension that allows usage of predefined styles, and I think I can tweak it to enable editing user styles:
Perhaps I might do it near weekend. |
Beta Was this translation helpful? Give feedback.
-
|
perfect. if i get some time this ill try to modify the t2i ui tab and
create more of a image generation workflow similar to the one i had on my
little t2i app and try to get more of that gallery space on screen. it
only needs an accordion that get hidden once the image settings where set
cause the majority of them generate at fixed size then upscale so no point
in keeping them to occupy that space that all about .. generating and
viewing the images
…On Tue, 2 May 2023 at 21:23, neutron_hare ***@***.***> wrote:
Hi. Well, actually there is already a built-in extension that allows usage
of predefined styles, and I think I can tweak it to enable editing user
styles:
[image: image]
<https://user-images.githubusercontent.com/26826215/235749872-7908ff2a-9163-429d-9ce4-aab1519d9ad4.png>
Perhaps I might do it near weekend.
—
Reply to this email directly, view it on GitHub
<#10 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABNVRW77GO5ASV5GSGYBNDXEFGJBANCNFSM6AAAAAAXJE66X4>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Yeah, I know, the gallery panel and UI in general needs to be improved, and at this point I don't take care of UX enough 😀 |
Beta Was this translation helpful? Give feedback.
-
|
i'll try to do it. you focus on the coding part. :)) i've tried to
understand the code but neah, its not mine if you know what i mean :)). aw
btw make a template for extensions when you have time , maybe others will
join and help
…On Tue, 2 May 2023 at 22:27, neutron_hare ***@***.***> wrote:
Yeah, I know, the gallery panel and UI in general needs to be improved,
and at this point I don't take care of UX enough 😀
—
Reply to this email directly, view it on GitHub
<#10 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABNVRUAESR3QAAIQRS6CITXEFNZHANCNFSM6AAAAAAXJE66X4>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
I will generate extension template for sure, just not finalized its generic implementation and API, so not ready yet to expose it. Once I get done outpainting and training widgets (the latter will also be a built-in extension), I'll add dev templates, unit tests and probably some sort of docs. |
Beta Was this translation helpful? Give feedback.
-
|
there's a weird bug in Gradio when using gr.Box() `OSError: [Errno 98] Address already in use During handling of the above exception, another exception occurred: Traceback (most recent call last): |
Beta Was this translation helpful? Give feedback.
-
|
Official training notebooks came out :) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! I will use use these notebooks and also scripts from https://github.com/TheDenk/Kandinsky-2-textual-inversion |
Beta Was this translation helpful? Give feedback.
-
|
Some notes: Also here is script for preparing dataset for training: import os
import pandas as pd
from PIL import Image
path_to_images_directory = '/workspace/data'
# Collect image filenames and captions
data = []
for filename in os.listdir(path_to_images_directory):
if filename.endswith(('.jpg', '.jpeg', '.png', '.bmp')):
# Assume the caption is the part of the filename before the extension
caption = os.path.splitext(filename)[0]
image_path = os.path.join(path_to_images_directory, filename)
data.append([image_path, caption])
# Create a DataFrame and save it as a CSV
df = pd.DataFrame(data, columns=['image_name', 'caption'])
df.to_csv('test.csv', index=False)
# Create a new directory named 'processed_images'
os.mkdir('images')
# Process images and save them in the 'processed_images' directory
for image_path, caption in data:
image = Image.open(image_path)
resized_image = image.resize((768, 768)) # Resize the image to 768x768 pixels
new_image_path = os.path.join('images', os.path.basename(image_path))
resized_image.save(new_image_path) |
Beta Was this translation helpful? Give feedback.
-
|
Btw this must be your post here: https://www.reddit.com/r/CodeLyoko/comments/13geunw/japanese_aelita_ai_art/ ? |
Beta Was this translation helpful? Give feedback.
-
|
yes and yes |
Beta Was this translation helpful? Give feedback.
-
|
Finished (for now) with outpainting and smaller stuff and will start working on extension for fine-tuning :) |
Beta Was this translation helpful? Give feedback.
-
|
thank you. i really like this model more then the autistic stable diffusion and the way it understands text. for the next few days ill just stay on the midjourney showcase feed and build my dataset then find a way to make that docker image work and start the training , if it wont work ill just make the training on my pc , i just need to find out how long it took the other guy to train both models for those 650 mentioned photos and make a rough timing assumption . PS : can't wait for those training tabs :D . IF they finish with diffuers you might have a good chance to controlnet, vram optimisation will come by default |
Beta Was this translation helpful? Give feedback.
-
For training I used A6000 48 GB VRAM. Trained on RunPod :) |
Beta Was this translation helpful? Give feedback.
-
|
ill do the training on my pc. hopefully nobody will unplug it :)) |
Beta Was this translation helpful? Give feedback.
-
|
I have published an extension for training prior and unclip models on the 'dev' branch. It includes a GUI for preparing datasets and training both models. Technically, it works fine as I have tested it on free Colab and Paperspace (where I have a pro subscription). By technically, I mean that I was able to successfully train models and obtain checkpoints. However, I have not tested (yet) the image output from these checkpoints 😉 Before merging it into the main branch, I need to perform more testing and add additional necessary functions. Currently, there is no UI for applying a trained checkpoint, and the original Kandinsky checkpoints need to be replaced manually. Additionally, the UI feels somewhat bloated, and I need to address that. I would greatly appreciate feedback from @adrianpuiu and @kodxana who have already trained Kandinsky via notebooks. Any ideas or suggestions to make the UI more convenient would be highly valued :) Screenshots
|
Beta Was this translation helpful? Give feedback.
-
|
So far looks good will give it a spin, capition editor would be great thing. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I plan to add editing captions via GUI as well. |
Beta Was this translation helpful? Give feedback.
-
|
tested it so far and its great I don't see anything missing . apart from what @kodxana said , a captioning editor but we can do without it for a while. better focus on some new feature or enhancement/plugin . |
Beta Was this translation helpful? Give feedback.
-
|
@seruva19 when you have some time. can you please add a clip interrogator for creating captions from data set folder ? |
Beta Was this translation helpful? Give feedback.
-
|
you're a butt saver man. thanks |
Beta Was this translation helpful? Give feedback.
-
|
Slightly modified script made by @Blucknote so it would filter out non-relevant images and output the result in single array: |
Beta Was this translation helpful? Give feedback.
-
|
@adrianpuiu I added GUI for batch processing in interrogation tab. |
Beta Was this translation helpful? Give feedback.
-
|
aw man, sweet. thank you! |
Beta Was this translation helpful? Give feedback.
-
|
330it/s * 86 pics done , splendid work. loving it |
Beta Was this translation helpful? Give feedback.
-
|
so im trying to work around something like this : http://jsfiddle.net/Snfst/6/ |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
hi buddy, saw your work and looks great...
you might also want to look at : https://github.com/TheDenk/Kandinsky-2-textual-inversion
the guy is working on drambooth and textual inversion scripts. would be great to have them integrated in your UI
Beta Was this translation helpful? Give feedback.
All reactions