Hypernetwork Style Training, a tiny guide

[Heathen]

The negative text preview during training appears to have been fixed a few patches ago, carry on.

tl;dr

Prep:

• Select good images, quality over quantity
• Train in 512x512, anything else can add distortion
• Use BLIP and/or deepbooru to create labels
• Examine every label and remove whatever is wrong, add whatever is missing
• For activation and initialization, check Hypernetwork Style Training, a tiny guide #2670 (comment)
• For network size, check Hypernetwork Style Training, a tiny guide #2670 (comment)

Training:

• Learning Rate: 5e-5:100, 5e-6:1500, 5e-7:10000, 5e-8:20000
• Prompt Template: a .txt with only [filewords] in it.
• Steps: 20000 or less should be enough.

Longer explanation:

• Select good images, quality over quantity
  My best trained model was done using 21 images. Keep in mind that hypernetwork style transfer is highly dependent on content. If you pick an artist that only does cityscapes and then ask the AI to generate a character with his style, it might not give the results you expect. The hypernetwork intercepts the words used during training, so if there are no words describing characters, it doesn't know what to do. It might work, might not.

• Train in 512x512, anything else can add distortion
  I've tested this several times. I haven't gotten good results out of it yet. So up to you.

• Use BLIP and/or deepbooru to create labels AND Examine every label and remove whatever it wrong, add whatever is missing
  It's tedious and might not be necessary, if you see blip and deepbooru are working well, you can let it as is. In any way, describing the images is important so the hypernetwork knows what it is trying to change to be more like the training image.

• Learning Rate: 5e-5:100, 5e-6:1500, 5e-7:10000, 5e-8:20000
  They added a training scheduler a couple days ago. I've seen people recommending training fast and this and that. Well, this kind of does that. This schedule is quite safe to use. I haven't had a single model go bad yet at these rates and if you let it go to 20000 it captures the finer details of the art/style.

• Prompt Template: a .txt with only [filewords] in it.
  If your blip/booru labels are correct, this is all you need. You might want to use the regular hypernetwork txt file if you want to remove photo/art/etc bias from the model you're using. Up to you.

• Steps: 20000 or less should be enough.
  I'd say it's usable in the 5000-10000 range with my learning rate schedule up there. Buuut you will notice that in the 10000-20000 range, a lot of the finer details will show up. So as the rock would say, put in the work, put in the hours.
  

Final notes after the rock intermission.

• If your model uses VAE, keep it on. Don't know if this makes a difference in training, but just making sure.
• Unload any other hypernetworks. I'm not sure if it interferes with training, but better be safe.
• If your model breaks and the preview tests show colorful noise, don't just go back a little, pick an even earlier model to train onward and reduce the learning rate even more
• Don't change your training data midway, better start over.
• https://www.birme.net/ is great for bulk cropping images.
• If your loss goes past 0.3 you done goofed and probably broke your hypernetwork, lower the learning rates.

Examples:
Trained NAI for 6500 steps on Andreas Rocha style. I plan on letting it train to 20000 later. And done.

Vanilla NAI

RTX On, I mean, Style on

20000 steps

Vanilla NAI

Rocha ON

20000 Steps

[ghost]

I find that my hypernetworks are starting to cook on 5e-6 somewhere after ~17k steps, so that might be a good stopping point.
On 5e-7 it's taking really tons of time to train, starting to be close to artist of my choice somewhere ~70k steps.

[Heathen]

It was added a couple days ago, the format is
learningrate:steps, ...
It does interpret scientific notation as well, so a schedule like this example works:
5e-6:15000, 5e-7:30000

[ogkalu2]

@andrei007999
How many images did you use to train at 5e-7 ?
How long past 70k steps did you go and how close did the style get ?

[ghost]

@ogkalu2 ~250 augmented i think, not much images produced by that author, but they were manually tagged and cropped.

I didn't, 70k did produce some outputs very close to style, so i stopped there, but it's likely that you can go for another 80k i think.

[ogkalu2]

@andrei007999
Did you try the scheduler ? Did it work better ?

[ghost]

@ogkalu2 I did, it seem to get faster to base of style, well, but that was obvious, because starting LR is much faster. But to see if it's better, i would need to train over 20k steps, and i did not have opportunity to do so yet on any hypernetwork.

What i can say, seem like lower LR make it work out smaller details better instead of generalized stylistic of art, so, in theory scheduler should achieve much better results in the long run.

[ExponentialML]

Thanks for the guide! I find that hypernetworks work best to use after fine tuning or merging a model. Trying to train things that are too far out of domain seem to go haywire. It makes sense considering that when you fine tune a Stable Diffusion model, it will learn the concepts pretty well, but will be somewhat difficult to prompt engineer what you've trained on. Hypernetworks seem to help alleviate this issue.

[ogkalu2]

Do you train the hypernetworks on the fine tuned or merged model ?

[Heathen]

A few more examples of NAI + Andreas Rocha hypernetwork now that it is trained.

[Chubler-XL]

Thanks for this guide I have been struggling to get an embedding of a particular atrists style sorted out and this helped no end to an acceptable result.

I had 26 examples of the artists work which I manually resized/cropped to 512x512.
I created the labels using BLIP and manually corrected them.

Embeddings need a much bigger Learning rate, and after some trial and error, I ended up with:

Initialization text: *
Vectors: 2
Learning rate: 0.02:200, 0.008:800, 0.002:2000, 0.0008:5000
Prompt template file: textual_inversion_templates\style_filewords.txt

Nice thing about the embeddings is I can use a standard model and just add "painting by [artist-name]" to my prompts.

I will try extending the final learn out to a much bigger number of steps and see if more details appear.

[mykeehu]

What I have noticed so far is that if the VAE was loaded for the basic SD model, it learned slowly, but more accurately. If it was not, it learned faster, but with txt2img it was very sensitive to steps and CFG.
Otherwise I had to set the CFG very low (3-5) to generate good results in both cases. At a value higher than 10, it transferred style rather than object.

[Chubler-XL]

Thank you very much for the ideas! I'm training for concept now, not style, but I'll try your numbers. Watching the reps and periods, they look good, but I'll only bring them down after a longer workout, but not to 0.00005 (for TI): 0.02:200, 0.008:800, 0.002:2000, 0.0008:8000, 0.0002:20000, 0.00008:30000 I always get very confused images, but it seems that as I lower the rate, it starts to learn what I want. The interesting thing about the training though is that even though I tried to step back from 5000 to 4000 and lower the rate, the training still continued from 5000, even though I deleted the newer models and changed the rate in the folder. Maybe I should try without VAE for TI as well? Although in @Heathen's case VAE was left on, elsewhere they say exactly to disable it anyway for training. I'm getting a little unsure about that. Maybe it should be turned off for HN but not for TI?

The current step (and everything learned) is stored in the .pt file embeddings\xyz.pt so you want to replace this file with textual_inversion\<date>\xyz\embeddings\xyz-4000.pt to back track to step 4000.

If you are training a hypernetwork you would replace models\hypernetworks\xyz.pt with the file from textual_inversion\<date>\xyz\hypernetworks\xyz-4000.pt

If you want to scrap and start again you would delete the file and then use "Create Embedding" or "Create Hypernetwork" to build an unpopulated start file.

[mykeehu]

@Chubler-XL I admit, I still don't understand these LR values. I've been trying different solutions for several days: training with a constant value of 0.005 for at least 20000 steps, or aggressive values at first, then lower values, but the loss always stays below 0.2 and above 0.05, average between 0.09-0.12. I almost always get the same loss values whichever LR I choose. Of course, the images always change and are sometimes better, sometimes worse, sometimes better again... So how to find out a good learning rate that works in many cases?

The other is that if I don't train on style but on concept (i.e. general, e.g. a motto, a posture) and I use [filewords], TI generates better images, but if I retrieve an image for the keyword, it doesn't understand it, as if it was just an effect, even though I train on subject_filewords.txt. Conversely, if I just use subject.txt, the images are blurry, but then txt2img generates the same blurry image, so that's the better way to go. I'm training with 10 images right now, but I also used 60 (30x2) and that seemed like too many for it.

That's why I'm really interested in LR, is it worth starting coarser (like 0.5 or 0.2) and then tweaking if I want to train something new and unknown, or should I start lower so it doesn't rush and then I can go up to 40,000 steps?

[Chubler-XL]

I've been playing around with quite small LRs like the guys are using for the Hyopernet "5e-5:300, 5e-6:1200, 5e-7:10000, 5e-8:20000" with some fairly good results. Everything I've tried so far has been a style rather than a single object / posture etc.

I'm trying to do "art by Ken Done" at the moment and that is quite difficult, very colorfull and abstract style which SD is struggling with.

[mykeehu]

For me, it would be a specific pose, shot from several angles, 10 pictures in total. I've been trying on TI beyond 20000, and with subject.txt (filewords just messed it up), but it still generates a similar one for every twentieth image (with a step of 50). It's not efficient that way, and the one that is similar is still far from what I want. It's like it has stopped learning and can't move on. He is training at 5e-8:50000 on the TI. Unfortunately Hypernetwork is not good for me because it is more style than subject or pose.
if I train from subject_filewords.txt, I see a good result (although it doesn't cover what I'm training for, but fewer bad pictures), but as soon as I generate for the keyword, I get again some deformed result. It's as if I fed it based on the filewords in vain.

[chekaaa]

can you share the post processed imgs that you used for the training? if its possible. Just to have a better idea of what works

[Heathen]

I use a mix of zoomed out images and close ups in details. These are from a single image.

What you really need to avoid is having images of the same thing that are only a few pixels apart, you might start getting double-vision-like images if you train on those.

[tearxinnuan]

Very good tutorial, although my VRAM currently doesn't support me to use Train XD

[chekaaa]

I trained a mob psycho Hypernetwork , here are the results with 26k steps

No mob psycho prompts where use to generate these images.

Some extras

[chekaaa]

Something that I noticed and Idk why It happens, I stopped the training and then started again with the same prompts and seed for the sample generation and I got a totally different img. It has happened to me before with other trainings. Any idea why this happens?

[thezakman]

I notice that too and I wonder why, sometimes I don't see any improvement and when I stop and start the training again, the same or next render is super different. Not sure if the hyper are being truly loaded in the actual render or something else.

[iosiflazotis]

@chekaaa Hello, do you mind uploading what your previews were like during the training of the Spy x Family hypernetwork? I want to get an idea of how I would know if I'm on the right track. I would greatly appreciate it.

[chekaaa]

Sadly I deleted all the previews bc of space.

Ill keep it in mind next time

[iosiflazotis]

Sadly I deleted all the previews bc of space.

Ill keep it in mind next time

It's okay. Thank you very much for considering it. I just find your training amazing, is all. I mean, that's literally Loid Forger. Hahaha.

[gissleh]

I'm trying to train it on Mass Effect aliens. I know the SD 1.4/1.5 model has a vague idea of what they are, but training goes in circles.

Is Hypernetwork the wrong tool for that?

[Heathen]

It's a bit pushing it. It only modifies the results of a model. If the result is bad or nonexistent, hypernetwork isn't it for you. I'd say train on everything mass effect, aliens, humans, style, etc and see if that works. If not, you probably need Dreambooth or Textual Inversion. But for TI you will have to train each character as its own embedding.

[gissleh]

Thanks, you're probably right. I was waiting on getting a used M40 working for Dreambooth, but maybe my card (2080ti) could handle it now that Dreambooth's VRAM usage has been optimized.

[AndreRatzenberger]

Dreambooth or native fine tuning/training would be the tool of your choice

[horribleCodes]

Do you know where the default values for training are kept? I'd like to edit change the usual 0.005 to your recommended schedule. There's usually always one value I forget to set and I have to start all over.

[horribleCodes]

I found it: modules\ui.py, line 1232. You can find the path to the template at line 1236. Unfortunately, this file is being tracked, so you'd either have to force pull and redo the edit, or stash the change and pop it back in after the pull.

[Heathen]

stable-diffusion-webui/modules/ui.py

Line 1232 in c8045c5

learn_rate = gr.Textbox(label='Learning rate', placeholder="Learning rate", value="0.005")

Yup, was going to link it, but you beat me to it.

[mykeehu]

It would be nice if this would be included in the ui settings file, and then you wouldn't have to touch the code

[horribleCodes]

Already started an issue: #2900

[Heathen]

For anyone wanting to test something, this is an annealing learning rate I'm trying out:
5e-5:100, 5e-6:1500, 5e-7:2000, 5e-5:2100, 5e-7:3000, 5e-5:3100, 5e-7:4000, 5e-5:4100, 5e-7:5000, 5e-5:5100, 5e-7:6000, 5e-5:6100, 5e-7:7000, 5e-5:7100, 5e-7:8000, 5e-5:8100, 5e-7:9000, 5e-5:9100, 5e-7:10000, 5e-6:10100, 5e-8:11000, 5e-6:11100, 5e-8:12000, 5e-6:12100, 5e-8:13000, 5e-6:13100, 5e-8:14000, 5e-6:14100, 5e-8:15000, 5e-6:15100, 5e-8:16000, 5e-6:16100, 5e-8:17000, 5e-6:17100, 5e-8:18000, 5e-6:18100, 5e-8:19000, 5e-6:19100, 5e-8:20000

It would be better if we could put math expressions in the learning rate field instead.

[Heathen]

It doesn't seem loss converges in this model. I think there's a minimal downward trend, but that would take hundreds of thousands of steps to converge if at all.

It also appears that in my particular data set there is a group of images with small loss and the majority has high loss. I would like for the CSV to include the picture name of the step taken, if you set it to log every 1 step.

[chekaaa]

its time to experiment with multilayer structure now, and I have no idea what it does

[enn-nafnlaus]

@Heathen I see what you're doing and I like it. :) Helps avoid getting stuck in local minima. Will try this out this evening, on a "1,2,4,2,1" model. I went and extended it out to >80k cycles, with decreasing sizes of bumps and annealing cycles.

[enn-nafnlaus]

Made a quick-and-dirty customizable generator to make learning rate prompts that emulate your "punctuated evolution". :)

learning_rate.txt

The burst rates are able to be varied in magnitude, and you can set the exponent for the distribution - a high exponent means that the highest bursts are rare while most are low.

[JonathanDotCel]

annealing

Tbh, I also find that this approach works nicely with manual intervention.
E.g. if things are starting to look warped, or go off topic, bump the learn rate up & down from a known-good checkpoint. It's far more hands-on, but does yield decent results quite quickly.

[chekaaa]

There is a PR for multilayer structure settings for hypernetworks #3086. Does anyone have an idea on this affects training?

[SeverianVoid]

From this which is the original pull request
#3086

hyper network layer structure
If write "1, 2, 1", hypernetworks are composed of 2 fully connected layers whose intermediate dim is 2x, which is same as up to now.
The more you add the number, like "1, 2, 4, 2, 1", the more the structure of hypernetworks becomes deeper. Deep hypernetworks are suited for training with large datasets.

Add layer normalization
If checked, add layer normalization after every fully connected layer. It would be meaningful to prevent hypernetworks from overfitting and make training more stable.

[enn-nafnlaus]

My experiments with normalization have been... less than stellar. But maybe I'm doing something wrong. Seems you have to use much higher training rates to see changes, but then they wonk out. Experimenting with deeper networks now.

[Heathen]

I've been able to train faster with normalization, but increasing the neural network density only slowed training down without any perceivable gain, at least on 100ish picture dataset.

[enn-nafnlaus]

What learning rate did you use? fim., 20. okt. 2022 kl. 17:42 skrifaði Pirate Kitty < ***@***.***>:

…

I've been able to train faster with normalization, but increasing the neural network density only slowed training down without any perceivable gain, at least on 100ish picture dataset. — Reply to this email directly, view it on GitHub <#2670 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3XG2HZZR47QM2FXR3N7R2LWEGABLANCNFSM6AAAAAARFV5TAI> . You are receiving this because you commented.Message ID: <AUTOMATIC1111/stable-diffusion-webui/repo-discussions/2670/comments/3927066 @github.com>

[Heathen]

I'm currently using 5e-3:200, 5e-4:400, 5e-5:1000, 5e-6:2000, 5e-7:3000 for normalized. Only training up to 3000 steps. But the results aren't good and they don't seem to get better with normalized. So it's only if you want something fast, I suppose.

[enn-nafnlaus]

Yeah, my goal is good, not fast. fim., 20. okt. 2022 kl. 19:43 skrifaði Pirate Kitty < ***@***.***>:

…

I'm currently using 5e-3:200, 5e-4:400, 5e-5:1000, 5e-6:2000, 5e-7:3000 for normalized. Only training up to 3000 steps. But the results aren't good and they don't seem to get better with normalized. So it's only if you want something fast, I suppose. — Reply to this email directly, view it on GitHub <#2670 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3XG2H2NL22VKOMDM6G22UDWEGOGJANCNFSM6AAAAAARFV5TAI> . You are receiving this because you commented.Message ID: <AUTOMATIC1111/stable-diffusion-webui/repo-discussions/2670/comments/3927937 @github.com>

[Heathen]

Trained another hypernetwork with okayish results, but I feel the loss we're getting back isn't indicative of anything. I know it's median loss and not step loss, but even so.

There doesn't seem to be a downward trend per epoch.

But analyzing the loss density, there are local downward trends.

This hypernetwork was trained with VAE on, I'm going to try again with VAE off and compare results and loss data.

[Heathen]

Trained the same data with VAE off

Loss was generally lower, but also all over the place. Still doesn't seem indicative of anything as far as hypernetworks go.

[ifeelagood]

if you take the EMA with a window size of ~100, there is a decently clear, small downward trend.
activation: softsign
network shape: 1, 1.5, 1.5, 1
dropout: no
layer initialisation: xavier normal
model: waifu-diffusion-v1.3
learning rate scheduler: 1e-4:200, 1e-5:2000,1e-6:4000,1e-7:20000,1e-8:100000

[mom333]

I don't know if I am late to the party and this is how it looks for everyone now that #4886 (comment) is a thing.

This is loss per epoch for 247 images split into 64 accumulation steps with LR 1e-3 and weight decay of 0.1. Result is obviously looking bad now (trying something here) but this is the first time I have seen optimizer actually converging towards a fraction of initial value.

[giteeeeee]

Just a quick question on how to read "loss convergence".

For example, a loss from "0.30-0.10" to "0.25-0.15", can be interpreted as converging? Am I understanding this correctly?

[Heathen]

Technically yes, but I haven't seen it actually do that, as you can see in the graphs above.

[Kallamamran]

I can't get Hypernetwork training "to work". Training a model on myself via Dreambooth creats great results, but trying the same with Hypernetwork I look like a** 😂 Like a long lost cousin or something. I tried Hypernetwork on a friend of mine and no matter what I did he turned up a good lookin asian dude and he is not asian at all 😂 Lol... Wish I could use this training since 10GB VRAM is to little to train Dreambooth locally

[XiteSDF]

If you haven't yet, try applying #4886. The difference is literally night and day.

[msampson3d]

So much has changed in the recent commits that I feel like most of the info in this thread is no longer relevant. Kinda feel like its time to start a new one with revised research and findings.

[ifeelagood]

agreed.

[mykeehu]

Yes, it is time to summarise the experience so far in a new group.

[DarkAlchy]

Agreed, since 2.1 rolled up I no longer can create a HN on a T4 no matter what I do. Prior I had made one HN.

[ProfaneServitor]

I tried training a hypernetwork on a character. So far, the best model was done with 20000 steps on 32 images and (1,3,1) architecture. However, it has big limitations:

• It requires a long prompt. 1girl, iwakura_lain, asymmetrical_hair, blunt_bangs, single_sidelock, x_hair_ornament, short_hair, brown_hair brown eyes is absolute minimum.
• As result, if you add more details, some tags start falling off: hair becomes symmetrical, the hairclip falls off or gets cloned, face becomes blurry, etc..