Unable to reproduce anime-style faces for specific characters in LoRA training due to Danbooru influence

[shirokitune7113-coder]

[Issue: Unable to reproduce anime-style faces for specific characters in LoRA training due to Danbooru influence]

I am training LoRAs using anime images from Magia Record and Madoka Magica.
To improve the reproduction of the art style, I include the series title in the captions:

• For Madoka and the others: mahou shoujo madoka magica
• For Sana only: magia record: mahou shoujo madoka magica gaiden

Sorry for the long post!

Target characters:

• Sana Futaba
• Madoka Kaname
• Ultimate Madoka
• Homura Akemi
• Devil Homura
• Mami Tomoe
• Holy Mami
• Sayaka Miki
• Kyoko Sakura
• Nagisa Momoe

Training environment:

• XL LoRA Trainer by Hollowstrawberry
• Base model: Illustrious 0.1
• Generation model: Primarily WAI14
• lora_type: Locon

The Issue:
Even after LoRA training, the faces and art style are heavily influenced by the visuals found on Danbooru, making it impossible to faithfully reproduce the anime-style faces.
For Ultimate Madoka and Devil Homura, the output fails to look like anime unless prompts such as anime style, anime capture, or anime screen are used (though the anime look is somewhat maintained at higher epoch counts).
For characters other than Sana, the quality of the magical girl outfits and other elements degrades slightly if anime style, anime capture, or anime screen are omitted.
With the exception of Sana, the volume of Danbooru data for these characters is massive compared to others; excluding Sana and Nagisa, the dataset sizes are 1.5 to 3 times larger than that of Nami from One Piece (which has 10k images).
Changing the number of training images alters the faces slightly, but the underlying issue of being influenced by Danbooru remains unchanged.

Regardless of adjustments to learning rate, Rank, repeats, or epochs, the resulting faces remain almost identical every time; for everyone except Sana, the faces look nearly the same as those generated using only Danbooru tags without the LoRA.

On the other hand, the reproduction of outfits is highly accurate for all characters; even when the LoRA strength is lowered to 0.1, the outfit details and facial features remain intact. For outfits that cannot be accurately reproduced using only Danbooru tags, the fidelity drops significantly (e.g., Sana's uniform, Holy Mami).
I have attached an image of Madoka with her "Danbooru-style" face.

Things I tried:

• Removing the series title from the caption
• Removing character feature tags (e.g., "green eyes")
• Changing trigger words to meaningless terms
• Adjusting training settings (LR, Rank, Epoch, etc.)

→ Almost no effect.

Sana Futaba is a particularly puzzling case.

Unlike other Magia Record characters (Iroha, Yachiyo, Tsuruno, Felicia, etc.), no matter how many times I train her using Illustrious 0.1, her face ends up looking subtly different from the anime version.
Even though there are relatively few images of her in the Danbooru database (only 743)—unlike Madoka and the others—she actually shows high fidelity when generated using only Danbooru tags without a LoRA, and including "magia record: mahou shoujo madoka magica gaiden" in the prompt further improves accuracy.
Just like with the other five characters, the face remains the same regardless of changes to settings or learning rates. This issue does not occur with Iroha, Yachiyo, Tsuruno, or Felicia.

However,

• WAI14
• NoobAI-EPS 1.1

...result in a face that is closer to the anime version when used as the base model.

That said, the images lack the high resolution and quality achieved with Illustrious 0.1. When using WAI-based models, the lack of artistic polish becomes apparent. Testing with "anima" resulted in terrible quality, with blocky, angular features—especially around the face.
When I tested with Iroha, however, the quality was excellent and the image looked beautiful. For Sana: ① used the Illustrious 0.1 base model (generated with WAI14); ② used the WAI14 base model (generated with WAI14); ③ used the NoobAI-Eps1.1 base model (generated with Anime Screenshot Merge NoobAI v4); and ④ used the anima base model (generated with wai-anime).

Hypothesis:
I suspect that for characters with a massive amount of training data on Danbooru (such as Madoka and the others), the base model's inherent knowledge is too strong, preventing the LoRA from effectively overwriting it.

Questions:

1. Is there a way to reduce the influence of the face and art style derived from Danbooru?
2. Are there any caption elements—other than the series title or character feature tags—that I should review?
3. What training methods would you suggest for faithfully reproducing the anime-style face using Illustrious 0.1?
4. Would it be better to switch to a different base model entirely in this case?

Many high-quality AI illustrations that closely resemble the anime versions of Sana Futaba, Madoka, Homura, Mami, Sayaka, Kyoko, and Nagisa are being posted on Pixiv.

Therefore, I believe that reproducing this style is technically possible.

However, I do not know whether those works use LoRA or if they are generated using other models or services like Gemini or NovelAI.

I would like to know if there is a way to achieve a similar anime-style reproduction using an Illustrious-based model combined with LoRA.