Basic dataset pipelines

[cloneofsimo]

• Basic stuff : Preprocessor that handles SR, BLIP for autocaption.

Further tricks are planned to go to other repo. Might as well remove these in the future as well, but I'll have these for quick CLI sake

• SR with SwinIR
• BLIP autocaption
• CLIPSeg
• Face Detection
• Support Mask in dataset
• Magic Saving CLI

[cloneofsimo]

Simple CLI will now take any directory to create BLIP captioned + SwinIR upscaled + CLIPSeged mask.

[cloneofsimo]

Example output :

[cloneofsimo]

Usage :

lora_ppim ./data/data_cat ./data/data_catpp, --target_prompts="cat"

[Dango233]

Aw...we built the same stuff pretty much at the same time
Masking is helping with the final quality massively.

[cloneofsimo]

By we do you mean stability AI? BTW I've actually developed masking quite a long time ago haha #96

[Dango233]

I mean you and me lol. I was exploring the data pipeline for fine-tuning last weekend.

And then I saw your PR lol

[cloneofsimo]

Oh yes, lots of room for improvement here. Would gladly accept any PRs 🤗