Recursive support for captioning/tagging scripts

[Linaqruf]

Hi! Thanks for the good work as always.

In this PR, I want to propose some small changes, mostly about recursive support for captioning/tagging scripts so users can annotate their dataset recursively. However, I am not sure if it is implemented correctly, so I hope for your review to make it better.

For recursive args, I borrowed glob_images_pathlib() from train_util.

1. tag_images_by_wd14_tagger.py

• Added --recursive to find and preprocess datasets inside sub-directories.
• Added --remove_underscore args.
• Added --undesired_tags, so users can delete undesired tags from the tagging process.
• Added character tags (category = 4) as well as --character_threshold. I don't know if it's a good idea, but I think it might be helpful for character training. SmilingWolf released SmilingWolf/wd-v1-4-convnextv2-tagger-v2, and I think it's a great and up-to-date model for character tagging.
• Renamed --thresh to --general_threshold.
• Added --frequency_tags to print tag frequency after the tagging process is done.
• Due to character tags added, I updated how --debug works with the following new template:

    {image_path} 
    Character Tags = {character_tags}
    General Tags = {general_tags}

2. make_captions_by_git.py

• Added --recursive to find and preprocess datasets inside sub-directories.

3. make_captions.py

• Added --recursive to find and preprocess datasets inside sub-directories.

4. prepare_buckets_latents.py

• Added --recursive to find and preprocess datasets inside sub-directories. I thought it was already covered by --full_path, but it cannot preprocess datasets inside sub-directories and make latents of them. It might not be useful for multi-concept training, but I think it's useful for multi-directories training. So users can keep the datasets inside the respective folder and preprocess them without needing to re-run the scripts every time.

5. convert_diffusers20_original_sd.py

• It might be a QoL rather than a new change, but I added --save_precision_as with choices of ["fp16", "bf16", "float"].

Thank you!

[kohya-ss]

Thank you for this! It looks good! I will review it when I have time.

[kohya-ss]

I've merged this PR. Sorry for the delay. I've restored thresh option for the backward compatibility. Thank you for this again!

One minor thing, there seems to be a problem with the underscore in ^_^ or >_< etc. tags, they are removed. What do you think?

[Linaqruf]

I'm sorry for the late reply, and thanks for the merge!

I forgot emoji tags exists 😅

Also I'm sorry but I cant use my pc right now for an hour or two

How about using the same clean_tags_and_captions.py approach? Like this:

  tags = tags.replace('^_^', '^@@@^')
  tags = tags.replace('_', ' ')
  tags = tags.replace('^@@@^', '^_^')

[kohya-ss]

Thank you for your reply! Your approach seems to work well!

I thought your suggestion looked good, but just to be sure, I checked the selected_tags.csv. I found 12 emoji tags and no normal tags by searching with the regular expression ,?_?,. Based on this, I would like to proceed with the following implementation, which is in line with your idea:

if len(tag) > 3 and '_' in tag:
    tag = tag.replace('_', ' ')

I will update the script later today after work!