Lilypad-ify four new modules for SDXL & Mistral-7B fine-tuning and inference

[lukemarsden]

We want four new Lilypad modules:

• sdxl-finetune
• sdxl-inference
• mistral-finetune
• mistral-inference

Dockerfiles:

• https://github.com/lukemarsden/axolotl/blob/main/docker/Dockerfile (mistral)
• https://github.com/lukemarsden/sd-scripts/blob/main/Dockerfile (sdxl)

Docker images:

• https://quay.io/repository/lukemarsden/axolotl?tab=tags
• https://quay.io/repository/lukemarsden/sd-scripts?tab=tags

They should copy the formula of: https://github.com/bacalhau-project/lilypad-module-lora-training and https://github.com/bacalhau-project/lilypad-module-lora-inference which is the same for stable diffusion 1.5

Here are the commands you need to run inside the containers:

sdxl-finetune

bind-mount /config.toml, /input and /output
config.toml should contain

# for sdxl fine tuning

[general]
enable_bucket = true                        # Whether to use Aspect Ratio Bucketing

[[datasets]]
resolution = 1024                           # Training resolution
batch_size = 4                              # Batch size

  [[datasets.subsets]]
  image_dir = '/input' # Specify the folder containing the training images
  caption_extension = '.txt'                # Caption file extension; change this if using .txt
  num_repeats = 10                          # Number of repetitions for training images

accelerate launch --num_cpu_threads_per_process 1 sdxl_train_network.py \
	  --pretrained_model_name_or_path=./sdxl/sd_xl_base_1.0.safetensors \
  	--dataset_config=/config.toml \
  	--output_dir=./output \
  	--output_name=lora \
  	--save_model_as=safetensors \
  	--prior_loss_weight=1.0 \
  	--max_train_steps=400 \
  	--vae=madebyollin/sdxl-vae-fp16-fix \
  	--learning_rate=1e-4 \
  	--optimizer_type=AdamW8bit \
  	--xformers \
  	--mixed_precision=fp16 \
  	--cache_latents \
  	--gradient_checkpointing \
  	--save_every_n_epochs=1 \
  	--network_module=networks.lora

The input should be a folder of images with captions of images in text files e.g. foo.jpg should have a foo.txt with a caption
Based on https://github.com/kohya-ss/sd-scripts

sdxl-inference

Given an input lora in bind-mounted /input directory, inference is then just:

accelerate launch --num_cpu_threads_per_process 1 sdxl_minimal_inference.py \
	--ckpt_path=sdxl/sd_xl_base_1.0.safetensors \
	--lora_weights=/input/lora.safetensors \
	--prompt="cj hole for sale sign in front of a posh house with a tesla in winter with snow" \
	--output_dir=/output

mistral-finetune

accelerate launch -m axolotl.cli.train examples/mistral/qlora-instruct.yml

mistral-inference

accelerate launch -m axolotl.cli.inference examples/mistral/qlora-instruct.yml