Leapfrog Latent Consistency Model for Medical Images Generation
The scarcity of accessible medical image data poses a significant obstacle in effectively training deep learning models for medical diagnosis, as hospitals refrain from sharing their data due to privacy concerns. In response, we gathered a diverse dataset named MedImgs, which comprises over 250,127 images spanning 61 disease types and 159 classes of both humans and animals from open-source repositories. We propose a Leapfrog Latent Consistency Model (LLCM) that is distilled from a retrained diffusion model based on the collected MedImgs dataset, which enables our model to generate real-time high-resolution images. You may find belwo some sample medical images generated by our LLCM at several inference steps
Results comparison with the other state-of-the-art (SOTA) models
- You may refer to our paper for more details
Demo
- If you want to generate medical images using our model weight, visit our website. From the drop-down menu, select any prompt and the number of images. You may generate any number of images as per your requirement. The more the number of images to be generated, the more the inference time. So, we restrict the maximum number of images to 4 for a single click on our website.
- An example is shown below
How to fine-tune our model with your dataset
- Install these libraries: pip install diffusers transformers accelerate
- Move to the directory of the codes and launch this script 'train_text_to_img_llcm_launch.py' using the command shown below, after loading our model weight and train it to obtain the fine-tuned weight.
- python train_text_to_img_llcm_launch.py
How to generate image with your fine-tuned weight
- Run 'inference.py' using the command shown below, after loading your fine-tuned weight to genetate images by specifying the prompt.
- python inference.py