This is the implementation of FashionDPO (Accepted by SIGIR'25).
FashionDPO:Fine-tune Fashion Outfit Generation Model using Direct Preference Optimization
Mingzhe Yu, Yunshan Ma, Lei Wu, Changshuo Wang, Xue Li, Lei Meng
- Environment
- Datasets
- Fashion Image Generation without Feedback
- Feedback Generation from Multiple Experts
- Model Fine-tuning with Direct Preference Optimization
- Release checkpoint
Clone this repository:
git clone https://github.com/Yzcreator/FashionDPO.git
cd ./FashionDPO/
Install PyTorch and other dependencies:
conda env create -f fashiondpo_environment.yml
conda activate FashionDPO
We follow the previous work DiFashion and use the datasets of iFashion and Polyvore-U, which include the required data of both fashion outfit and user-fashion item interactions.
To simplify the process of handling and merging the datasets, we provide the complete datasets for DiFashion and FashionDPO, which can be downloaded via the Google Drive link.
During the sampling phase, this code generates 7 recommended items for each outfit as a candidate set.
cd ./fashiondpo
python sample.py
If it is the first round of sampling in the iterative process, set args.resume=False, which will create the LoRA layers in the pre-trained model. In subsequent rounds, set args.resume=True to load the fine-tuned LoRA layers from the model specified in args.pretrained_model_name_or_path.
We locally deploy MiniCPM to evaluate "Quality". For "Compatibility", we train a VBPR model using paired outfit data from the POG and Polyvore-U dataset. "Personalization" is evaluated using the CLIP Score.
cd ./evaluation
python multiple_evaluate.py
This contains three evaluating perspectives:
- Quality: Loading the per-trained MiniCPM-Llama3-V 2.5. The prompt:
Consider whether the fashion elements in the image are complete and whether they conform to fashion design principles. The goal is to classify the quality into one of the following categories: 1-Very Poor Quality, 2-Poor Quality, 3-Low Quality, 4-Below Average Quality, 5-Moderate Quality, 6-Above Average Quality, 7-Good Quality, 8-Very Good Quality, 9-High Quality, 10-Exceptional Quality. Please provide the best possible category based on the available information.
Set aegs.evanumber=5 or 10 to specify the number of classification levels to be used in the MiniCPM evaluation.
-
Compatibility: We trained the VBPR model using the iFashion dataset. The checkpoint is avaliable at vbpr_compatibility. If you want to retrain the model, please refer to
train_VBPR.py. -
Personalization: We use the pre-trained CLIP (ViT-B/32) to encode the generated fashion items and user interaction history image items, and calculate the CLIP Score between them.
Normalize the scores and make preference judgments to construct positive-negative pairs:
cd ./evaluation
python analyse.py
In each outfit, if any two items from the 7 candidate generated items form a preference-non-preference relationship, we perform LoRA fine-tuning on the 50 saved timesteps:
cd ./fashiondpo
python finetune.py
The checkpoints are released via the huggingface repo. checkpoint_ifashion is fine-tuned on the iFashion dataset, while checkpoint_polyvore is fine-tuned on the Polyvore dataset.