ByteMorph: Benchmarking Instruction-Guided Image Editing
with Non-Rigid Motions

This repo is the official pytorch implementation of ByteMorph, include training, inference and evaluation.

📜 Requirements

• An NVIDIA GPU with CUDA support is required for inference.
  • We have tested on a single A100 and H100 GPU.
  • In our experiment, we used CUDA 12.4.
  • Feel free to visit Flux.1-dev for further details on environment.

🛠️ Dependencies and Installation

Clone the repository:

git clone https://github.com/ByteMorph/BM-code.git
cd BM-code

Installation Guide

We provide an requirements.txt file for setting up the environment.

Run the following command on your terminal:

# 1. Prepare conda environment
conda create -n bytemorph python=3.10

# 2. Activate the environment
conda activate bytemorph

# 3. Install dependencies
bash env_install.sh

🧱 Download Pretrained Models

We follow the implementation details in our paper and release pretrained weights of the Diffusion Transformer in this huggingface repository. After downloading, please put it under the pretrained_weights folder.

The Flux.1-dev VAE and DiT can be found here. The Google-T5 encoder can be found here. The CLIP encoder can be found here.

Please place them under ./pretrained_weights/.

Your file structure should look like this:

ByteMorph
|----...
|----pretrained_weights
  |----models--black-forest-labs--FLUX.1-dev
    |----flux1-dev.safetensors
    |----ae.safetensors
    |----...
  |----models--xlabs-ai--xflux
    |----...
  |----models--openai--clip-vit-large-patch14 
    |----...
  |----ByteMorpher
    |----dit.safetensors
|----...

Train and Inference

Using Command Line

cd ByteMorph
# Train
bash scripts/train/train.sh

# Inference
bash scripts/test/inference.sh

The config files for trainig and inference can be found in this file and this file.

The DeepSpeed config file for training is here.

Evaluation

Please visit this page.

License

This code is distributed under the FLUX.1-dev Non-Commercial License. See LICENSE.txt file for more information.

Acknowledgement

We would like to thank the contributors to the Flux.1-dev, x-flux, OminiControl, for their open-source research.

Disclaimer

Your access to and use of this dataset are at your own risk. We do not guarantee the accuracy of this dataset. The dataset is provided “as is” and we make no warranty or representation to you with respect to it and we expressly disclaim, and hereby expressly waive, all warranties, express, implied, statutory or otherwise. This includes, without limitation, warranties of quality, performance, merchantability or fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. In no event will we be liable to you on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this public license or use of the licensed material.The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability.