How does the inference command work for clothing transfer, exactly?

[0xymoro]

Hi, I've been trying to get this transfer code to work:

python inference.py --checkpoint checkpoints/deep_fashion
--cloth_dir [SOURCE] --texture_dir [SOURCE] --body_dir [TARGET]

For each of [SOURCE] and [TARGET], I'm assuming they have their set of cloth/texture/body/stats within them (I saw it was looking for the stats json file in each directory). However, when I tried to do a sanity check with the following command:

python inference.py --checkpoint checkpoints/deep_fashion
--cloth_dir ./data/deep_fashion/ --texture_dir ./data/deep_fashion/ --body_dir ./data/deep_fashion/

It had an error: KeyError: Caught KeyError in DataLoader worker process 0.

I haven't gotten through the data preprocessing since that's a lot of setup, but my question is when the transfer happens, does it match every style in the SOURCE directory with the body in the TARGET? So if I have n elements in SOURCE and m in TARGET, the results would be a n * m list of all the styles in SOURCE applied to every one in TARGET? Very new to the code so appreciate any help!

[andrewjong]

Hi Jerry, thanks for your question.

---

Regarding a missing stats file, check out this part of the Readme. I forgot to upload that file with the dataset, and haven't gotten around to reuploading the whole thing to add the single file yet.

---

For your question about what clothing gets transferred, at the moment, by default it does a one-to-one mapping based on the image index i (sorted alphabetically). So the model will use the clothes_i (Source) to transfer onto body_i (Target). As long as the number of files in Source >= number of files in Target, it should run fine.

In the first sanity check in the Readme, we use this:

python inference.py --checkpoint checkpoints/deep_fashion \
  --dataroot data/deep_fashion \
  --shuffle_data True

to perform inference on the DeepFashion dataset itself. The default one-to-one mapping would mean we'd transfer the clothing onto the same body it belonged to originally. This doesn't make much sense, so we use the --shuffle_data option so that clothes get transferred to random bodies.

For the inference command you mentioned,

python inference.py --checkpoint checkpoints/deep_fashion \
  --cloth_dir [SOURCE] --texture_dir [SOURCE] --body_dir [TARGET]
this is to control which cloth (Source) we want to transfer to what body (Target). 

The first important thing is that if you use the --texture_dir, --cloth_dir, --body_dir options individually, they should point to dataroot/texture, dataroot/cloth, dataroot/body/ folders respectively. They do not each point to just dataroot.

The reason these inputs are folders instead of single images is in case you want to process a video. You would put all the frames from a video in the same folder. If you want to transfer a single image, then just put a single image in each folder.

Hope that helps a bit. I know it's bit complicated, please let me know if you have more questions.

[andrewjong]

Another important thing to note about SwapNet's clothing transfer - it's not exactly "transfer" (at least I don't think so). It's more like: learn to repose the source into any arbitrary pose; during inference, just repose the clothes to match that of the target body. The identity of the target user is not preserved at all, and SwapNet resorts to copying over the face pixels from the target to "preserve" identity (I haven't implemented this part yet, see #6). There has definitely been better clothing transfer works published since.

[0xymoro]

I see, this makes a lot of sense (I was wondering why the target body directory didn't need a textures directory too - I guess it just doesn't use them if the target isn't preserved since it just needs pose info).

I got the basic inference to work well, but would you know of any open-source code that I can build on for those better transfer networks? Would love to hop on a call to talk more about the topic as well since I'm relatively new to this area of research - you can reach me at zmeng90@stanford.edu. Thanks!

[andrewjong]

Hi Jerry, we can connect on LinkedIn and set something up. I'm https://www.linkedin.com/in/ajong/. Cheers.