Testing on Custom Image

[Djvnit]

Hi @cuiaiyu , Thanks a lot for this great work.
I'm working on Cloth Virtual Try-On as my Final Year Project.
The demo worked fine for me but currently I'm facing some issues in performing virtual try-on on my own image.
Steps I followed:

1. Resized by full size image .jpg to 750x1101 pixels (as all the images in test folder are of this dimension) and added it to test folder.

2. Ran openpose on the image and obtained the keypoints in .json file, manually separated x and y keypoints as (x0, y0, c0, x1, y1, c1, ....) and added the file name along with 2D_pose_keypoint y and x keypoints respectively in fasion-annotation-test.csv .

3. Using SCHP found the human parsing and added it to testM_lip.

4. Added image name in test.lip and standard_test_anns.txt under print just for testing.

5. After that I just ran the demo.ipynb and got the following error in data loading step.
   

I tried a lot to resolve this error but I'm unable to get it also I'm approaching the deadline. Kindly help me to test the model on custom image.

Also I'm unable to understand the use of fasion-pairs-test.csv while running demo.

Hopeful for your kind reply.
Thanks a lot Cuiaiyu !!!

Originally posted by @Djvnit in #21 (comment)

[cuiaiyu]

1. When you run openpose, if using COCO label, 18 joints are detected, but if you use the default label (Body_25), then you will have 25 joints. The original data preparation is using 18 joints. That's probably the reason of dimension mismatch (18 and 25).
2. fashion-paris-test.csv is the train/val split and pose pairs.

[Djvnit]

1. while running openpose in the parameter default label( Body_25) is only taken.
   Using: !cd openpose && ./build/examples/openpose/openpose.bin --image_dir /content/openpose/inp --body 1 --write_json /content/openpose/op/ --display 0 --render_pose 0
   I obtained 25 key points for x and y each which is giving the error as mentioned above.

2. Do I have to make any change in fashion-paris-test.csv just for testing the cloth try on on new images.

[cuiaiyu]

1. Please try option 2 in About pose keypoints #21 to load body_25 pose
2. No. For custom data, just be sure that (img, parse, kpt) are all loaded.

[Djvnit]

Thanks a lot @cuiaiyu for your kind and continuous support. Testing is done successfully.

[Djvnit]

Hello @cuiaiyu , I'm really sorry to interrupt you again, I think there is some issue with the key points generated using Openpose as the results obtained from SHP i.e. human parsing for custom image is perfectly fine but the final result of Try-On is not at all acceptable for custom image, which is great for the preprocessed data provided by you for demo.

Kindly help me to resolve this issue.

[cuiaiyu]

It would be easier to identify the problem if you can show the example images of the (pose, parse, img, generated_img).

[mahmoudmhashem]

I want to thank you for your great work.
Unfortunately, I have the same problem that @Djvnit faced.

here is the info you asked

for source image
pose: y=[29, 65, 62, 90, 115, 68, 100, 126, 148, 217, 254, 152, 228, -1, 22, 26, 26, 35]
x=[98, 86, 51, 19, 28, 121, 158, 122, 51, 40, 44, 94, 81, -1, 92, 107, 79, 112]
skeleton:

parse:

img:

for target image:
pose: y=[35, 75, 78, 117, 99, 72, 127, 172, 155, 223, 254, 162, 228, 254, 30, 29, 38, 35]
x=[94, 103, 77, 56, 31, 130, 130, 121, 64, 46, 64, 94, 79, 74, 90, 103, 85, 116]
skeleton:

parse:

img:

generated image:

thanks in advance

[cuiaiyu]

@m-h34 In the given example, the pose figure seems not aligned with the input image, as it is shifted a little bit left? If that is the case, please the input of the pose conversion (coord -> heatmap) function below. the target_size should be the size you want to the heatmap to be, and the orig_size should be the size of the input size. Misspecifying those may given you shifted pose heatmap.

def load_pose_from_json(pose_json, target_size=(256,256), orig_size=(256,256)):
    with open(pose_json, 'r') as f:
        anno = json.load(f)
    if len(anno['people']) < 1:
        a,b = target_size
        return torch.zeros((18,a,b))
    anno = list(anno['people'][0]['pose_keypoints_2d'])
    x = np.array(anno[1::3])
    y = np.array(anno[::3])
    
    x[8:-1] = x[9:]
    y = np.array(anno[::3])
    y[8:-1] = y[9:]
    x[x==0] = -1
    y[y==0] = -1
    coord = np.concatenate([x[:,None], y[:,None]], -1)
    pose  = pose_utils.cords_to_map(coord, target_size, orig_size)
    pose = np.transpose(pose,(2, 0, 1))
    pose = torch.Tensor(pose)
    return pose[:18]

[mahmoudmhashem]

thanks a lot for your attention
after I've edited the code according to your notes:
I found that output:

There is a big improvement but still there some problems with hands
I'm sorry for annoying you again

[cuiaiyu]

@m-h34 The hands issue is probably expected, as the training data doesn't specify finger joints in pose and this target pose is somehow a "rare pose" w.r.t. the training data.

Besides, 1) The missing sleeves problem could be an issue of overfitting. The checkpoints dior_64 is known for a little overfitted and gives sleeves artifacts like this, you may try other checkpoints like dior_32 or diorv1_64 to see if they work better. 2) the inaccurate face could possibly be a result of lack of face diversity in training data as discussed in Issue #32

[mahmoudmhashem]

Ok, I got it
Thanks a lot for your efforts

[Djvnit]

Hi @m-h34 it seems you have worked with 18 keypoints using openpose.
Can you please help me with key points generation I used 25 key points representation of pose using the command !cd openpose && ./build/examples/openpose/openpose.bin --image_dir /content/temp_inp --body 1 --write_json /content/temp_op/ --display 0 --render_pose 0 in openpose directory.

Just for the sake of testing i used the same image that is already parsed for testing in this repo. I just generated the keypoints using openpose and got something like this:
For target image:

Keypoints: (25 points)
X Component: [418, 418, 322, 298, 283, 515, 541, 517, 415, 355, 328, 352, 469, 475, 451, 397, 439, 373, 463, 442, 466, 442, 301, 298, 373]
Y Component: [115, 250, 250, 412, 553, 250, 415, 539, 536, 532, 782, 1013, 538, 782, 1001, 103, 106, 124, 127, 1058, 1050, 1019, 1064, 1046, 1025]

using these I m getting bad result

[Djvnit]

The source image for pose was taken as:

The parse for both the image is already present in testM_lip provided by @cuiaiyu

[mahmoudmhashem]

If your keypoints is stored as numpy array containing x, y coords then apply the following code to convert your keypoints_25 to keypoints_18

Index = [0,1,2,3,4,5,6,7,9,10,11,12,13,14,15,16,17,18]

Keypoints_18 = keypoints_25[index]
Then save to your CSV file

Also you can check this issue #21 option 2

And the return of this function "load_pose_from_json" should be passed in somehow to the variable "pose" in load_img function which is in setup cell in demo notebook
Good luck!

[Djvnit]

@cuiaiyu

@m-h34 In the given example, the pose figure seems not aligned with the input image, as it is shifted a little bit left? If that is the case, please the input of the pose conversion (coord -> heatmap) function below. the target_size should be the size you want to the heatmap to be, and the orig_size should be the size of the input size. Misspecifying those may given you shifted pose heatmap.

def load_pose_from_json(pose_json, target_size=(256,256), orig_size=(256,256)):
    with open(pose_json, 'r') as f:
        anno = json.load(f)
    if len(anno['people']) < 1:
        a,b = target_size
        return torch.zeros((18,a,b))
    anno = list(anno['people'][0]['pose_keypoints_2d'])
    x = np.array(anno[1::3])
    y = np.array(anno[::3])
    
    x[8:-1] = x[9:]
    y = np.array(anno[::3])
    y[8:-1] = y[9:]
    x[x==0] = -1
    y[y==0] = -1
    coord = np.concatenate([x[:,None], y[:,None]], -1)
    pose  = pose_utils.cords_to_map(coord, target_size, orig_size)
    pose = np.transpose(pose,(2, 0, 1))
    pose = torch.Tensor(pose)
    return pose[:18]

Thanks a lot @cuiaiyu I was also having target and original image size mismatching problem. Finally resolved and got the output by following your note.