Training on Piano Roll data

[rafaelvalle]

Hey Jun-Yan, thanks for putting this repo together.
I'm trying to train it on piano roll data and have been seeing unexpected behavior: the generator outputs the same image even though the conditions, i.e. noise vector and real A, change.

Any thoughts on what it could be? I've added the loss log, options, output images during training and output images during inference.

loss_log.txt
opt.txt

Model outputs during training(fake_b_encoded, fake_b_random, real_a_encoded,real_b_encoded)

model.set_input(data)
encode = False
z_samples = model.get_z_random(1, opt.nz)
real_A, fake_B, real_B = model.test(z_samples, encode=encode)

Model outputs during inference(real_a, real_b, fake_b)

[junyanz]

I am not familiar with the piano roll data. Is it for future prediction? The current generator architectures assume strong spatial alignment between input and output, which might not be the case of your application. It also seems that L1 reconstruction does not work for your application as most of the pixels are black.

[rafaelvalle]

Yes, it is for future prediction. Does the spatial alignment come from the concatenation of features?
With other architectures L1 didn't seem to help...

[junyanz]

Our generator architecture works well for spatially aligned data. It might be struggling with other types of data.

[rafaelvalle]

What in your generator's architecture makes it work well for spatially aligned data?

[junyanz]

The generator is a U-Net with many skip connections. Please see the original paper for more details. Also, check out Sec 3.2.1 in the pix2pix paper.

[rafaelvalle]

Thank you for pointing out the section in the paper. I'll close the issue.