Is it the skeleton convolution making the unpaired training process possible?

[ANYMS-A]

Hi, I recently tried to transfer the facial animations(facial expression) between 2 person. Since the data is not paired, I followed your methodology to set a reconstruction loss and latent consistency loss as the training objective, but I didn't use the skeleton convolution operator because the data are quite different. However, the results are not really good, I'd like to know does the skeleton convolution are the key factor of this unsupervised training process?

Many thanks!

[kfiraberman]

The skeleton convolution is a key factor in transferring motions between two datasets that have two different skeletal structure. The ability to use skeleton-aware operators (conv+pooling) enables us to find a shared domain that is agnostic to the skeletal structure. In your case, I assume that the facial landmarks contain the same number of points in the two datasets, so I don't think that it's a key factor.

[ANYMS-A]

The skeleton convolution is a key factor in transferring motions between two datasets that have two different skeletal structure. The ability to use skeleton-aware operators (conv+pooling) enables us to find a shared domain that is agnostic to the skeletal structure. In your case, I assume that the facial landmarks contain the same number of points in the two datasets, so I don't think that it's a key factor.

Many thanks! This is exactly what I think.

Since your training strategy is similar to "Cycle GAN", I think I should add 2 discriminators to the training process.

In the ablation study part of your paper, it states that you discard the adversarial loss and " It can be seen that in the intra-structural setting omitting the adversarial loss actually improves performance." I'd like to know that does "discarding adversarial loss" means that you also discard the 2 discriminators too?

Because in my case, I only use the reconstruction loss and latent consistency loss, the discriminators are not added yet. Those 2 loss converged, but the results are not quite good, so I thought this "unpaired" training strategy is the most difficult part.

[kfiraberman]

Right, it means that we discarded the discriminators too, but note that we have the EE loss that helps to stabilize the transfer.

[ANYMS-A]

Right, it means that we discarded the discriminators too, but note that we have the EE loss that helps to stabilize the transfer.

Yep, I think your feature engineering and loss designing is quite great. Maybe it helps for transfer.

[ANYMS-A]

Right, it means that we discarded the discriminators too, but note that we have the EE loss that helps to stabilize the transfer.

Dose the end-effector means the "head", "hand", "feet" those joints which are at the end of the skeleton graph?

[PeizhuoLi]

Dose the end-effector means the "head", "hand", "feet" those joints which are at the end of the skeleton graph?

Exactly! More specifically, the joints with degree one in the corresponding skeleton graph.