Continuous target

[bkj]

Hi Eder,

Thanks for the Keras implementation! I'm excited about this idea as well. (Hopefully) quick question: I'm interested in learning an autoencoder for the output features of some other CNN (VGG16), so I'd like to maximize the cosine similarity of x and x_hat. Presumably this means I have to change

return data_dim * binary_crossentropy(x, x_hat) - KL 

to something else, since it doesn't seem like binary_crossentropy makes sense in this context. Any thoughts on what it should be changed to? (I'm new to variational methods, so I'm trying to derive the answer myself right now, but thought I'd post the question here as well..)

Thanks!
Ben

[EderSantana]

Hi @bkj,
The VAE losses are based on the KL divergence between the distribution of output and target. So you have to have a pdf assumption. BCE is the KL divergence for Bernoulli (binary) distributions. Categorical crossentropy is the one for multinomial distributions. If Gaussian is a good approximation for you case, there is a well define loss there as well…

That being said, try to just plugin your cost function and see if it works xD
it may not be statistically sound, but if it works, you're good. Just don't tell statisticians I suggested that for the sake of my reputation ahhahaha

[bkj]

Alright thanks -- I'll give that a shot, but I won't tell anyone... I'm not sure whether Gaussian would be a good approximation in my case -- since there are image featurizations, I would expect them to be multimodal and would think Gaussian is a bad fit for that reason, but I don't have much intuition about these methods.

Another issue: In Jang's implementation, he is maximizing the ELBO:

loss=tf.reduce_mean(-elbo)

but it doesn't seem like you're doing that here. Wouldn't you want to change gumbel_loss to:

def gumbel_loss(x, x_hat):
    q_y = K.reshape(logits_y, (-1, N, M))
    q_y = softmax(q_y)
    log_q_y = K.log(q_y + 1e-20)
    kl_tmp = q_y * (log_q_y - K.log(1.0/M))
    KL = K.sum(kl_tmp, axis=(1, 2))
    elbo = data_dim * bce(x, x_hat) + KL  # ** changed `-` to `+`
    return elbo

if you're minimizing that function? (This is also more in line w/ adding KL to BCE as in https://github.com/fchollet/keras/blob/master/examples/variational_autoencoder.py)

[EderSantana]

Yeah, if the distribution is multimodal, then MSE and Gaussian assumptions are not really good. The idea of the using a proper cost functions is to make sure the amount of contributions of the reconstruction loss (in this case BCE) and the KL regularization to the ELBO are correctly balanced. You can always play with weighting the contributions of each term to the cost function. But then you may have an unnecessary hyperparameter. Just plug and play while you look for a maximum likelihood approach that reduces the loss to one close to what you want to use. This way you get that new loss and apply here. For example, for MSE with zero mean and unity variance assumption, the contributions would be data_dim *0.5*MSE + KL. That 0.5 comes from the log of the Gaussian pdf once you ignore all constant terms.

good catch! yeah, please double check if the signs are correct. I'll review the PR.