Surprising results with no convergence

[dataislife]

Hi there,
I am training this VAE on top of an input space of size 2048 (Sentence embedding space), I am trying to tweak the parameters in order to reconstruct the input space correctly but I cannot make it converge.

With config as follows:

config = """
latent_size: 128
variational: flow
flow_depth: 2
data_size: 2048
learning_rate: 0.00001
batch_size: 128
test_batch_size: 64
max_iterations: 100000
log_interval: 1000
early_stopping_interval: 5
n_samples: 128
use_gpu: false
train_dir: $TMPDIR
data_dir: $TMPDIR
seed: 582838
"""

It have the following results:

step:   0       train elbo: -1458.64
step:   0               valid elbo: -1452.85    valid log p(x): -1437.75
step:   1000    train elbo: 175476434.95
step:   1000            valid elbo: 177791955.55        valid log p(x): 177795714.31
step:   2000    train elbo: nan
step:   2000            valid elbo: nan valid log p(x): nan

I thought about reducing the learning rate, changing the size of the latent space (I don't know whether increasing it or decreasing it is better in that case where input_dim = 2048), changing batch size, but nothing seems to be conclusive.

Also, why is the log-likelihood getting such high values? (p(x) should be in [0,1] ...)

Any idea on this matter? :)
Thanks for the great work.

[jaanli]

Thanks @dataislife - mind emailing me a description of the model and sentence embedding data you are using? This is an interesting application and probably easiest over email :)

[garenkwan]

Hi, did you ever resolve this? I am facing a similar issue and cannot get the it to converge.

[jaanli]

Thanks @garenkwan - can you email me a description of the problem + data? I'm at j@jaan.io.

(This is not designed for training on sentence embeddings, but I can try to help!)

[garenkwan]

Thanks for the comment @altosaar . I realized that my inputs and sentence embeddings use both positive and negative values in the input, however, I overlooked that fact that you have designed this for working on images which have inputs that are always positive and between 0 and 1.

To avoid the loss from blowing up, I rescaled the input values to be between 0 and 1 and it worked great!
But I've not tested on values greater than 1.