module.log_prior is -inf

[jordantkohn]

I'm implementing a BayesianLinear layer implemented like this:
self.dense = BayesianLinear(opt.hidden_dim, opt.polarities_dim, freeze = False)

my loss from model.sample_elbo() returns "inf", and more specifically the module.log_prior() is "-inf".
What could be causing this issue?

bayesian module: BayesianLinear(
(weight_sampler): TrainableRandomDistribution()
(bias_sampler): TrainableRandomDistribution()
(weight_prior_dist): PriorWeightDistribution()
(bias_prior_dist): PriorWeightDistribution()
)
log_vp: tensor(-1425.3866, grad_fn=)
log_prior: tensor(-inf, grad_fn=)'

[piEsposito]

Hi, please send me your snippet to reproduce the possible bug. Em dom., 5 de dez. de 2021 às 18:31, Jordan Kohn ***@***.***> escreveu:

I'm implementing a BayesianLinear layer implemented like this: self.dense = BayesianLinear(opt.hidden_dim, opt.polarities_dim, freeze = False) my loss from model.sample_elbo() returns "inf", and more specifically the module.log_prior() is "-inf". What could be causing this issue? bayesian module: BayesianLinear( (weight_sampler): TrainableRandomDistribution() (bias_sampler): TrainableRandomDistribution() (weight_prior_dist): PriorWeightDistribution() (bias_prior_dist): PriorWeightDistribution() ) log_vp: tensor(-1425.3866, grad_fn=) log_prior: tensor(-inf, grad_fn=)' — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#98>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLYRXR3DOEJNKTECWQZYODUPPKZTANCNFSM5JNHHTPA> .

-- -- *Pi Esposito | piesposito.github.io <http://piesposito.github.io>*

[jordantkohn]

It's not really a self-contained example. But I've traced the -inf value back to this specific call in BayesianLinear module:

module.weight_prior_dist.log_prior(w)

[jordantkohn]

There are negative elements in w, which leads to 0 elements in prior_pdf.
This causes -inf elements after taking the log of prior_pdf tensor.