-
-
Notifications
You must be signed in to change notification settings - Fork 986
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bayesian neural nets #40
Comments
I like this and agree we need a compact way to write that up, currently it is much too contrived. Will spend some time on it to play with and see if I have any useful comments. |
I like the idea of def make_guide(fn, sites=None):
def guide(*args, **kwargs):
model_trace = poutine.block(poutine.trace(fn))(*args, **kwargs)
if sites is None:
sites = {name: name for name in model_trace.keys()}
for name in model_trace.keys():
if model_trace[name]["type"] == "sample" and name in sites:
pyro.sample(sites[name], make_site_posterior(model_trace[name], *args, **kwargs))
return guide
guide_dist = make_guide(pyro.random_module(mod, prior)) As written this generates mean-field guides, but you can write more sophisticated guides in a similar style. |
Riffing on this some more because I quite like it: there's no reason the parameter-lifting operation has to be pyro.random_module = lambda name, mod, prior: poutine.lift(pyro.module, prior)(name, mod) but now the same principles, as well as guide generators like the one above, can be applied to any stochastic function that has I'm not completely happy with this structure, though, because conceptually it would be nicer if, like the proposed Edit: Ok, I thought about the last problem some more. I don't think there's a way to do that in general, and it's not even a probabilistically coherent request because the joint distribution over However, suppose we happen to know that all execution traces of a stochastic function class LiftableFunction(nn.Module):
def __init__(self, fn, *args, **kwargs):
self.fn = fn
initial_trace = poutine.block(poutine.trace(fn))(*args, **kwargs)
for name in initial_trace.keys():
if initial_trace[name]["type"] == "param":
# XXX something like this? not exactly correct
setattr(self, "_weight_" + name, nn.Parameter(pyro.param(name, ...)))
def forward(self, *args, **kwargs):
return self.fn(*args, **kwargs) |
i like the idea of a more general "lift" function that promotes params to samples! you're right that it couldn't cleanly separate the new randomness from the original randomness in the fn. it's not totally clear to me if this is an important separation. without that separation the bayesian nn example would look something like:
which is actually simpler! we've basically just upgraded the deterministic (but parameterized) function defined by the module to a stochastic function of the same signature. btw. the |
for future reference, a nice but pretty straightforward use of bayesian rnns: https://arxiv.org/pdf/1704.02798.pdf |
Addressed by #121 |
pyro should have a really beautiful, idiomatic way to describe bayesian neural nets.... currently the only way to do it is to construct the model net from raw tensors + tensor math (i.e. not using the predefined nn modules from pytorch). i've been thinking through some options.
here is the one i like best so far. This ony requires the addition of a
pyro.random_module(module, prior)
helper that intercepts the params ofmodule
and samples them withprior
instead of registering them as parameters.Comments on this approach, or alternatives, are welcome! If folks like this, then I or someone can add the helper and an example.
The text was updated successfully, but these errors were encountered: