Overloading the sampling functions of probability distributions

[damdaepark]

Hi,

I'm trying to overload some sampling functions of probability distributions according to Jankowiak(2018) http://arxiv.org/abs/1806.01851

Fortunately, Matlab already supports functions like gamrnd so what I have to do is just overloading those functions via modifying Layer.m and autonn_der.m

However, as you can see in the equation (58) of the paper, it requires to get the original output value of the overloaded function, which is represented as z in the paper. It seems like that all the functions in autonn_der.m are currently having a form of dx = function(x, dy). I don't know how to pass the output value (y) and change it to have a form of dx = function(x, y, dy)

Can you give me a hint about how I can give the output value in the backpropagation stage?

[jotaf98]

Hi, first of all instead of overloading Layer by modifying its source code, you could also use the convenience function Layer.fromFunction. It will return a Layer generator from your function (i.e. equivalent to overloading by modifying the source). There's a detailed guide in the docs (help Layer.fromFunction).

As for saving the output, there are a few work-arounds:

1. Recompute the forward pass in your backward function to get it (while it seems wasteful, if this is not the bottleneck of your network then it might be ok, and easier than any alternative).

2. Save it using the persistent keyword (check official Matlab docs). If you can guarantee only one instance of this layer, this will work without any complication. If there's many, you'll want separate state variables for each one; this can be implemented with the same persistent variable but with some additional bookkeeping to ensure it accesses a different state for each layer (I don't recommend it though).

3. Many layers with state can actually be implemented in a state-less way if you decompose them into smaller layers. For example, DropOut would need state, but in autonn it's broken up in 2: one is the mask generator (samples random masks), the other is the dropout layer that just applies the mask. So what would normally have to be stored as state, is now just a simple variable passed from one layer to another. It might take some creativity and I realize it might not apply when you're stuck with someone else's monolithic function, but if possible this is the cleanest choice in my opinion.

[damdaepark]

Wow, I didn't expect to get an answer this quick.

What a neat solution. I tried Option 1, and it seems to work well so far!

Thanks!