Transform the MAF layer into an IAF layer

[mtsatsev]

Hi Edvard,

I need to invert a MAF layer (using another flow function and not an affine one), however I am having issues grasping the concept of it.
So:

1. When training normalizing flows to use the change of variables we actually use f^{-1} of the transformer function. For an affine function this is (x - mu) * e^{0.5 * sigma} (x - mu) * torch.exp(0.5 * logp) in your code.
2. When sampling from the distribution we use the forward function of the transformer. For an affine function like the one here that is done by u_i + mu_i * sigma_i. In your code that is mu[:, dim] + u[:, dim] * torch.exp(mod_logp[:, dim]).

Now if I invert that layer by saying forward = backward and backward=forward. Then I have f in the forward pass and f^{-1} in the backward. Is this the way to obtain an IAF?

[StephenCurry7]

I think what's you said is right. By the way, can you tell me how you run the 'test.py' code? The 'data.loader' module used in this code is not included in this github code, I don't know how to test the trained model.

[mtsatsev]

Unfortunately I did not run this code. I only looked at it for reference when building my own. I think this is just a function which loads the given dataset (as string) with the given batch_size. So your best bet is to write one yourself. Since he is testing with mnist you can just load mnist into a normal pytorch data loader with the given batch and test it.

[StephenCurry7]

Unfortunately I did not run this code. I only looked at it for reference when building my own. I think this is just a function which loads the given dataset (as string) with the given batch_size. So your best bet is to write one yourself. Since he is testing with mnist you can just load mnist into a normal pytorch data loader with the given batch and test it.

Thanks very much! I will try it my own.