Question about AR(1) noise variables

[cboulay]

Hi there,

Thanks so much for making this repo public. It's really helped me understand a lot of the inner workings of LFADS as well as the CD and other recent enhancements.

When going over the LearnableAutoRegressive1Prior, I'm a little bit confused by something. It's quite possible I don't have the prerequisite knowledge so I apologize that this question might just be the result of my ignorance.

I hope you don't mind but I'll use the wikipedia nomenclature found here.

So the process model takes the form:
E(X_t) = E(c) + phi * E(X_{t-1}) + e_t
c is a constant, typically 0, and e_t is white noise at time t.

When there are no previous samples,
E(X_t) = E(c) + e_t
which, I believe, is a normal distribution: N(c, sigma_e**2)

When there is a previous sample, we have a normal distribution: N(c + phi * prev, sigma_p**2),
where prev is a draw from X_{t-1} and the combined variance sigma_p**2 = phi**2 * var(X_{t-1}) + sigma_e**2, or equivalently
sigma_p**2 = sigma_e**2 / (1 - phi**2).

In your code, I think sigma_e**2 is stored in the more tractable logevars and sigma_p**2 in logpvars, cofirmed by the fact that logpvars is a transformation of logevars and phis, here:

lfads-cd/helper_funcs.py

Lines 364 to 365 in 1d6bb5e

	self.logpvars_1xu = \
	logevars_1xu - tf.log(1.0-phis_1xu) - tf.log(1.0+phis_1xu)

So then later, in the logp_t method, I would expect the 0th-sample branch to use logevars and the >=1th sample branch to use logpvars, but it seems the opposite is the case:

lfads-cd/helper_funcs.py

Lines 386 to 393 in 1d6bb5e

	if z_tm1_bxu is None:
	logp_tgtm1_bxu = diag_gaussian_log_likelihood(z_t_bxu, self.pmeans_bxu,
	self.logpvars_bxu)
	else:
	means_t_bxu = self.pmeans_bxu + self.phis_bxu * z_tm1_bxu
	logp_tgtm1_bxu = diag_gaussian_log_likelihood(z_t_bxu,
	means_t_bxu,
	self.logevars_bxu)

I'm inclined to think I'm misunderstanding something, but I suppose it's also possible that there are a couple typos here e <-> p, so I wanted to check with you first.

Cheers, and thanks again for this great repo!