Add unnormalized distributions to the language

[VMatthijs]

Based on discussions with @seantalts and @bob-carpenter :

For each distribution, e.g. poisson_lpmf, make the compiler accept the function name poisson_unnormalized_lpmf as well.
This should code gen to poisson_lpmf<true>.
This allows us to treat
x ~ poisson(42);
as syntactic sugar for
target += poisson_unnormalized_lpmf(x| 42);.

This is related to issue stan-dev/stan#1299 .

Also make sure that algebraic optimizations like bernoulli_lpmf(x|inv_logit(theta))->bernoulli_logit_lpmf(x|theta) and automatic vectorization optimizations apply to these unnormalized distributions as well.

[seantalts]

just hitting this now - should this actually be inverted? We have a propto__ type parameter to log_prob and it seems like sometimes explicitly want to set it to false rather than sometimes explicitly setting it to true (false is the expensive, more accurate case). So perhaps poisson_normalized_lpmf? Or what about poisson_lpmf_normalized?

[bob-carpenter]

_lpmf should remain the suffix to keep the name mangling rules and doc simple.

I think we should support marked cases and leave the unmarked cases the same. So the following two groups would contain equivalent statements, leading off with our existing forms.

y ~ foo(theta);

y ~ foo_propto(theta);
target += foo_propto_lpdf(y | theta);

target += foo_lpdf(y | theta);

y ~ foo_normalized(theta);
target += foo_normalized_lpdf(y | theta);

You can think of the first statement in each of these two groups as syntactic sugar and the ~ statements can now be fully defined as syntactic sugar for target +=.

I could be convinced that propto should be replaced with unnormalized, but it's a handful to type.

[seantalts]

Oh, for supporting these for users? Sure, makes sense.

I'm having trouble parsing the two groups. Are you saying y ~ foo(theta) = target += foo_lpdf(y | theta) and so on, line 1 = line 1, line 2= line 2, line 3 = line 3? Or also that line 1 = line 2 = line 3?

[seantalts]

@VMatthijs likes "_unnormalized" and Bob likes "_propto." Unfortunately I don't really care - propto is nice in that it's what we use in code gen

[VMatthijs]

I don't really care either. Let's go with _propto.

[VMatthijs]

I don't think these have been exposed to the language yet. They're in the backend now, but the type checker should still be made to accept them.

[seantalts]

Yep, sorry about that.

[bob-carpenter]

Our current y ~ foo(theta) is propto/unnormalized. Our current target += foo_lpdf(y | theta) is normalized. The ones grouped together are equivalent. I don't really care so much about propto vs. unnormalized. I can see arguments either way. We should ask a wider group of statisticians about what they think is natural.

[seantalts]

Agreed! I'll write up a discourse post.

[seantalts]

poll! https://discourse.mc-stan.org/t/statisticians-propto-vs-unnormalized-vs/9054

[VMatthijs]

Just a reminder to ourselves that whatever ends up being implemented should be documented in the wiki. (And then in the manual when we are ready to release.)

[seantalts]

Does anyone remember why we wanted to expose this to the end users in the language? Was there a motivating use-case for when you needed to use target += foo but didn't want it to be normalized?

[VMatthijs]

@bob-carpenter , weren't you concerned about the normalizing constant of a gamma distribution being expensive to compute or something like that?

[bob-carpenter]

Yes, in general there's a lot of expensive computation that's avoided if we can do unnormalized/propto computations. But then in some contexts, like mixtures, we need the normalized forms. Also, if we want to reduce sampling statements to target increment statements, then we need this flexibility to be able to actually implement the sampling statements as they're currently implemented (unnormalized, that is).

[seantalts]

Also, if we want to reduce sampling statements to target increment statements, then we need this flexibility to be able to actually implement the sampling statements as they're currently implemented (unnormalized, that is).

We don't actually need this exposed to end users to do that, right?

Yes, in general there's a lot of expensive computation that's avoided if we can do unnormalized/propto computations. But then in some contexts, like mixtures, we need the normalized forms.

Agreed, but are there cases where you can't use ~ but need to get the propto=true version?

In other words, are there specific motivating examples for exposing this to end users of the Stan language?

[rok-cesnovar]

Superseed by #253 and #541. Closing.