Added chapter on proportionality constants (Issue #287) #288

bbbales2 · 2020-11-17T20:59:04Z

Submission Checklist

Builds locally
Declare copyright holder and open-source license: see below

Summary

@rok-cesnovar @jgabry Can both of you have a look at this?

I'm not sold on the organization yet, and we need to make sure we really want to do this as a separate chapter (the chapter titles need to stay compatible into the future cause this is how links pointing to the old docs are forwarded to the new docs)

I was realizing when I wrote this that maybe I should have done the reference manual version of this first. I'm not sure how much of what I wrote here is duplicate or what.

Copyright and Licensing

Issue: #287

Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company): Columbia University

By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

rok-cesnovar

Review!

rok-cesnovar · 2020-11-19T12:29:31Z

src/stan-users-guide/proportionality-constants.Rmd

+
+## User-defined Distributions
+
+While it is possible to define custom distributions in Stan, it is not yet


Not sure we need to say "not yet". No one is currently thinking about/working on allowing these. We might end up adding it soon, but if we don't this will be a dangling "soon" :)

Just mention that currently you always define a UDF as normalized but can call it unnormalized. Calling it as unnormalized will however only help with any calls to Stan-defined lpdfs/lpmfs.

bbbales2 · 2020-11-19T13:04:28Z

@rok-cesnovar @jgabry actually I forgot to ask.

proportionality constants/unnormalized distributions/additive constants (for the log)

What lingo should I be leaning towards? These are all kinda referring to the same thing, but it might be confusing to carelessly swap between all three (which I do).

rok-cesnovar · 2020-11-19T13:19:15Z

Not a native speaker nor a statistician by profession, so I have no clue which one is preffered so will let Jonah take this one.

jgabry · 2020-11-19T21:12:47Z

proportionality constants/unnormalized distributions/additive constants (for the log)

What lingo should I be leaning towards? These are all kinda referring to the same thing, but it might be confusing to carelessly swap between all three (which I do).

Yeah that's a good question. Maybe at the beginning just mention that all of these terms will be used? I think the reality is that it's pretty natural to use all of these terms and people will see all of them in different places already. Maybe just one sentence towards the beginning (if it's not there already) that connects these three concepts to make sure the reader is prepared, i.e., unnormalized distributions are those without the constant term and when working on the log scale (like Stan does) that results in additive rather than multiplicative constant.

That said, if you want to just pick one way to talk about it then that's fine too, but I think it will be tricky!

jgabry · 2020-11-19T21:15:12Z

src/stan-users-guide/proportionality-constants.Rmd

+compute the functions up to a proportionality constant (or similarly compute
+log densities up to an additive constant). In MCMC this comes from the fact that


In relation to the question you asked about which terminology to use, I think you've done a good job here! You use both "proportionality constant" and "additive constant" but you make it clear that additive is when working on the log scale.

jgabry · 2020-11-19T21:16:50Z

src/stan-users-guide/proportionality-constants.Rmd

+MCMC, variational inference, or optimization, it is usually only necessary to
+compute the functions up to a proportionality constant (or similarly compute
+log densities up to an additive constant). In MCMC this comes from the fact that
+the distribution being sampled does not need to be normalized (similarly


Also in relation to your question about terminology, maybe just make the connection between "normalized" and the "constants" that you previously mentioned explicit here. Then I think you can freely use all the terms below and not worry about it.

jgabry · 2020-11-19T21:19:09Z

src/stan-users-guide/proportionality-constants.Rmd

+There are three different syntaxes to work with distributions in Stan. The way
+to select between them is by determining if the proportionality constants are
+necessary.


I wonder if it's worth mentioning the speed issue here. Because really the user shouldn't even bother thinking about using the unnormalized versions if they're not worried about the computation time, right? Plenty of models are fast enough that the user could ignore this and save themselves the time understanding the differences between the different versions. What do you think?

jgabry · 2020-11-19T21:20:45Z

src/stan-users-guide/proportionality-constants.Rmd

+
+```
+x ~ normal(0, 1);
+target += normal_lupdf(x | 0, 1);


The first time we show this syntax should we say that "u" stands for "unnormalized" to make the connection between the name and the purpose clear?

jgabry · 2020-11-19T21:22:16Z

In addition to my other comments: thanks for working on this!

bbbales2 · 2020-11-20T19:49:55Z

Edits made. I'm still not totally happy with how this renders (there's a separate page for each section and it's not obvious how they feed into each other).

As a follow up, where in the reference manual should we add stuff? May as well be part of this pull request. I guess I add a couple sections here: https://mc-stan.org/docs/2_25/reference-manual/increment-log-prob-section.html ?

jgabry · 2020-11-20T22:06:07Z

Edits made. I'm still not totally happy with how this renders (there's a separate page for each section and it's not obvious how they feed into each other).

Is whether it creates a separate page or not determined by the header level (is that the right terminology?), that is how many # there are?

As a follow up, where in the reference manual should we add stuff? May as well be part of this pull request. I guess I add a couple sections here: https://mc-stan.org/docs/2_25/reference-manual/increment-log-prob-section.html ?

Yeah that seems like a good spot. Maybe also a very quick mention of lupdf in the section after that on sampling statements since they're equivalent?

bbbales2 · 2020-11-30T21:10:13Z

I added docs to the reference manual for lupdf and lupmf. @rok-cesnovar @jgabry have a look at these.

rok-cesnovar · 2020-12-02T19:15:28Z

src/reference-manual/expressions.Rmd

@@ -156,7 +156,9 @@ or `add`, but it may be named `pi` or `e`.
 Variable names will also conflict with the names of distributions
 suffixed with `_lpdf`, `_lpmf`, `_lcdf`, and `_lccdf`, `_cdf`, and
 `_ccdf`, such as `normal_lcdf_log`; this also holds for the deprecated
-forms `_log`, `_cdf_log`, and `_ccdf_log`.
+forms `_log`, `_cdf_log`, and `_ccdf_log`. No user-defined variable


Hm, we need to revise this section actually. As of 2.25 Stan Math function names are allowed as identifiers. The only restriction is that identifier can not end with _lupdf/_lupmf

So allowed: add, normal_lpdf, sum,...
Not allowed: normal_lupdf, p_lupmf

Feel free to push directly into this

jgabry

I made a few comments, but this looks good. Thanks @bbbales2!

src/reference-manual/statements.Rmd

jgabry · 2020-12-02T23:57:40Z

src/reference-manual/statements.Rmd

+The `normal_lupdf` function returns the log density of an unnormalized distribution.
+With the unnormalized version of the function, Stan does not define what the
+normalization constant will be, though usually as many terms as possible are dropped
+to make the calculation fast. One possible definition of `normal_lupdf` is:


Why "One possible definition of" and not "The definition of"? Is this saying that Stan has multiple possible versions of normal_lupdf?

I changed this to try to make it less ambiguous

src/reference-manual/statements.Rmd

jgabry · 2020-12-03T00:05:13Z

src/reference-manual/user-functions.Rmd

+Other `_lupdf` and `_lupmf` functions used in the definition of
+`foo_lpdf` will drop additive constants. If there are no `_lupdf`


I might just be confused (as is often the case!) but it says just above that

if any other unnormalized density functions are used inside the user-defined function, the _lpdf and _lpmf forms of the user-defined function will change these densities to be normalized

Does that contradict this?

I think this is right but I rewrote it to make it clearer.

…rtionality_constants

bbbales2 · 2021-02-11T14:44:48Z

@jgabry I must have just forgotten documentation existed sometime in December. This is the second piece of nearly finished doc that was just chilling.

Added chapter on proportionality constants (Issue #287)

7379a7f

rok-cesnovar self-requested a review November 17, 2020 21:31

rok-cesnovar reviewed Nov 19, 2020

View reviewed changes

jgabry reviewed Nov 19, 2020

View reviewed changes

Responding to reviews (Issue #287)

782e8c9

More documentation

d406e4c

rok-cesnovar reviewed Dec 2, 2020

View reviewed changes

jgabry reviewed Dec 3, 2020

View reviewed changes

bbbales2 added 2 commits February 11, 2021 09:24

Changed text a bit to try to clarify some things

c2a713c

Merge remote-tracking branch 'origin/master' into feature/lupxf_propo…

a3e46a8

…rtionality_constants

jgabry approved these changes Mar 4, 2021

View reviewed changes

bbbales2 merged commit d8800a6 into master Mar 4, 2021

rok-cesnovar mentioned this pull request Mar 30, 2021

Docs for lupmf/lupdf #287

Closed

WardBrian deleted the feature/lupxf_proportionality_constants branch March 28, 2022 13:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added chapter on proportionality constants (Issue #287) #288

Added chapter on proportionality constants (Issue #287) #288

bbbales2 commented Nov 17, 2020

rok-cesnovar left a comment

rok-cesnovar Nov 19, 2020

rok-cesnovar Nov 19, 2020

bbbales2 commented Nov 19, 2020

rok-cesnovar commented Nov 19, 2020

jgabry commented Nov 19, 2020

jgabry Nov 19, 2020

jgabry Nov 19, 2020

jgabry Nov 19, 2020

jgabry Nov 19, 2020

jgabry commented Nov 19, 2020

bbbales2 commented Nov 20, 2020

jgabry commented Nov 20, 2020

bbbales2 commented Nov 30, 2020

rok-cesnovar Dec 2, 2020 •

edited

bbbales2 Dec 2, 2020

rok-cesnovar Dec 2, 2020

jgabry left a comment

jgabry Dec 2, 2020

bbbales2 Feb 11, 2021

jgabry Dec 3, 2020

bbbales2 Feb 11, 2021

bbbales2 commented Feb 11, 2021


		## User-defined Distributions

		While it is possible to define custom distributions in Stan, it is not yet

		compute the functions up to a proportionality constant (or similarly compute
		log densities up to an additive constant). In MCMC this comes from the fact that

		Other `_lupdf` and `_lupmf` functions used in the definition of
		`foo_lpdf` will drop additive constants. If there are no `_lupdf`

Added chapter on proportionality constants (Issue #287) #288

Added chapter on proportionality constants (Issue #287) #288

Conversation

bbbales2 commented Nov 17, 2020

Submission Checklist

Summary

Copyright and Licensing

rok-cesnovar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bbbales2 commented Nov 19, 2020

rok-cesnovar commented Nov 19, 2020

jgabry commented Nov 19, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgabry commented Nov 19, 2020

bbbales2 commented Nov 20, 2020

jgabry commented Nov 20, 2020

bbbales2 commented Nov 30, 2020

rok-cesnovar Dec 2, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgabry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bbbales2 commented Feb 11, 2021

rok-cesnovar Dec 2, 2020 •

edited