Add support and tests for standalone generated quantities

[ariddell]

See stan-dev/stan#2434 . If there's an RStan example already PyStan could borrow that.

[mathDR]

Does this now work in 2.18? Is there an example showing this anywhere?

[bob-carpenter]

They're not in RStan yet. I'm not sure about CmdStan---@mitzimorris was going to plumb them through, but I don't know if that ever happened.

[mitzimorris]

not plumbed through yet. good intentions, but no promises.

[lukerm]

This is a really interesting feature that I'd love to see in PyStan or RStan. It would allow us to see how the fitted model performs on "test" data.

Is there realistically anything I can do to help push this forward?

[mitzimorris]

It would allow us to see how the fitted model performs on "test" data.

how would you use this feature to do this?

this feature will be easy to plumb through the interfaces as soon as stan-dev/stan#2725 is merged into stan.

[bob-carpenter]

If you have posterior draws from p(theta | y, x) for data y and predictors x, then you can plug them into generated quantities to estimate p(y' | y, x, x') where x' are predictors for new data y'. Specifically, you need to calculate log p(y' | theta, x') in generated quantities, then use log_sum_exp to get log p(y' | y). I have an example of how to do that using generated quantities in my repeated binary trials case study.

…

On Jan 16, 2019, at 10:32 AM, Mitzi Morris ***@***.***> wrote: It would allow us to see how the fitted model performs on "test" data. how would you use this feature to do this? this feature will be easy to plumb through the interfaces as soon as stan-dev/stan#2725 is merged into stan. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[lukerm]

I have an example of how to do that using generated quantities in my repeated binary trials case study.

@bob-carpenter I think you're referring to this (interesting!) study? I see what you mean about having K, y and K_new, y_new (as train, test resp.).

Perhaps it's the lazy in me, but if you're interested in the generated quantities for both train and test datasets, then you'll end up repeating a lot of code in your Stan program. That's the situation I'm in. Even just for y_rep and y_rep_new it's a hassle (I think). However, if you could run only the generated quantities part of a trained Stan model, then all you'd need to do is pass in the test / unseen data. That would remove the need for _new suffix. I hope that makes sense.

Anyway, that's why I'm excited about this feature. I'll keep on the lookout for updates.

[mathDR]

Bumping this. Was this implemented?

[mitzimorris]

not sure about PyStan, but it's available in CmdStanPy - there's doc and a notebook - https://cmdstanpy.readthedocs.io/en/latest/generate_quantities.html