update data_container example to pymc 5.6 #559

jessegrabowski · 2023-07-11T09:13:06Z

Update `how_to/data_container` to PyMC 5.6

Related to #333

The data container notebook is out of date. It uses PyMC3, and uses several APIs that are no longer needed/suggested/supported. Most prominent is using pm.Data instead of specifying pm.MutableData, but other small updates include:

Remove return_inferencedata=True from pm.sample
It is no longer necessary to make a custom group with az.from_pymc to add a predictions group; users should instead use the predictions=True flag in pm.sample_posterior_predictive
The same is also true about manually extending idata to include a posterior_predictive group; users should pass extend_idata=True instead
A referenced error that didn't allow setting a single observation has since been fixed

I also added a small explanation about the difference between pm.ConstantData and how to set mutable coords using model.add_coord. There might be a better way to do that last one.

📚 Documentation preview 📚: https://pymc-examples--559.org.readthedocs.build/en/559/

review-notebook-app · 2023-07-11T09:13:11Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

jessegrabowski · 2023-07-11T14:32:51Z

Just saw #527, I didn't even know this feature existed. I'll update this notebook to use it in the final example.

OriolAbril · 2023-07-11T15:12:22Z

Thanks! This is great, we get questions about this quite often

OriolAbril

also add yourself as author: https://www.pymc.io/projects/docs/en/latest/contributing/jupyter_style.html#jupyter-authors

examples/howto/data_container.myst.md

jessegrabowski · 2023-07-12T08:18:08Z

I refactored the last baby weight example to:

Remove unicode greek names (following the examples style guide)
Include both mutable and constant coords, to highlight the difference

I think (2) is important, since it's not really a documented feature anywhere yet.

review-notebook-app · 2023-07-12T09:26:25Z

View / edit / reply to this conversation on ReviewNB

twiecki commented on 2023-07-12T09:26:25Z
----------------------------------------------------------------

Should we just rename this to Using Data containers?

jessegrabowski commented on 2023-07-12T09:46:55Z
----------------------------------------------------------------

I like that better, since users don't need to know what a "shared variable" is to use the data containers API

review-notebook-app · 2023-07-12T09:26:26Z

View / edit / reply to this conversation on ReviewNB

twiecki commented on 2023-07-12T09:26:26Z
----------------------------------------------------------------

*allow you to

Maybe also mention why anyone would want to use ConstantData (performance).

Also, I think that pm.Data also allows specification of dims, no? The only reason we have Data, MutableData, and ConstantData is historical and I think we should say that best practice is to use MutableData and ConstantData and only mention Data as a side-note.

jessegrabowski commented on 2023-07-12T09:44:57Z
----------------------------------------------------------------

I was thinking it would be nice to change one of the examples to use ConstantData, so that there's one usage for both. Maybe the first example. Then we could change around some headings (to point out which container is being used in which section, and why)?

A short passage at the beginning could be added, like "In past versions of PyMC, data was added to a model using pm.Data. While this is still possible for backwards-comparability reasons, the current best practice is to use either pm.MutableData or pm.ConstantData,depending on your needs. This notebook will demonstrate how each of these data containers are used, and when they are appropriate".

twiecki commented on 2023-07-13T10:20:53Z
----------------------------------------------------------------

Yes, I think that's perfect.

jessegrabowski commented on 2023-07-13T10:59:44Z
----------------------------------------------------------------

I tried adding some details about each of the containers and give an example for each, let me know if it falls short of the mark.

review-notebook-app · 2023-07-12T09:26:27Z

View / edit / reply to this conversation on ReviewNB

twiecki commented on 2023-07-12T09:26:27Z
----------------------------------------------------------------

size is internal API, either use shape, or, better yet, also a dummy-dim.

jessegrabowski commented on 2023-07-12T09:51:08Z
----------------------------------------------------------------

What do you mean by a dummy dim?

twiecki commented on 2023-07-13T10:20:20Z
----------------------------------------------------------------

Oh, I see what you're doing here now. I think it should just be shape though.

jessegrabowski · 2023-07-12T09:44:58Z

I was thinking it would be nice to change one of the examples to use ConstantData, so that there's one usage for both. Maybe the first example. Then we could change around some headings (to point out which container is being used in which section, and why)?

A short passage at the beginning could be added, like "In past versions of PyMC, data was added to a model using pm.Data. While this is still possible for backwards-comparability reasons, the current best practice is to use either pm.MutableData or pm.ConstantData,depending on your needs. This notebook will demonstrate how each of these data containers are used, and when they are appropriate".

View entire conversation on ReviewNB

jessegrabowski · 2023-07-12T09:46:56Z

I like that better, since users don't need to know what a "shared variable" is to use the data containers API

View entire conversation on ReviewNB

jessegrabowski · 2023-07-12T09:51:10Z

What do you mean by a dummy dim?

View entire conversation on ReviewNB

twiecki · 2023-07-13T10:20:21Z

Oh, I see what you're doing here now. I think it should just be shape though.

View entire conversation on ReviewNB

twiecki · 2023-07-13T10:20:54Z

Yes, I think that's perfect.

View entire conversation on ReviewNB

jessegrabowski · 2023-07-13T10:59:45Z

I tried adding some details about each of the containers and give an example for each, let me know if it falls short of the mark.

View entire conversation on ReviewNB

examples/howto/data_container.myst.md

OriolAbril · 2023-07-13T14:57:41Z

examples/howto/data_container.myst.md

@@ -112,67 +116,100 @@ idata.posterior.coords
 az.plot_trace(idata, var_names=["europe_mean_temp", "city_temperature"]);


Suggested change

az.plot_trace(idata, var_names=["europe_mean_temp", "city_temperature"]);

az.plot_trace(idata, var_names=["europe_mean_temp", "city_temperature"], legend=True);

this should add a legend to both rows, first with chain linestyle mapping, then with city color mapping.

The auto-placement leaves a bit to be desired -- is there any way to pass legend_kwargs to the underlying plot?

examples/howto/data_container.myst.md

OriolAbril

added one extra comment, also forgot to mention that I review on the myst file, but you should work on the ipynb and let pre-commit update the myst one

examples/howto/data_container.myst.md

…mples, rename variables to be more meaningful

ricardoV94

Looks good, I left a couple of smaller suggestions

examples/howto/data_container.myst.md

…utableData

ricardoV94

Shaping up really nice, just some minor suggestions

examples/howto/data_container.myst.md

…NDOM_SEED in all sampling functions, promote GLM example to heading 2

examples/howto/data_container.myst.md

OriolAbril

noticed one typo (again, fix in ipynb, then pre-commit), but other than that I think it is good to merge

examples/howto/data_container.myst.md

update data_container example to pymc 5.6

2ca63ba

jessegrabowski added 2 commits July 11, 2023 17:32

run pre-commit hooks

9ba20e7

rerun notebook

4519864

OriolAbril reviewed Jul 11, 2023

View reviewed changes

examples/howto/data_container.myst.md Outdated Show resolved Hide resolved

jessegrabowski added 3 commits July 12, 2023 13:48

add author information

08a46ff

remove link to out-of-date example

43ee41e

refactor baby weight example

ba3d2c8

change title, fix typo

d8e5b0d

add more detail on the ConstantData vs MutableData

3939e7e

fix typos

4370d39

OriolAbril reviewed Jul 13, 2023

View reviewed changes

examples/howto/data_container.myst.md Show resolved Hide resolved

address comments from review

708b6a3

ricardoV94 reviewed Jul 14, 2023

View reviewed changes

examples/howto/data_container.myst.md Outdated Show resolved Hide resolved

examples/howto/data_container.myst.md Outdated Show resolved Hide resolved

jessegrabowski added 2 commits July 14, 2023 15:00

remove reference to PyMC5

c8f29d0

add examples showing data on DAG, add more analysis/discussion to exa…

070b730

…mples, rename variables to be more meaningful

ricardoV94 reviewed Jul 14, 2023

View reviewed changes

jessegrabowski added 5 commits July 15, 2023 12:16

reduce emphasis on observed keyword in introduction

5b9c347

more distinction between endogenous and exogenous (X and y) uses of M…

74fcfb1

…utableData

remove reference to

1275eae

convert multiple model example to parameter recovery exercise

aaa9bca

remove filterwarnings

bf069d8

ricardoV94 reviewed Jul 15, 2023

View reviewed changes

examples/howto/data_container.myst.md Outdated Show resolved Hide resolved

examples/howto/data_container.myst.md Outdated Show resolved Hide resolved

examples/howto/data_container.myst.md Outdated Show resolved Hide resolved

jessegrabowski added 3 commits July 15, 2023 21:34

fix typos, add example showing exogenous data stored in idata, use RA…

3913f48

…NDOM_SEED in all sampling functions, promote GLM example to heading 2

fix broken references

5bb72cc

fix broken references

a050f09

OriolAbril reviewed Jul 15, 2023

View reviewed changes

examples/howto/data_container.myst.md Show resolved Hide resolved

OriolAbril approved these changes Jul 17, 2023

View reviewed changes

examples/howto/data_container.myst.md Outdated Show resolved Hide resolved

fix typo

dfb7bf9

ricardoV94 reviewed Jul 18, 2023

View reviewed changes

examples/howto/data_container.myst.md Outdated Show resolved Hide resolved

examples/howto/data_container.myst.md Outdated Show resolved Hide resolved

more typo fies

d24fb04

ricardoV94 approved these changes Jul 18, 2023

View reviewed changes

ricardoV94 merged commit d6df689 into pymc-devs:main Jul 18, 2023
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update data_container example to pymc 5.6 #559

update data_container example to pymc 5.6 #559

jessegrabowski commented Jul 11, 2023 •

edited

review-notebook-app bot commented Jul 11, 2023

jessegrabowski commented Jul 11, 2023

OriolAbril commented Jul 11, 2023

OriolAbril left a comment

jessegrabowski commented Jul 12, 2023 •

edited

review-notebook-app bot commented Jul 12, 2023 •

edited

review-notebook-app bot commented Jul 12, 2023 •

edited

review-notebook-app bot commented Jul 12, 2023 •

edited

jessegrabowski commented Jul 12, 2023

jessegrabowski commented Jul 12, 2023

jessegrabowski commented Jul 12, 2023

twiecki commented Jul 13, 2023

twiecki commented Jul 13, 2023

jessegrabowski commented Jul 13, 2023

OriolAbril Jul 13, 2023

jessegrabowski Jul 14, 2023

OriolAbril left a comment

ricardoV94 left a comment

ricardoV94 left a comment

OriolAbril left a comment •

edited

		@@ -112,67 +116,100 @@ idata.posterior.coords
		az.plot_trace(idata, var_names=["europe_mean_temp", "city_temperature"]);

update data_container example to pymc 5.6 #559

update data_container example to pymc 5.6 #559

Conversation

jessegrabowski commented Jul 11, 2023 • edited

Update how_to/data_container to PyMC 5.6

review-notebook-app bot commented Jul 11, 2023

jessegrabowski commented Jul 11, 2023

OriolAbril commented Jul 11, 2023

OriolAbril left a comment

Choose a reason for hiding this comment

jessegrabowski commented Jul 12, 2023 • edited

review-notebook-app bot commented Jul 12, 2023 • edited

review-notebook-app bot commented Jul 12, 2023 • edited

review-notebook-app bot commented Jul 12, 2023 • edited

jessegrabowski commented Jul 12, 2023

jessegrabowski commented Jul 12, 2023

jessegrabowski commented Jul 12, 2023

twiecki commented Jul 13, 2023

twiecki commented Jul 13, 2023

jessegrabowski commented Jul 13, 2023

OriolAbril Jul 13, 2023

Choose a reason for hiding this comment

jessegrabowski Jul 14, 2023

Choose a reason for hiding this comment

OriolAbril left a comment

Choose a reason for hiding this comment

ricardoV94 left a comment

Choose a reason for hiding this comment

ricardoV94 left a comment

Choose a reason for hiding this comment

OriolAbril left a comment • edited

Choose a reason for hiding this comment

jessegrabowski commented Jul 11, 2023 •

edited

Update `how_to/data_container` to PyMC 5.6

jessegrabowski commented Jul 12, 2023 •

edited

review-notebook-app bot commented Jul 12, 2023 •

edited

review-notebook-app bot commented Jul 12, 2023 •

edited

review-notebook-app bot commented Jul 12, 2023 •

edited

OriolAbril left a comment •

edited