Conditional autoregressive priors #547

daniel-saunders-phil · 2023-05-26T00:28:57Z

#417 is got stuck and the original author isn't pursing the PR anymore. However, it would be nice to get this notebook finished. I just don't have the permissions to change #417 directly. So I've made a new PR to, at the very least, test out the preview of read the docs and see if it passes continuous integration checks on github.

review-notebook-app · 2023-05-26T00:29:02Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

daniel-saunders-phil · 2023-05-26T00:42:40Z

What's the right technique to generate and then fix these merge conflicts locally?

daniel-saunders-phil · 2023-05-26T23:12:46Z

I've managed to pass all the pre-commit checks locally, so that's progress for this notebook. I ended up finding a half dozen other typos that were causing pre-commit to fail. However, the preview file is rendered terribly plus the branch conflicts. Not sure how to fix those yet.

$ git commit -m "changed aesara.tensor to pytensor.tensor"
jupytext......................................................................................Passed
black-jupyter.................................................................................Passed
nbqa-isort....................................................................................Passed
nbqa-pyupgrade................................................................................Passed
Check cells were executed sequentially........................................................Passed
bibtex-tidy...............................................................(no files to check)Skipped
Check notebooks have watermark (see Jupyter style guide from PyMC docs).......................Passed
Check no internal links are in the docs.......................................................Passed
jupytext......................................................................................Passed
codespell.....................................................................................Passed
[conditional_autoregressive_priors 4d2741d] changed aesara.tensor to pytensor.tensor
 2 files changed, 346 insertions(+), 17 deletions(-)

…-saunders-phil/pymc-examples into conditional_autoregressive_priors

daniel-saunders-phil · 2023-05-31T23:38:26Z

I think it might be done! @bwengals and I fought through a lot of pre-commit to get here 🫠

OriolAbril

Added some comments after going over the notebook. I haven't gone over the contents in depth yet though. Before going over the comments and further review, first note:

I have commented everything on the myst file. Don't edit the myst file directly, apply the changes to the notebook and then run pre-commit.
I am completely lost on what is the relation between this notebook and the existing one. Are they complementary, should this replace the other one, is the old one still relevant and salvageable (even if this one is somewhat complementary)?

OriolAbril · 2023-06-01T15:28:17Z

examples/case_studies/conditional_autoregressive_priors.myst.md

+# Conditional Autoregressive (CAR) Models for Spatial Data
+
+:::{post} Jul 29, 2022 
+:tags: spatial


There is also an autoregressive tag https://www.pymc.io/projects/examples/en/latest/blog/tag/autoregressive.html which seems relevant for the notebook based on the title. I'd also recommend going over the list of tags (left sidebar of https://www.pymc.io/projects/examples/en/latest/gallery.html) and add other that might be relevant too.

OriolAbril · 2023-06-01T15:40:40Z

examples/case_studies/conditional_autoregressive_priors.myst.md

+:::{post} Jul 29, 2022 
+:tags: spatial
+:category: beginner, tutorial
+:author: Conor Hassan 


Also add yourself here

OriolAbril · 2023-06-01T15:45:43Z

examples/case_studies/conditional_autoregressive_priors.myst.md

+substitutions:
+  extra_dependencies: bambi seaborn


Suggested change

substitutions:

extra_dependencies: bambi seaborn

myst:

substitutions:

extra_dependencies: bambi seaborn

I need to update the docs, they recenly updated the metadata keys for this. Ref: https://myst-parser.readthedocs.io/en/latest/syntax/optional.html#substitutions-with-jinja2. It also looks like geopandas and pylibsal and the like should also be added here.

Is there anything I should do on my end? The ipynb file just has:

:::{include} ../extra_installs.md :::

yes, only the metadata and these two lines are enough. You can see in the preview how this two lines add into the rendered notebook a predefined template with install instructions with pip, conda and from within jupyter: https://pymcio--547.org.readthedocs.build/projects/examples/en/547/case_studies/conditional_autoregressive_priors.html

i'm not clear on where I can add the metadata. The substitutions lines only appears in the myst file (as far as I can tell) and I thought that's entirely generated from the .ipynb.

Also, if it matters we don't need seaborn or bambi for this one. Just geopandas and pylibsal. Both should be installed with conda-forge.

The section I linked of the style guide covers how to edit the metadata from jupyter. This (together with cell level metadata which we also use) is one of the main reasons we use myst files in addition to ipynb. The ipynb is needed to store outputs and render the website properly, myst ones are needed for proper reviews to be possible (this might change with the improvements in jupyer diffs github is doing but for now I think they are still needed)

I'm still pretty unsure - the .ipynb metadata now reads:

"substitutions": { "extra_dependencies": "geopandas libpysal" },

but the document preview is unchanged. What might I be missing? Is there an example of a notebook in the pymc library that has used this myst substitutions trick successfully I can model?

It has been updated, you might have been a bit too eager and not waited for the rtd check (has status mark below) finished. I do see the new libraries now.

This behaviour is the one that appears in the style guide (and still works) but was recently deprecated. It should be:

"myst": { "substitutions": { "extra_dependencies": "geopandas libpysal" } },

OriolAbril · 2023-06-01T15:50:44Z

examples/case_studies/conditional_autoregressive_priors.myst.md

+We need to load in the dataset to access the variables $\{y_i, x_i, E_i\}_{i=1}^N$. But more unique to the use of CAR models, is the creation of the necessary spatial adjacency matrix. For the models that we fit, all neighbours are weighted as $1$, circumventing the need for a weight matrix. The dataset can be accessed via the `pm.get_data` function.
+
+```{code-cell} ipython3
+df_scot_cancer = pd.read_csv(pm.get_data("scotland_lips_cancer.csv"))


this should be updated to https://www.pymc.io/projects/docs/en/latest/contributing/jupyter_style.html#reading-from-file

OriolAbril · 2023-06-01T15:56:21Z

examples/case_studies/conditional_autoregressive_priors.myst.md

+
+```{code-cell} ipython3
+independent_stacked = az.extract(independent_idata)
+spat_df["INDEPENDENT_RES"] = independent_stacked.res.mean(axis=1)


Suggested change

spat_df["INDEPENDENT_RES"] = independent_stacked.res.mean(axis=1)

spat_df["INDEPENDENT_RES"] = independent_stacked.res.mean(dim="sample")

or, given I don't see independent_stacked variable being used later, it could even be:

spat_df["INDEPENDENT_RES"] = independent_data["posterior"]["res"].mean(dim=["chain", "draw"])

OriolAbril · 2023-06-01T15:57:53Z

examples/case_studies/conditional_autoregressive_priors.myst.md

+fixed_spatial_stacked = az.extract(fixed_spatial_idata)
+spat_df["SPATIAL_RES"] = fixed_spatial_stacked.res.mean(axis=1)


similar comment as above, axis=... should not be used with xarray objects

OriolAbril · 2023-06-01T15:58:44Z

examples/case_studies/conditional_autoregressive_priors.myst.md

+```{code-cell} ipython3
+car_stacked = az.extract(car_idata)
+```


this cell is not used anywhere, should be removed

OriolAbril · 2023-06-01T16:06:58Z

examples/case_studies/conditional_autoregressive_priors.myst.md

+* Adapted from a previous PyMC example notebook, authored by Junpeng Lao {ref}`conditional_autoregressive_model` by Conor Hassan on July, 2022.
+* Re-executed by Daniel Saunders in May, 2023


I'd probably say something like:

* Adapted from {ref}`another PyMC example notebook <conditional_autoregressive_model>` by Conor Hassan and Daniel Saunders ([pymc-examples#417](https://github.com/pymc-devs/pymc-examples/pull/417) and [pymc-examples#547](https://github.com/pymc-devs/pymc-examples/pull/547/)).

Authorship of the other notebook should be on the other notebook I think.

OriolAbril · 2023-06-01T16:07:27Z

examples/case_studies/conditional_autoregressive_priors.myst.md

+
+```{code-cell} ipython3
+%load_ext watermark
+%watermark -n -u -v -iv -w -p pytensor,xarray


Suggested change

%watermark -n -u -v -iv -w -p pytensor,xarray

%watermark -n -u -v -iv -w -p xarray

pytensor is imported, so adding it here duplicates it on the watermark

daniel-saunders-phil · 2023-06-05T18:23:56Z

I am completely lost on what is the relation between this notebook and the existing one. Are they complementary, should this replace the other one, is the old one still relevant and salvageable (even if this one is somewhat complementary)?

I think this notebook replaces the other one for most users. The old CAR notebook is great for explaining why PyMC's car prior is implemented the way it is. So users who are interested in understanding how to write and modify their own algorithms for similar models autoregressive models would benefit from reading the old notebook (albeit, with up dates to pymc 5). For everyone else, the new notebook is more useful - it explain a bit about why we should care about spatial autocorrelation and is far more concise.

OriolAbril · 2023-06-05T18:36:40Z

I think this notebook replaces the other one for most users. The old CAR notebook is great for explaining why PyMC's car prior is implemented the way it is. So users who are interested in understanding how to write and modify their own algorithms for similar models autoregressive models would benefit from reading the old notebook (albeit, with up dates to pymc 5). For everyone else, the new notebook is more useful - it explain a bit about why we should care about spatial autocorrelation and is far more concise.

Thanks, this is great!

Would you be interested in updating the other one too? If so we can continue the discussion once the time comes, otherwise (or in the meantime if it is expected to be a longish wait), what do you think about updating the title of the other notebook to be "About CAR models in PyMC"? I think it is more fitting given your description

…ussion of divergences

daniel-saunders-phil · 2023-06-06T00:11:39Z

Yeah I'd like to revise the other one at some point. I think I'll get to it after my next task (trying to develop the ICAR) so an interim title change sounds good. Would I update the (conditional_autoregressive_model)= at the top of the file plus the references to the title in my current notebook?

A quick explanation of the other changes to the file: previously the notebook suggested model 3 was unidentifiable and trying to sample from it would typically result in chains with loads of divergences. I wasn't able to reliably reproduce these divergences and can't find evidence of degenerate geometries in the posterior distribution. I think the divergences discussed in the earlier drafts might be due to bad luck. To avoid discussion of divergences, I rewrote the bottom few cells to provide a different motivation for the ICAR.

…ussion of divergences

OriolAbril · 2023-06-12T18:47:50Z

Would I update the (conditional_autoregressive_model)= at the top of the file plus the references to the title in my current notebook?

The main thing that needs to be updated is the title, the # Conditional Autoregressive (CAR) model. The ()= part is a target/anchor. It is something to be used for cross-referencing, and doesn't appear anywhere on the rendered website. As it is brand new and we can be sure no notebooks are using it, it can be updated too. Otherwise, it'd be best to add two targets, so we don't need to search for references to that in all notebooks and modify them (with potential git conflicts...). We already do that in https://github.com/pymc-devs/pymc-examples/blob/main/examples/case_studies/multilevel_modeling.ipynb for example

daniel-saunders-phil · 2023-06-13T02:45:47Z

Hi @OriolAbril I think it might be ready to go again - I adjusted the metadata and the title of the old CAR notebook as you suggested. My read the docs preview doesn't seem to be up to date but the committed files are all where I think they should be.

…utoregressive_priors

OriolAbril

left a comment about the categories for the "about" notebook, other than that I think it is good to go, thanks!

OriolAbril · 2023-06-15T12:06:44Z

examples/case_studies/conditional-autoregressive-model.myst.md

+
+:::{post} Aug 14, 2020 
+:tags: spatial, autoregressive, count data
+:category: advanced, reference


I think the notebook is more explanation type than reference.

OriolAbril · 2023-06-15T17:39:22Z

Thanks!

conorhassan and others added 2 commits September 9, 2022 08:08

CAR notebook

d5350bb

implementing oriol's suggestions + typo

7145e4e

changed aesara.tensor to pytensor.tensor

4d2741d

daniel-saunders-phil added 2 commits May 26, 2023 17:00

fixed execution order, made watermark render properly

7574d2c

Merge branch 'main' into conditional_autoregressive_priors

f842b2b

daniel-saunders-phil marked this pull request as ready for review May 31, 2023 20:35

daniel-saunders-phil changed the title ~~DRAFT: Conditional autoregressive priors~~ Conditional autoregressive priors May 31, 2023

daniel-saunders-phil added 4 commits May 31, 2023 15:53

Merge branch 'main' into conditional_autoregressive_priors

41dc905

Merge branch 'conditional_autoregressive_priors' of github.com:daniel…

6468dec

…-saunders-phil/pymc-examples into conditional_autoregressive_priors

syncing myst nb to ipynb

0b85c44

fix latex rendering

95095d0

OriolAbril reviewed Jun 1, 2023

View reviewed changes

oriol's suggestions + simplified pm.sample() arguments + removed disc…

851aa56

…ussion of divergences

daniel-saunders-phil added 4 commits June 5, 2023 17:39

oriol's suggestions + simplified pm.sample() arguments + removed disc…

6fcc1ec

…ussion of divergences

metadata for myst substitutions

fa2f6bd

metadata for myst substitutions

1d6c8d7

still trying to get metadata right

6796f19

daniel-saunders-phil added 2 commits June 12, 2023 13:45

adjust title for old CAR notebook + correct myst substitutions syntax

dbe6600

fixing preamble in previous commit

a8e0302

change reference to conditional_autoregressive_prior -> conditional_a…

27795d1

…utoregressive_priors

OriolAbril approved these changes Jun 15, 2023

View reviewed changes

reference -> explanation

941572c

OriolAbril merged commit 35cd42f into pymc-devs:main Jun 15, 2023

daniel-saunders-phil mentioned this pull request Jun 17, 2023

Conditional autoregressive prior example notebook #417

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Conditional autoregressive priors #547

Conditional autoregressive priors #547

daniel-saunders-phil commented May 26, 2023 •

edited

Loading

review-notebook-app bot commented May 26, 2023

daniel-saunders-phil commented May 26, 2023

daniel-saunders-phil commented May 26, 2023 •

edited

Loading

daniel-saunders-phil commented May 31, 2023

OriolAbril left a comment

OriolAbril Jun 1, 2023

OriolAbril Jun 1, 2023

OriolAbril Jun 1, 2023

daniel-saunders-phil Jun 5, 2023 •

edited

Loading

OriolAbril Jun 5, 2023

daniel-saunders-phil Jun 5, 2023

OriolAbril Jun 7, 2023

daniel-saunders-phil Jun 7, 2023 •

edited

Loading

OriolAbril Jun 12, 2023

OriolAbril Jun 1, 2023

OriolAbril Jun 1, 2023

OriolAbril Jun 1, 2023

OriolAbril Jun 1, 2023

OriolAbril Jun 1, 2023

OriolAbril Jun 1, 2023

daniel-saunders-phil commented Jun 5, 2023

OriolAbril commented Jun 5, 2023

daniel-saunders-phil commented Jun 6, 2023 •

edited

Loading

OriolAbril commented Jun 12, 2023

daniel-saunders-phil commented Jun 13, 2023

OriolAbril left a comment

OriolAbril Jun 15, 2023

OriolAbril commented Jun 15, 2023

	spat_df["INDEPENDENT_RES"] = independent_stacked.res.mean(axis=1)
	spat_df["INDEPENDENT_RES"] = independent_stacked.res.mean(dim="sample")

		fixed_spatial_stacked = az.extract(fixed_spatial_idata)
		spat_df["SPATIAL_RES"] = fixed_spatial_stacked.res.mean(axis=1)

		* Adapted from a previous PyMC example notebook, authored by Junpeng Lao {ref}`conditional_autoregressive_model` by Conor Hassan on July, 2022.
		* Re-executed by Daniel Saunders in May, 2023

	%watermark -n -u -v -iv -w -p pytensor,xarray
	%watermark -n -u -v -iv -w -p xarray

Conditional autoregressive priors #547

Conditional autoregressive priors #547

Conversation

daniel-saunders-phil commented May 26, 2023 • edited Loading

review-notebook-app bot commented May 26, 2023

daniel-saunders-phil commented May 26, 2023

daniel-saunders-phil commented May 26, 2023 • edited Loading

daniel-saunders-phil commented May 31, 2023

OriolAbril left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daniel-saunders-phil Jun 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daniel-saunders-phil Jun 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daniel-saunders-phil commented Jun 5, 2023

OriolAbril commented Jun 5, 2023

daniel-saunders-phil commented Jun 6, 2023 • edited Loading

OriolAbril commented Jun 12, 2023

daniel-saunders-phil commented Jun 13, 2023

OriolAbril left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OriolAbril commented Jun 15, 2023

daniel-saunders-phil commented May 26, 2023 •

edited

Loading

daniel-saunders-phil commented May 26, 2023 •

edited

Loading

daniel-saunders-phil Jun 5, 2023 •

edited

Loading

daniel-saunders-phil Jun 7, 2023 •

edited

Loading

daniel-saunders-phil commented Jun 6, 2023 •

edited

Loading