Refactor nb/nhanes survival model

[CloseChoice]

Overview

Works towards #3036

Description of the changes proposed in this pull request:

• fix a couple of typos
• correct column names for interaction value calculation
• add a description to the calculation of the interaction matrix

Note that there are a couple more warnings now, since I use xgboost 2.1.2 which deprecates the file format we are using. But this is an issue with a linked PR and will be fixed once this PR is merged.

Checklist

• All pre-commit checks pass.
• [ ] Unit tests added (if fixing a bug or adding a new feature)

[connortann]

Thanks for the PR! I have a few suggestions.

Unfortunately I can't comment on the diff, so I'll leave a few suggestions here as screenshots.

• This might no longer be true:
  

• Suggest moving this comment to its own line to avoid the unusual formatting:
  

• Suggest removing this, either by clearing that cell output or editing the JSON directly:
  

• This long paragraph could be broken up into multiple paragraphs to help readability:
  

• Same comment about the warning:
  

• May as well move these imports to the top:
  

• The final image has much many more columns than before, which I think is less readable. Can we recreate the previous image?
  

[connortann]

A few suggested changes as above

[connortann]

Hi @CloseChoice - I was just going through the PR backlog and saw this one. Are you happy to take another look at the touchups above? Or if you'd prefer, I can finish it off if you're stuck into other stuff at the moment.

[CloseChoice]

Hi @CloseChoice - I was just going through the PR backlog and saw this one. Are you happy to take another look at the touchups above? Or if you'd prefer, I can finish it off if you're stuck into other stuff at the moment.

Thanks for coming back to this. Will have a look sometime this week

[CloseChoice]

@connortann I added the requested changes. The dataset seems to contain more columns now but I limited the confusion matrix to only the 12 with the highest shap values.

[connortann]

The "Exact Explainer" notebook was changed, is that intentional? It now has an exception:

Other than that, looks great! Thank you for making the changes I suggested above.

A minor observation, I notice the dependence plot of Red Blood Cells looks quite different. I wonder if the source data changed at some point: there seem to be values of red_blood_cells_unacceptable which might indicate "bad" measurements which were originally dropped from the dataset? I think the change is fine though, I don't think we need to reproduce the original images exactly.

[CloseChoice]

The "Exact Explainer" notebook was changed, is that intentional? It now has an exception:

Other than that, looks great! Thank you for making the changes I suggested above.

A minor observation, I notice the dependence plot of Red Blood Cells looks quite different. I wonder if the source data changed at some point: there seem to be values of red_blood_cells_unacceptable which might indicate "bad" measurements which were originally dropped from the dataset? I think the change is fine though, I don't think we need to reproduce the original images exactly.

Good catch. I removed the changes in the exact notebook but didn't look into the changed plot. I would suspect that it has to do with changes in the underlying data but wouldn't spend more time on that. Thanks for the review ;)