Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Infra updates] Generate and publish a Quarto doc with performance results on each model run #24

Closed
Tracked by #175
jeancochrane opened this issue Oct 20, 2023 · 0 comments · Fixed by #62
Closed
Tracked by #175
Assignees

Comments

@jeancochrane
Copy link
Contributor

jeancochrane commented Oct 20, 2023

This year, we're going to consolidate all diagnostic reporting into a single Quarto document that gets created for each model run. The document will knit at the end of each run, once the model is finished training, has created performance statistics, etc. The output PDF (or HTML) will be uploaded to S3 along with the other model artifacts, then linked in SNS completion notifications.

This Quarto doc will be the primary way we evaluate individual model performance. Cross-model comparison will still be done via Tableau.

model_qc.qmd might be a good starting place for defining the Quarto doc itself.

Tasks

  • Update the 05-finalize.R step of the model so that it generates a Quarto doc containing performance results, uploads it to S3, and updates the SNS notification body to include a link to the doc in S3.
  • Consolidate existing Quarto docs (in reports/) into a single document
  • Add simple residuals scatterplots by township
  • Moran's I / spatial autocorrelation stats
  • IAAO performance stats by area
  • Lorenz curves by area
  • Checking for highly correlated variables, always check residuals plot (calibration plots with probably) cal_plot_regression
  • Explore using dtreeviz - https://github.com/parrt/dtreeviz/blob/master/notebooks/dtreeviz_lightgbm_visualisations.ipynb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants