Add US hospitalization forecast group #7

Serena-Wang · 2021-10-29T17:32:08Z

Close issue add support for getting hospitalizations data #1
In mechbayes/jhu.py:
- add load_us_covidcast() to load data for all target variables from covidcast.
- use incident data for all locations, all target variable.
Update data cleaning functions and plot function to work with incident data.
In scripts/run_model.py, add boolean parameter use_hosp_as_death to indicate whether a model needs to replace death observations with hospitalization observations
Add hospitalization models and forecast group configuration
Add adjustment cases to hospitalization data
update existing adjustments to work with inc level case/death data
update score_place() to work with inc level truth data

…gends

dsheldon

Nice work, @Serena-Wang! Most of the changes seem spot on. (I didn't check pandas manipulations very carefully.)

The only comments of significance were about the seeding process and the implications of dropping the first cumulative observation in SEIRD.py and SEIRD_renewal.py.

Seeding has always been a little fuzzy. My best guess is the changes are insignificant for incident forecasts but could matter for cumulative forecasts. I think we probably need to discuss exactly what we want there. It would be good if you could test what happens with cumulative forecasts, esp. with a start date well into the pandemic, to see what happens.

mechbayes/jhu.py

scripts/config.json

scripts/run_model.py

mechbayes/models/SEIRD.py

dsheldon · 2021-10-31T20:03:09Z

mechbayes/models/SEIRD_renewal.py

        if death is None:
            death0, death = (None, None)
        else: 
            death0 = death[0]
-            death = clean_daily_obs(onp.diff(death))
+            death = clean_daily_obs(death[1:])



I have a similar comment to the one in SEIRD.py --- how does this affect seeding and cumulative forecasts?

elray1 · 2021-11-01T14:42:48Z

mechbayes/util.py

@@ -1,3 +1,4 @@
+from pickle import FALSE


What is this doing?

elray1

minimal comments other than what Dan already noted.

elray1 · 2021-11-01T14:52:41Z

mechbayes/util.py

                else:
                    variables = ['y', 'z']
-                    observations= [confirmed, death]
+                    observations= [confirmed.cumsum(), death.cumsum()]


Flagging this as a place where we might want to add in cum death on date before start_date

mechbayes/models/SEIRD.py

elray1 · 2021-11-01T15:10:01Z

scripts/data_cleaning.py

@@ -332,7 +372,7 @@ def make_manual_adjustments(data, forecast_date):
    # JHU: 2,029 historical cases; Ellis County reported 294
    util.redistribute(data['TX']['data'], '2021-03-26', 2029+294, 90, 'confirmed')

-    util.redistribute(data['MN']['data'], '2021-03-25', 20, 20, 'death')
+    #util.redistribute(data['MN']['data'], '2021-03-25', 20, 20, 'death')


did we just decide this was no longer necessary?

elray1 · 2021-11-01T15:10:27Z

scripts/data_cleaning.py

@@ -445,7 +485,7 @@ def make_manual_adjustments(data, forecast_date):

    # https://github.com/CSSEGISandData/COVID-19/issues/3705
    # (backdistributed week of March 1)
-    #util.redistribute(data['IA']['data'], '2021-02-19', 26775, 200, 'confirmed')
+    util.redistribute(data['IA']['data'], '2021-02-19', 26775, 200, 'confirmed')


this was previously commented out but isn't anymore?

…run_model and config, fix typo

…issue_1

…function

…issue_1

…ssue_1

…d death0

…issue_1

Serena-Wang added 19 commits September 29, 2021 14:36

load data from covidcast

b33ead0

add test_hosp model

395c900

merge changes from branch test

7ecb1ac

load hosp data to model

31b2cf4

udpate loading hosp data

7ec6704

update data in plot:

d8d7651

update config

fed09a6

train on inc data from covidcast and update plots

c42707a

add hospitalization to plot legend, train model on inc data

5062293

add extra paramter in run_model, add hosp adjustments, update plot le…

4c512bc

…gends

update model config

3c121eb

update config and jhu.py

560c026

update function parameter

5075d50

update smooth_to_weekly() to use inc data

6ffe356

add col parameter in set_trailing_weekend_zeros_to_missing()

01d3c52

add test config, update plot legend and data cleaning script

3edf14b

Merge branch 'main' into issue_1

4917055

update hard-coded cumulative value adjustment in data_cleaning.py

4b64160

update dest in config

5665636

dsheldon reviewed Oct 31, 2021

View reviewed changes

elray1 reviewed Nov 1, 2021

View reviewed changes

mechbayes/util.py Outdated

@@ -1,3 +1,4 @@

from pickle import FALSE

Copy link

Collaborator

elray1 Nov 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this doing?

elray1 requested changes Nov 1, 2021

View reviewed changes

Serena-Wang added 8 commits November 1, 2021 13:49

update data adjustment cases

bf3c996

update observation data shape in models, update boolean parameter in …

62f14c3

…run_model and config, fix typo

Merge branch 'main' into issue_1

539632d

update model start date, add initial value parameters

abab13c

Merge branch 'issue_1' of https://github.com/dsheldon/mechbayes into …

90147ef

…issue_1

update config

3b67149

Merge branch 'issue_1' of https://github.com/dsheldon/mechbayes into …

8a65ccf

…issue_1

add confirm0 and death0 to obs and obs0, add condition in predictive …

d604b5c

…function

Serena-Wang and others added 15 commits December 9, 2021 17:47

Merge branch 'issue_1' of https://github.com/dsheldon/mechbayes into …

9efc77b

…issue_1

update comments and take out print statment

8ade0ba

merge with main

816f4ed

modify submit_util to create forecast file for hosp

c722256

erge branch 'issue_1' of https://github.com/dsheldon/mechbayes into i…

a694b1d

…ssue_1

Fix logic for cleaning confirmed and death to not unset confirmed0 an…

68adb81

…d death0

update config

45207ee

merge data_cleaning from main branch

974e335

update TN data cleaning to work with inc value

03a22c0

update score_place to work with inc level observations

001d117

update logic in SEIRD to process cumulative values

ae46742

update logic in SEIRD to process cumulative values

3d27250

Merge branch 'main' into issue_1

968f635

Merge branch 'issue_1' of https://github.com/dsheldon/mechbayes into …

d087a86

…issue_1

clean up

73d0114

elray1 self-requested a review January 12, 2022 17:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add US hospitalization forecast group #7

Add US hospitalization forecast group #7

Serena-Wang commented Oct 29, 2021 •

edited

Loading

dsheldon left a comment

dsheldon Oct 31, 2021

elray1 Nov 1, 2021

elray1 left a comment

elray1 Nov 1, 2021

elray1 Nov 1, 2021

elray1 Nov 1, 2021

Add US hospitalization forecast group #7

Are you sure you want to change the base?

Add US hospitalization forecast group #7

Conversation

Serena-Wang commented Oct 29, 2021 • edited Loading

dsheldon left a comment

Choose a reason for hiding this comment

dsheldon Oct 31, 2021

Choose a reason for hiding this comment

elray1 Nov 1, 2021

Choose a reason for hiding this comment

elray1 left a comment

Choose a reason for hiding this comment

elray1 Nov 1, 2021

Choose a reason for hiding this comment

elray1 Nov 1, 2021

Choose a reason for hiding this comment

elray1 Nov 1, 2021

Choose a reason for hiding this comment

Serena-Wang commented Oct 29, 2021 •

edited

Loading