Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to new flusurv API endpoint #1278

Open
wants to merge 43 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
d99d2d0
initial switch to new API endpoint; doesn't account for format change
nmdefries Aug 21, 2023
70c3e16
separate fetch fn for whole api obj
nmdefries Aug 31, 2023
3580f3f
get_current_issue to use existing json response
nmdefries Aug 31, 2023
9cb1da4
cleanup names and comments
nmdefries Sep 1, 2023
9bf3c82
store extracted rates by strata name rather than position
nmdefries Sep 1, 2023
cea25ab
define function to convert json obs to dict grouped by location and e…
nmdefries Sep 6, 2023
24dc088
auto-map from valueids to ordinal and label-based group names
nmdefries Sep 6, 2023
26eff97
add new strata to sql insert statement by name, not order
nmdefries Sep 6, 2023
229a96c
pass seasonids around to use in requests for location-specific data
nmdefries Sep 12, 2023
1e942a5
include old and new example API responses
nmdefries Sep 8, 2023
f8a6706
flusurv tests
nmdefries Sep 12, 2023
d9581ac
Merge branch 'dev' into ndefries/flusurv-new-endpoint
nmdefries Sep 15, 2023
9ac2918
pass metadata around to reduce API calls
nmdefries Sep 15, 2023
aa3dd8c
add season label as a descriptive column
nmdefries Sep 15, 2023
ca7e478
move example API responses to make it clear they are not for prod use
nmdefries Sep 15, 2023
b979910
Merge branch 'dev' into ndefries/flusurv-new-endpoint
nmdefries Sep 27, 2023
c1614a1
review cleanup
nmdefries Sep 28, 2023
19a5b42
capitalize constant max_age
nmdefries Oct 27, 2023
2f18451
move paren to subtract from # of dates
nmdefries Oct 27, 2023
d64c4c0
move n_exected_groups and big groupid comment to global
nmdefries Oct 27, 2023
c7829ee
convert asserts to if-raise
nmdefries Oct 27, 2023
c91b7e8
for robust group checking, list all expected groups and do set diff
nmdefries Oct 31, 2023
03926dd
separate data fetching fns into metadata and location classes
nmdefries Nov 1, 2023
c5d8d4e
programmatically create location-networkid map; create constants file
nmdefries Nov 2, 2023
9ed92d8
switch missing data assert to exception
nmdefries Nov 2, 2023
1c9e730
Merge branch 'dev' into ndefries/flusurv-new-endpoint
nmdefries Nov 2, 2023
eb920cb
mock metadata and api objs for tests
nmdefries Nov 6, 2023
b191602
stop doing recent season filtering
nmdefries Nov 6, 2023
b7ba101
warn when no location data returned
nmdefries Nov 6, 2023
e1d9d06
test metadata attributes together
nmdefries Nov 6, 2023
772e127
correct local module name to load classes from
nmdefries Nov 7, 2023
991fde1
map some long race groups to abbreviated names
nmdefries Nov 7, 2023
5d00dbb
programmatically create sql query to avoid ordering issues
nmdefries Nov 7, 2023
5cedb30
return empty result with right format when no data available
nmdefries Nov 7, 2023
e7ad06e
warning prefix
nmdefries Nov 7, 2023
75283d7
add tests for geo-abbr conversion and location fetching
nmdefries Nov 7, 2023
276944a
don't error for missing groups -- some are expected
nmdefries Nov 7, 2023
6c531a6
Merge branch 'dev' into ndefries/flusurv-new-endpoint
nmdefries Nov 7, 2023
9431489
fn to add empty sex breakdowns to UT
nmdefries Nov 7, 2023
205c061
with UT sex subsets getting filled, error on missing groups again
nmdefries Nov 7, 2023
5248a97
Merge branch 'dev' into ndefries/flusurv-new-endpoint
nmdefries Dec 8, 2023
9de7d64
Update flusurv schema and docs with new age, sex, and race groups (#1…
nmdefries May 8, 2024
40212bb
Revert "Update flusurv schema and docs with new age, sex, and race gr…
melange396 May 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
406 changes: 406 additions & 0 deletions src/acquisition/flusurv/api.py

Large diffs are not rendered by default.

110 changes: 110 additions & 0 deletions src/acquisition/flusurv/constants.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
from delphi_utils import GeoMapper

"""
As of Sept 2023, for new data we expect to see these 23 groups, as described
in the top-level "master_lookup" element, below, of the new GRASP API
(https://gis.cdc.gov/GRASP/Flu3/PostPhase03DataTool) response object.
See `./reference/new_grasp_result.json` for a full example response.
'master_lookup' = [
{'Variable': 'Age', 'valueid': 1, 'parentid': 97, 'Label': '0-4 yr', 'Color_HexValue': '#d19833', 'Enabled': True},
{'Variable': 'Age', 'valueid': 2, 'parentid': 97, 'Label': '5-17 yr', 'Color_HexValue': '#707070', 'Enabled': True},
{'Variable': 'Age', 'valueid': 3, 'parentid': 98, 'Label': '18-49 yr', 'Color_HexValue': '#44b3c6', 'Enabled': True},
{'Variable': 'Age', 'valueid': 4, 'parentid': 98, 'Label': '50-64 yr', 'Color_HexValue': '#516889', 'Enabled': True},
{'Variable': 'Age', 'valueid': 5, 'parentid': 98, 'Label': '65+ yr', 'Color_HexValue': '#cc5e56', 'Enabled': True},
{'Variable': 'Age', 'valueid': 7, 'parentid': 5, 'Label': '65-74 yr', 'Color_HexValue': '#cc5e56', 'Enabled': True},
{'Variable': 'Age', 'valueid': 8, 'parentid': 5, 'Label': '75-84 yr', 'Color_HexValue': '#cc5e56', 'Enabled': True},
{'Variable': 'Age', 'valueid': 9, 'parentid': 5, 'Label': '85+', 'Color_HexValue': '#cc5e56', 'Enabled': True},
{'Variable': 'Age', 'valueid': 10, 'parentid': 3, 'Label': '18-29 yr', 'Color_HexValue': '#44b3c6', 'Enabled': True},
{'Variable': 'Age', 'valueid': 11, 'parentid': 3, 'Label': '30-39 yr', 'Color_HexValue': '#44b3c6', 'Enabled': True},
{'Variable': 'Age', 'valueid': 12, 'parentid': 3, 'Label': '40-49 yr', 'Color_HexValue': '#44b3c6', 'Enabled': True},
{'Variable': 'Age', 'valueid': 21, 'parentid': 2, 'Label': '5-11 yr', 'Color_HexValue': '#707070', 'Enabled': True},
{'Variable': 'Age', 'valueid': 22, 'parentid': 2, 'Label': '12-17 yr', 'Color_HexValue': '#707070', 'Enabled': True}
{'Variable': 'Age', 'valueid': 97, 'parentid': 0, 'Label': '< 18', 'Color_HexValue': '#000000', 'Enabled': True},
{'Variable': 'Age', 'valueid': 98, 'parentid': 0, 'Label': '>= 18', 'Color_HexValue': '#000000', 'Enabled': True},

{'Variable': 'Race', 'valueid': 1, 'parentid': None, 'Label': 'White', 'Color_HexValue': '#516889', 'Enabled': True},
{'Variable': 'Race', 'valueid': 2, 'parentid': None, 'Label': 'Black', 'Color_HexValue': '#44b3c6', 'Enabled': True},
{'Variable': 'Race', 'valueid': 3, 'parentid': None, 'Label': 'Hispanic/Latino', 'Color_HexValue': '#d19833', 'Enabled': True},
{'Variable': 'Race', 'valueid': 4, 'parentid': None, 'Label': 'Asian/Pacific Islander', 'Color_HexValue': '#cc5e56', 'Enabled': True},
{'Variable': 'Race', 'valueid': 5, 'parentid': None, 'Label': 'American Indian/Alaska Native', 'Color_HexValue': '#007d8e', 'Enabled': True},

{'Variable': 'Sex', 'valueid': 1, 'parentid': None, 'Label': 'Male', 'Color_HexValue': '#44b3c6', 'Enabled': True},
{'Variable': 'Sex', 'valueid': 2, 'parentid': None, 'Label': 'Female', 'Color_HexValue': '#F2775F', 'Enabled': True},

{'Variable': None, 'valueid': 0, 'parentid': 0, 'Label': 'Overall', 'Color_HexValue': '#000000', 'Enabled': True},
]

All 23 strata are available starting epiweek 200935, inclusive.

The previous version of the GRASP API
(https://gis.cdc.gov/GRASP/Flu3/GetPhase03InitApp) used the following age-id
mapping, as described in the top-level "ages" element, below. See
`./reference/old_grasp_result.json` for a full example response.
'ages' = [
{'label': '0-4 yr', 'ageid': 1, 'color_hexvalue': '#1B9E77'},
{'label': '5-17 yr', 'ageid': 2, 'color_hexvalue': '#D95F02'},
{'label': '18-49 yr', 'ageid': 3, 'color_hexvalue': '#4A298B'},
{'label': '50-64 yr', 'ageid': 4, 'color_hexvalue': '#E7298A'},
{'label': '65+ yr', 'ageid': 5, 'color_hexvalue': '#6AA61E'},
{'label': 'Overall', 'ageid': 6, 'color_hexvalue': '#000000'},
{'label': '65-74 yr', 'ageid': 7, 'color_hexvalue': '#A6CEE3'},
{'label': '75-84 yr', 'ageid': 8, 'color_hexvalue': '#CAB2D6'},
{'label': '85+', 'ageid': 9, 'color_hexvalue': '#1f78b4'}
]

In addition to the new age, race, and sex breakdowns, the group id for overall
reporting has changed from 6 to 0. Age ids 1-5 and 7-9 retain the same the
same meanings; age id 6 is not reported.
"""
HISTORICAL_GROUPS = (
"rate_overall",

"rate_age_0",
"rate_age_1",
"rate_age_2",
"rate_age_3",
"rate_age_4",
"rate_age_5",
"rate_age_6",
"rate_age_7",
)
NEW_AGE_GROUPS = (
"rate_age_18t29",
"rate_age_30t39",
"rate_age_40t49",
"rate_age_5t11",
"rate_age_12t17",
"rate_age_lt18",
"rate_age_gte18",
)
RACE_GROUPS = (
"rate_race_white",
"rate_race_black",
"rate_race_hisp",
"rate_race_asian",
"rate_race_natamer",
)
SEX_GROUPS = (
"rate_sex_male",
"rate_sex_female"
)
EXPECTED_GROUPS = HISTORICAL_GROUPS + NEW_AGE_GROUPS + RACE_GROUPS + SEX_GROUPS


MAX_AGE_TO_CONSIDER_WEEKS = 52


gmpr = GeoMapper()
map_state_names = gmpr.get_crosswalk("state", "state")
map_state_names = map_state_names.to_dict(orient = "records")
map_state_names = {elem["state_name"]: elem["state_id"].upper() for elem in map_state_names}

map_nonstandard_names = {"New York - Albany": "NY_albany", "New York - Rochester": "NY_rochester"}

MAP_REGION_NAMES_TO_ABBR = {**map_state_names, **map_nonstandard_names}

MAP_ENTIRE_NETWORK_NAMES = {
"FluSurv-NET": "network_all",
"EIP": "network_eip",
"IHSP": "network_ihsp"
}
219 changes: 0 additions & 219 deletions src/acquisition/flusurv/flusurv.py

This file was deleted.