Add `popsafe-fips` to geomapper #1787

nmdefries · 2023-02-21T16:46:05Z

Description

Create new level in the geomapper to support changes to chng data. Background and design considerations.

CHNG would like to stop publishing county-level information for counties with low population (<20k). They have proposed a list of ~400 county groups which bundle together low-population counties until the population is above the 20k threshold. The groups are reasonably contiguous and combining data from low-population areas is a common technique to alleviate privacy concerns

To allow this, add a popsafe-fips level to the geomapper that codes high-population (>20k) counties with their official FIPS codes and low-population counties with a code of the format <two digit state FIPS>g<two digit group number within the state>, e.g. "01g05". Add mappings for fips -> popsafe-fips and popsafe-fips -> state.

Changelog

Original county groups file, data_proc/geomap/lowpop_county_groups.csv
geo_data_proc.py - functions to generate popsafe-fips map tables from input county groups file
2019 and 2020 files mapping fips -> popsafe-fips and popsafe-fips -> state
geomap.py
test_geomap.py

Doing so makes it easier to add new mappings by reducing the number of lines with hard-coded geo/key names.

nmdefries · 2023-02-21T18:52:27Z

The provided county groupings don't seem to meet the stated requirements. Using our population data from 2020, there are many ungrouped counties and county groups with populations less than 20k (data file generated with script).

Some counties and county groups are just barely below the cutoff and would likely meet the 20k threshold using updated population data, but there are many counties that are far (>50%) below the population requirement.

chng could purposefully have not bothered grouping counties they don't report, however, all but 46 of the ungrouped counties included in the csv above were reported by chng in the last three weeks. (The 46 unreported counties are mostly territories and low-pop counties in Alaska.)

krivard · 2023-02-23T14:55:50Z

I'm checking the population results with Mina and will hold my final review until that gets resolved. Thank you for doing the initial analysis!

nmdefries · 2023-02-24T17:28:34Z

The data provider wants to keep the county groups as-is. We should rename the new level to make it clear that it isn't generally applicable nor "population safe". Maybe chng_fips (along the same lines as the indicator-specific geo level jhu_uid)?

dshemetov

Looks good. Thanks for adding good tests too! Just one (non-blocking) question: does CHNG have any prop signals? Might be good to add some population methods if so.

_delphi_utils_python/data_proc/geomap/geo_data_proc.py

dshemetov · 2023-02-28T06:08:04Z

_delphi_utils_python/delphi_utils/geomap.py

-        elif contained_geocode_type == "county" and container_geocode_type == "state":
-            crosswalk = self._crosswalks["fips"]["state"]
-            return set(crosswalk[crosswalk["state_id"] == container_geocode]["fips"])
+        elif (contained_geocode_type in ("county", "fips", "popsafe-fips") and


praise: thanks for enforcing consistency!

krivard · 2023-03-01T22:17:14Z

@nmdefries
I'd accept any of the following:

chng-fips
chng-county

(use dashes so it's easier to parse out of the CSV filenames)

krivard

Modulo the name change, looks great!

krivard · 2023-03-01T22:24:04Z

_delphi_utils_python/delphi_utils/geomap.py

            "pop": int,
-            "weight": float
+            "weight": float,
+            **{geo: str for geo in self._geos - set("nation")}


praise: nicely crystallized!

nmdefries · 2023-03-02T16:37:56Z

Will go with chng-fips for the new name.

nmdefries · 2023-03-02T18:08:07Z

does CHNG have any prop signals? Might be good to add some population methods if so.

CHNG doesn't currently report any prop signals or directly use any population data, so we don't need to do this.

krivard

(best remove all implied relationships between chng groups and population)

_delphi_utils_python/data_proc/geomap/geo_data_proc.py

Co-authored-by: Katie Mazaitis <krivard@cs.cmu.edu>

nmdefries · 2023-03-03T20:36:49Z

@krivard I won't get to splitting the fips_list field today. This PR works as-is so we could merge if someone will be working on getting changehc to use this in the next week while I'm gone. Otherwise, I'll pick up this and #1803 when I get back.

krivard

CHNG probably needs another couple weeks to fix their end, but approving this provisionally anyhow

We can't rely on the component counties of a group being listed individually, so reconstruct those fields ourselves based on the concatenated `fips_list` field. Split by the separator (pipe `|`) and save each result to a new column.

nmdefries · 2023-03-13T14:27:45Z

Splitting the fips_list field has been added in #1803.

…geomapper/chng-fips-map-generation

Co-authored-by: Dmitry Shemetov <dshemetov@ucdavis.edu>

nmdefries · 2023-03-13T19:40:20Z

@korlaxxalrok This is ready to merge.

…-map-generation generate local county mapping from CHNG spreadsheet

nmdefries added 9 commits February 21, 2023 10:37

add popsafe county -> fips and state data files

4d04b19

construct class attributes using CROSSWALK keys

b3a03d4

Doing so makes it easier to add new mappings by reducing the number of lines with hard-coded geo/key names.

support popsafe-fips in geomap.py; get_geos_within to take "fips"

0284c28

combine fips and popsafe-fips get_geo_within logic; comments

1f8a21b

code to generate popsafe tables

fb82be4

document group assignment procedure

f284b91

add lowpop county group mapping file

ba62241

test popsafe-fips

a1c491c

linting

f74aefd

nmdefries marked this pull request as ready for review February 21, 2023 18:51

nmdefries requested review from dshemetov and krivard February 21, 2023 18:51

dshemetov approved these changes Feb 28, 2023

View reviewed changes

krivard suggested changes Mar 1, 2023

View reviewed changes

change code name

1a1bba7

nmdefries requested a review from krivard March 3, 2023 17:51

generate local county mapping from CHNG spreadsheet

bb7df1a

krivard suggested changes Mar 3, 2023

View reviewed changes

nmdefries and others added 2 commits March 3, 2023 15:22

remove "population" wording in relation to chng-fips

be8da31

Co-authored-by: Katie Mazaitis <krivard@cs.cmu.edu>

rename local county groups csv to not mention population

54faa45

krivard approved these changes Mar 6, 2023

View reviewed changes

nmdefries added 2 commits March 13, 2023 10:14

drop split-out county fields; don't always exist in input

18a08a5

split out single counties from fips_list field

0ef6b13

We can't rely on the component counties of a group being listed individually, so reconstruct those fields ourselves based on the concatenated `fips_list` field. Split by the separator (pipe `|`) and save each result to a new column.

raise error if new groups seen

f9ed4d4

nmdefries and others added 3 commits March 13, 2023 10:29

state_fips and group are already str; don't need to cast

4fa39b7

Merge branch 'ndefries/geomapper/popsafe-county-level' into ndefries/…

4b9e836

…geomapper/chng-fips-map-generation

make error more actionable

dc02bb5

Co-authored-by: Dmitry Shemetov <dshemetov@ucdavis.edu>

Merge pull request #1803 from cmu-delphi/ndefries/geomapper/chng-fips…

f785b22

…-map-generation generate local county mapping from CHNG spreadsheet

korlaxxalrok merged commit 3943bd4 into main Mar 14, 2023

korlaxxalrok deleted the ndefries/geomapper/popsafe-county-level branch March 14, 2023 16:13

krivard mentioned this pull request Mar 29, 2023

Release covidcast-indicators 0.3.34 #1816

Merged

nmdefries mentioned this pull request Aug 4, 2023

Support chng-fips county group reporting in chng for privacy #1883

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `popsafe-fips` to geomapper #1787

Add `popsafe-fips` to geomapper #1787

nmdefries commented Feb 21, 2023

nmdefries commented Feb 21, 2023

krivard commented Feb 23, 2023

nmdefries commented Feb 24, 2023

dshemetov left a comment

dshemetov Feb 28, 2023

krivard commented Mar 1, 2023

krivard left a comment

krivard Mar 1, 2023

nmdefries commented Mar 2, 2023

nmdefries commented Mar 2, 2023

krivard left a comment

nmdefries commented Mar 3, 2023

krivard left a comment

nmdefries commented Mar 13, 2023

nmdefries commented Mar 13, 2023

Add popsafe-fips to geomapper #1787

Add popsafe-fips to geomapper #1787

Conversation

nmdefries commented Feb 21, 2023

Description

Changelog

nmdefries commented Feb 21, 2023

krivard commented Feb 23, 2023

nmdefries commented Feb 24, 2023

dshemetov left a comment

Choose a reason for hiding this comment

dshemetov Feb 28, 2023

Choose a reason for hiding this comment

krivard commented Mar 1, 2023

krivard left a comment

Choose a reason for hiding this comment

krivard Mar 1, 2023

Choose a reason for hiding this comment

nmdefries commented Mar 2, 2023

nmdefries commented Mar 2, 2023

krivard left a comment

Choose a reason for hiding this comment

nmdefries commented Mar 3, 2023

krivard left a comment

Choose a reason for hiding this comment

nmdefries commented Mar 13, 2023

nmdefries commented Mar 13, 2023

Add `popsafe-fips` to geomapper #1787

Add `popsafe-fips` to geomapper #1787