update: new web agg write #105

nmmarquez · 2021-06-03T14:59:51Z

New aggregate function which aggregates data by different jurisdictions. The function logic is very similar to calc_aggregate_counts except each state has a count for each jurisdiction. The main part of the code is here.

ucla_df <- read_scrape_data(
        window = window, all_dates = all_dates, wide_data = FALSE) %>%
        mutate(Web.Group = case_when(
            Jurisdiction == "immigration" ~ "ICE",
            Jurisdiction == "federal" ~ "Federal",
            Age == "Juvenile" ~ "Juvenile",
            Jurisdiction == "state" ~ "Prison",
            Jurisdiction == "psychiatric" ~ "Psychiatric",
            Jurisdiction == "county" ~ "County",
            TRUE ~ NA_character_
        ))

Leaving this up as a draft for now to see if we agree on this categorizing and how we want to deal with incorporating MP data.

erika-tyagi

Yay thank you for doing this! And thank you for adding max_na_rm as a function too! A few little thoughts:

Do you think it's worth having a state param that works the same way as in the other calc_aggregate_counts? Obviously extremely easy to get the state = F version from the output of this, so not sure if it's worth it.
Is there a reason you called the web grouping Prison instead of State? (thinking about federal prisons)
Re: MP – I think we have to assign the whole totals to the state web grouping...? Probably whenever MP value > our value (as opposed to just when we don't have a value) to catch Texas, Ohio, etc., I think?

One thing I'm still iffy about is rates (literally never sure about rates). For State-Web.Group combinations where we have Residents.Population and/or Population.Feb20 for ALL (or basically all) facilities, that seems super straightforward. And for combinations where we don't have ANY population data, that also seems like a straightforward NA situation. But when we have population values for some (but not all) facilities in a grouping, do we use the total numerator for the measure but the partial denominator? Even if that overestimates rates? I honestly don't know what we're doing right now on the website to address this, so might be a perfect enemy of good situation.

erika-tyagi · 2021-06-03T20:33:09Z

R/alt_aggregate_counts.R

+            Jurisdiction == "immigration" ~ "ICE",
+            Jurisdiction == "federal" ~ "Federal",
+            Age == "Juvenile" ~ "Juvenile",
+            Jurisdiction == "state" ~ "Prison",
+            Jurisdiction == "psychiatric" ~ "Psychiatric",
+            Jurisdiction == "county" ~ "County",


The order here means that

facilities that are Juvenile and immigration or federal would NOT be Juvenile, but

facilities that are Juvenile and county, psychiatric, or state WOULD be

There's only one federal or immigration row this affects right now (population data for ALL BOP CONTRACT JUVENILES), so I feel like it's fine the way it is – but maybe worth adding a comment for future selves!

erika-tyagi · 2021-06-04T13:42:03Z

R/alt_aggregate_counts.R

+        ))
+
+    fac_long_df <- ucla_df %>%
+        filter(State != "Not Available") %>%


Thoughts on leaving these in and just keeping the state as Not Available? Means we'd keep both federal / ice facilities that we haven't gotten around to adding yet, but also federal aggregations (e.g. RRCs) where they're cross-state so the actual assigned state is Not Available.

I like this idea but I'm not sure, practically, whether this matters. If state = "Not Available", would the numbers show up anywhere on the website? I don't think so

I think they actually do show up on the website, just with the jurisdiction where the name of the state would be!

just checking, since these are aggregated they wouldn't show up here right? this is only for "facilities"

that's my understanding!

hjohns12 · 2021-06-21T20:52:04Z

R/alt_aggregate_counts.R

+            Age == "Juvenile" ~ "Juvenile",
+            Jurisdiction == "state" ~ "Prison",
+            Jurisdiction == "psychiatric" ~ "Psychiatric",
+            Jurisdiction == "cou nty" ~ "County",


misspelling here on "county"

hjohns12

Aside from one tiny typo, this all looks good to me! Thank you so much for dealing with all this craziness and using good variable names to wit :)

erika-tyagi

This looks wonderful! Made one suggestion, and I still feel like we should keep the Not Available state rows, but overall looks great // thank you so much for working through this!!

erika-tyagi · 2021-06-21T21:51:40Z

R/alt_aggregate_counts.R

+        pri_df <- state_df %>%
+            filter(Web.Group == "Prison") %>%
+            full_join(mp_df, by = c("State", "Date", "Measure"))


Do we want the Web.Group for everything from MP to be Prison? If so, should we assign that here? (or wherever might make more sense) otherwise there are NA Web.Groups so that data won't make it onto the website :(

…hindbarstools into new_agg_func

nmmarquez · 2021-06-29T14:44:18Z

Added back in the "state not available" and changed the calculation of rate to include values for population in the denominator even when we dont have a numerator. This makes our rates more comparable to Marshall project and makes our denominators more sensible.

erika-tyagi · 2021-06-29T15:39:19Z

R/read_mpap_pop_data.R

+#'
+#' @examples
+#' \dontrun{
+#' read_staff_popfeb20()


I think this should be read_mpap_pop_data().

erika-tyagi · 2021-06-29T15:41:48Z

R/read_staff_popfeb20.R

@@ -0,0 +1,25 @@
+#' Get Feb 20, 2020 population data for applicable rows in the fac_data


Do we use this function anywhere? // I think it's useful, but curious why it ended up here!

I wanted to add it in if were to start using staff data denoms anywhere. Not using yet.

erika-tyagi

Yay ty ty lgtm! Added a few last comments, but I don't think they need to preclude merging this. I guess the next step is to update write_latest_data // let HO know?

update: new web agg write

135f1df

nmmarquez requested review from erika-tyagi, hjohns12 and mceverett8 June 3, 2021 14:59

nmmarquez marked this pull request as draft June 3, 2021 15:01

nmmarquez linked an issue Jun 3, 2021 that may be closed by this pull request

New aggregation function #99

Closed

erika-tyagi reviewed Jun 4, 2021

View reviewed changes

nmmarquez added 2 commits June 14, 2021 13:23

Merge branch 'master' into new_agg_func

0581f09

update: add in rates for alt agg counts

e0c77c0

nmmarquez marked this pull request as ready for review June 15, 2021 17:55

update: add pop threshold check

8a6eab3

hjohns12 reviewed Jun 21, 2021

View reviewed changes

hjohns12 approved these changes Jun 21, 2021

View reviewed changes

erika-tyagi requested changes Jun 21, 2021

View reviewed changes

erika-tyagi and others added 3 commits June 29, 2021 09:59

fix: remove WIP fn

e7e96df

update: alter sums to include denoms even when numer is missing

de641fd

Merge branch 'new_agg_func' of github.com:uclalawcovid19behindbars/be…

f249797

…hindbarstools into new_agg_func

erika-tyagi reviewed Jun 29, 2021

View reviewed changes

erika-tyagi approved these changes Jun 29, 2021

View reviewed changes

Update read_mpap_pop_data.R

9336bee

nmmarquez merged commit 1e688d7 into master Jun 29, 2021

erika-tyagi deleted the new_agg_func branch July 21, 2021 16:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update: new web agg write #105

update: new web agg write #105

nmmarquez commented Jun 3, 2021

erika-tyagi left a comment

erika-tyagi Jun 3, 2021

erika-tyagi Jun 4, 2021

hjohns12 Jun 7, 2021

erika-tyagi Jun 7, 2021

nmmarquez Jun 15, 2021

hjohns12 Jun 17, 2021

hjohns12 Jun 21, 2021

hjohns12 left a comment •

edited

erika-tyagi left a comment

erika-tyagi Jun 21, 2021

nmmarquez commented Jun 29, 2021

erika-tyagi Jun 29, 2021

erika-tyagi Jun 29, 2021

nmmarquez Jun 29, 2021

erika-tyagi left a comment

		@@ -0,0 +1,25 @@
		#' Get Feb 20, 2020 population data for applicable rows in the fac_data

update: new web agg write #105

update: new web agg write #105

Conversation

nmmarquez commented Jun 3, 2021

erika-tyagi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hjohns12 left a comment • edited

Choose a reason for hiding this comment

erika-tyagi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nmmarquez commented Jun 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erika-tyagi left a comment

Choose a reason for hiding this comment

hjohns12 left a comment •

edited