Get initial version of aggregation working

[itsjon]

I believe this is working as expected, pending manual checks.

Near-term improvements:

• Improve facility name matching to reduce the need to manually add more aliases, while not trying to be too clever and accidentally mismatching names. See helpers.data.Mapper. A few options:
  • Ignore punctuation when matching
  • Ensure any whitespace between words in names is treated as a single space, so excess spaces, newlines, etc. won't cause matching to miss an otherwise valid match
  • Assume "CC" == "Correctional Center", and similarly for other common acronyms
  • Ignore acronyms if the acronym is simply an acronym of the preceding words ("California Correctional Institution (CCI)" == "California Correctional Institution")
• Consider updating the merge logic. Currently, when merging multiple rows for the same date for the same facility, each numeric column will receive the largest value for that column among all the merged rows, which means columns may not sum as expected if the merged row draws from multiple source rows. See helpers.merging.merge_rows.
• Add County column, drawn either from the master mapping file or the source datasets
• Decide what, if anything, to do with the Inmates Pending column in the covidprisondata.com dataset

[jessex]

Thanks a ton for getting this off the ground, @itsjon! Is the idea here that the user will manually go grab the latest files from a given source, add them to data/{source}/ and then run the output script? If so, let's add some general operating instructions to the readme that basically say, "Get the latest files for a given source, place them in the right location, generate the output, then commit the latest files and the output to this repository to keep it publicly up-to-date."

[jessex]

So this conditional (if source_id in sources_used) depends on sources_used having been populated by the previous for loop, right? I think that's somewhat non-intuitive. Why would it be that the presence of a source in the numeric columns would dictate its presence in the text columns?

[itsjon]

At least for the existing text columns, Source and Notes, it doesn't seem to make sense to include that information from an input row in an output row if the numeric values from that input row weren't ultimately used in the output row. If that doesn't make sense to you, let's file an issue and circle up with C.

[jessex]

Okay I buy that. Let's proceed as is for now!

[jessex]

If we do change get_value to return None instead of an empty string, we should ensure that this particular call returns an empty string if the function returns None or have L45 below translate from None to empty string.

[itsjon]

Yeah, any row that doesn't have a value for the column at hand is filtered out before determining the row with the max value, so if get_value is invoked here, it will definitely have some value. Else, if no rows with a value for the column were provided, the empty string is returned.

[jessex]

Great job, @itsjon! Thanks for the headstart on this.

[itsjon]

@jessex sorry I missed your reply. Thank you, and you're welcome!