Doctor_visits patching code

[minhkhul]

Description

This change automates multi-day patching for doctor_visits indicators, producing csv files organized in batch issue format: [name-of-patch]/issue_[issue-date]/nssp/actual_data_file.csv, making it easier to ingest large batches of patch data.

Users start by specify in params.json the start issue and end issue of the patch, along with a directory where output is stored in. For example:

  "patch": {
    "patch_dir": "/Users/minhkhuele/Desktop/delphi/covidcast-indicators/doctor_visits/AprilPatch",
    "start_issue": "2024-04-20",
    "end_issue": "2024-04-21"
  }

the run patch.py will create a directory 'covidcast-indicators/doctor_visits/AprilPatch' containing:

AprilPatch/
    issue_20240420/
        doctor-visits/
            20240408_state_smoothed_adj_cli.csv
            20240408_state_smoothed_cli.csv
            ...
    issue_20240421/
        doctor-visits/
            20240409_state_smoothed_adj_cli.csv
            20240408_state_smoothed_cli.csv
            ...

[aysim319]

As a general thought: can we introduce using typing for the function parameters? I checked that typing was introduced as a part of the standard library at 3.5 so should be able to use it for this repo. As a newcomer it would be helpful to know the type.

[nmdefries]

@aysim319 can you turn this request into an issue?

[aysim319]

I know we mentioned testing in person yesterday and was wondering if you were planning to add a basic pytest for sanity checks

[nmdefries]

I've made some comments about adding example input/output.

Please add unit tests for download() and get_latest_filename that use the issue_date arg to make sure that is working as desired. Merge at will once that's done.

Everything else looks good.

[nmdefries]

suggestion (optional): since we're using this chunk in multiple places, consider factoring it out into a separate function. (2 duplicates is okay to leave, if we use this in a third place, we definitely should factor it out.)

[nmdefries]

suggestion: add example output

Suggested change

	[name-of-patch]/issue_[issue-date]/doctor-visits/actual_data_file.csv
	[name-of-patch]/issue_[issue-date]/doctor-visits/actual_data_file.csv
	The run using the above example `params.json` will create a directory 'covidcast-indicators/doctor_visits/AprilPatch' containing:
	AprilPatch/
	issue_20240420/
	doctor-visits/
	20240408_state_smoothed_adj_cli.csv
	20240408_state_smoothed_cli.csv
	...
	issue_20240421/
	doctor-visits/
	20240409_state_smoothed_adj_cli.csv
	20240408_state_smoothed_cli.csv
	...

[nmdefries]

question: do we have or do we need a helper script to put these files into the DB? That sounds useful. Due to the output dir structure defined here, this seems like it will need a specialized acquisition script. That would belong in delphi-epidata rather than here.

[minhkhul]

@nmdefries No we don't need new script to go from csv to db. Acquisition on epidata already includes code that allow files in this structure to get put into db using the flag specific_issue_date. I set up this cronicle job that calls acquisition using the flag, and been loading patches that way to our db.