Have nowcasting acquisition use issue date from file

[chinandrew]

Since nowcasting uses sensor values computed as_of certain dates, the issue dates need to be the as_of dates and not datetime.today() like other acquisitions. These as_of dates are specified during sensorization (it calls the covidcast as_of indicator data to generate the sensors) and added to the files to be ingested as a new column.

cc @mariajahja

[krivard]

Just to confirm, when running in production you expect each line of the CSV might have a different issue date?

[chinandrew]

Just to confirm, when running in production you expect each line of the CSV might have a different issue date?

That's supported here, though the way it's currently implemented only 1 issue gets produced at a time so the CSV files should be homogeneous in that field

[krivard]

If you don't have a use case that needs a different issue per line, I'd prefer putting the issue into the filename instead of the file content -- saves a ton on disk space for the archived csvs.

[chinandrew]

If you don't have a use case that needs a different issue per line, I'd prefer putting the issue into the filename instead of the file content -- saves a ton on disk space for the archived csvs.

That makes sense to me. Since I'm sharing the covidcast CsvImporter that auto-detects source/geo/signal from the filename, would you recommend embedding the issue into signal name and parsing it out later (e.g. make the signal name _), adjusting CsvImporter, ,writing new code to parse, or something else

[krivard]

Borrow the issue-specific importer instead 😄

delphi-epidata/src/acquisition/covidcast/csv_importer.py

Line 82 in 2395b48

def find_issue_specific_csv_files(scan_dir, glob=glob):

Format is issue_{issue}/{source}/{normal covidcast filename}

[chinandrew]

Borrow the issue-specific importer instead

delphi-epidata/src/acquisition/covidcast/csv_importer.py

Line 82 in 2395b48

def find_issue_specific_csv_files(scan_dir, glob=glob):

Format is issue_{issue}/{source}/{normal covidcast filename}

Oh cool, didn't know about that. Thanks for pointing it out, I'll work on integrating it now

[chinandrew]

@krivard added issue specific csv importer, ready for review again

[krivard]

xlnt