Increase Script / Data reliability

[franTarkenton]

At the moment the script is setup to run a single date. Sometimes this doesn't work as the modis data or the viirs data is not available when the script runs.

This epic will implement the following:

• when the pipeline runs it will evaluate what data we already have and the data that is available and process based on that configuration
• Separate the steps taken in the dailypipeline so that if one step fails it doesn't impact other unrelated scripts.
• Reconfigure scripts so that they can automatically pickup where they last got to.
  • when data is downloaded immediately push it back to object storage.
  • before downloading data check to see if the data exists in ostore first
  • general improvements to try to make the code more readable. (use utility methods with descriptive names vs file name / date / data hacking using string manipulation regex's etc.) Methods should also provide clear description of what they do with examples

Dependencies

Epic

• Modify Daily pipeline so fills in missing data for a given time window #44
• Implement timeout on async functions #60
• Modis / Download and process step #63
• Processing: as new data files get generated, upload them to object storage #67