Code quality improvements and HWRF fix

Issues have been fixed

1. Generate HWRF output when there is no HWRF data release: Issue 41
   It compares the current time to HWRF release schedule, and generates the outputs in the similar 6-hours interval.
2. Fix the performance issue when generating MoM final output. Issue 55
   The "final_alert_pdc" function is totally rewritten, and it takes seconds to generate the final output instead of minutes.

Improvements

Issue 50

1. added several time-related functions to check and compare the current time and HWRF data release schedule.
2. added a wrapper function to simply the call to generate MoM output.
3. use sort by iso time string to get the latest file instead of the creation time.
4. formatted the code with Black Code Formatter

How to update

"git pull" is enough

How to test

Check the final outputs when there is no HWRF data release, the final outputs shall be generated at least three times per day.

HWRF fix and minitor service update

Issues have been fixed.

The recent change in HWRF has caused that all four MoM outputs in a day generated at once in the morning.
-- change in HWRF data #32
-- fix in HWRF ingestion #41

The changes in this fix:
 1. if there is HWRF data listed for the day, it will try to generate the outputs roughly in six hours schedule.
 2. if there is no HWRF data for the day, it will generate outputs two times per day as possible, one in the morning (for 00-hour, after GLOFAS processed), the others (06, 12, 18) generated once late at night (after DFO processed).

Other minor fixes:
-- incomplete GFMS data #38
-- clean out unnecessary print #36
-- final output update #30
-- more logging

Updates in monitor service

• SMTP support has been merged into monitor.py from monitor_dev.py, no need to update any configuration file.
• False warning #26: when HWRF falls behind, it will be marked as read, however, it will not impact the system status, it also tests if there is HWRF data released or not.

Operation Status: Normal
Report time: 20221115, 20:25:02
Data Processing
GLOFAS : 20221115 : threspoints_2022111500.csv
GFMS : 20221115 : Flood_byStor_2022111521.csv
HWRF : 20221113 : hwrf.2022111318rainfall.csv
HWRF data release at 20221115 : False
DFO : 20221114 : DFO_20221114.csv
VIIRS : 20221114 : VIIRS_Flood_20221114.csv

How to update

"git pull" is enough, the fix will take efforts next morning.

DFO GFMS fix and optional Monitor module

What's been fixed

• Upstream DFO data release: #16
• Missing GFMS outputs: #18

What's new

• Optional module: Monitor Service
• Updated Server Setup Guide:
  • Setup the timezone at UTC
  • Guide on disk space management

How to update

1. Disable current cronjobs

run crontab -e and delete or comment out all the cronjobs

2. Update the code

run git pull to update the code to the latest version.

3. Download the data from the testing server

• create a temporary folder for downloading
• in this folder, run
  wget -m -R .html,.tmp,.gif,.png -nH "https://js-170-28.jetstream-cloud.org/ModelofModels/" -P .
• overwrite the folders under Products with the new data downloaded in ModelofModels

4. Set timezone to UTC

Setup the timezone at UTC

5. Update the cronjobs

run crontab -e, and copy/paste the content in crontab_list.txt. Please check if the path is right.

6. Optional Monitor Service

Guide on Monitor Service

Report issues and discussion

Emails
GitHub discussions
GitHub issues
or other preferred channels