URL strdatetime problem in WeatherFaxInternetRetrieval.xml

[rgleason]

Australia Met
Singapore Met
France Met
India Met
UK Met
Uk Met Office
and others have changed their URL for weatherfaxes to have a strdatetime code at the end, rather than having a static URL for the current files. This causes frequent revisions and updates to the XML files.

http://www.cruisersforum.com/forums/f134/weatherfax-97533.html#post2458260
Stelian42 has provided some code for reading python scripts which can do strdatetime.
https://github.com/stelian42/weatherfax_pi
I have implemented the ability to call an external helper (script) from weatherfax in order to parse the HTML pages and find the correct URL for the weather faxes.
Find this here: https://github.com/stelian42/weather...83518796f6924d
This should be tested with Lyndon's script (which I didn't see...), and if it works well, I'll push everything to Sean.

http://www.cruisersforum.com/forums/f134/weatherfax-97533.html#post2460695
Lydon LJH was going to provide a python script for Australia.

I plan on testing Stelian's script, but got sidetracked fixing other things in weatherfax.

[rgleason]

From the urls wih date and hour commit
325fa72

url_hour_round_offset=0

const char*hour = g->Attribute( "Hour" );
+     if(hour)
+     sscanf(hour, "%d;%d;%d", &url.hour_offset, &url.hour_round, url.hour_round_offset);

  /* compute actual url */
+        wxString url = faxurl->Url;
+        wxDateTime now = wxDateTime::Now();

       // now deal with hour
+        int hour = now.GetHour(wxDateTime::UTC);
+        if(faxurl->hour_round) {
+            int nhour = wxRound(((double)hour+faxurl->hour_offset-faxurl->hour_round_offset)/faxurl->hour_round)*faxurl->hour_round+faxurl->hour_round_offset;
+            now+=wxTimeSpan::Hours(nhour - hour);
+        }
+            
+        wxString formats[] = {"%Y", "%y", "%m", "%d", "%H"};
+        for(unsigned int i=0; i<(sizeof formats) / (sizeof *formats); i++)
+            url.Replace(formats[i], now.Format(formats[i], wxDateTime::UTC));
+

It would be helpful to have this explained a little bit.

Obviously %Y is the full year (2018) and %y is a decayear (18)
Then %m is the numeric month (01...12)
and %d is the numeric day (01...31)
Next would be %H which is hours, but I am not sure how hours works. The "H" is capitalized, I am not sure what that means...
Does 3 days x24 = 72 hours, so %H could be 72?

When we have a normal image url like
http://www.weather.gov.sg/files/marine/surfacewinds/surfacewinds_24Hr_20180228_213000.gif
`http://www.weather.gov.sg/files/marine/surfacewinds/surfacewinds_24Hr_%Y%m%d_213000.gif'
are the numbers after '213000' something to do with hours? How would we represent that?

Also what do we do if a particular image that is expected at a certain time or UTC hour is not normally uploaded until an hour later? Is there a way to have an hour offset or something?

How do we enter the hour schedule for something like every three hours? or every 6 hours or every 12 hours? (0,3,6,9,12,15,18,21,24) ?
(0,6,12,18,24)?
(0,12,24)?
Can it be something like (4,8,12,16,20,24)? or (3,9,15,21)?
What happens and when do you use this?
It would be nice to have some syntax and formatting rules and a few examples.
I would add them to the manual too.

[seandepagnier]

rick You guessed right. These are standard date/time formats The date and time is current UTC. The Hour field in the xml provides an offset, rounding and rounding offset. So if you want every 3 hours, you would use the offset either 0, -3, -6 etc for past, or 3, 6, 9 future. Then 3 for the rounding, and finally 0 for rounding offset. Rounding offset is needed in japan which provides hours like 3, 9, 15, 21, and so the rounding offset is 3.

…

On 3/1/18, Rick Gleason ***@***.***> wrote: From the urls wih date and hour commit 325fa72 ``` url_hour_round_offset=0 const char*hour = g->Attribute( "Hour" ); + if(hour) + sscanf(hour, "%d;%d;%d", &url.hour_offset, &url.hour_round, url.hour_round_offset); /* compute actual url */ + wxString url = faxurl->Url; + wxDateTime now = wxDateTime::Now(); // now deal with hour + int hour = now.GetHour(wxDateTime::UTC); + if(faxurl->hour_round) { + int nhour = wxRound(((double)hour+faxurl->hour_offset-faxurl->hour_round_offset)/faxurl->hour_round)*faxurl->hour_round+faxurl->hour_round_offset; + now+=wxTimeSpan::Hours(nhour - hour); + } + + wxString formats[] = {"%Y", "%y", "%m", "%d", "%H"}; + for(unsigned int i=0; i<(sizeof formats) / (sizeof *formats); i++) + url.Replace(formats[i], now.Format(formats[i], wxDateTime::UTC)); + ``` It would be helpful to have this explained a little bit. Obviously %Y is the full year (2018) and %y is a decayear (18) Then %m is the numeric month (01...12) and %d is the numeric day (01...31) Next would be %H which is hours, but I am not sure how hours works. The "H" is capitalized, I am not sure what that means... Does 3 days x24 = 72 hours, so %H could be 72? When we have a normal image url like `http://www.weather.gov.sg/files/marine/surfacewinds/surfacewinds_24Hr_20180228_213000.gif` `http://www.weather.gov.sg/files/marine/surfacewinds/surfacewinds_24Hr_%Y%m%d_213000.gif' are the numbers after '213000' something to do with hours? How would we represent that? Also what do we do if a particular image that is expected at a certain time or UTC hour is not normally uploaded until an hour later? Is there a way to have an hour offset or something? How do we enter the hour schedule for something like every three hours? or every 6 hours or every 12 hours? (0,3,6,9,12,15,18,21,24) ? (0,6,12,18,24)? (0,12,24)? Can it be something like (4,8,12,16,20,24)? or (3,9,15,21)? What happens and when do you use this? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #58 (comment)

[rgleason]

Sean this is very helpful. Not sure I have all the concepts and format right.
So using the Japan server example:

  </Server>
    <Server Name="Japan Met" Url="http://www.jma.go.jp/en/g3/images/asia/">
    <Region Name="Japan">
      <Map Url="%y%m%d%H.png" Contents="Analysis Now" Area="1" Hour="6;6;3" />
      <Map Url="%y%m%d%H.png" Contents="Analysis -6" Area="1" Hour="0;6;3" />
      <Map Url="%y%m%d%H.png" Contents="Analysis -12" Area="1" Hour="-6;6;3" />
      <Map Url="%y%m%d%H.png" Contents="Analysis -18" Area="1" Hour="-12;6;3" />
      <Area Name="1" lat1="0N" lat2="70N" lon1="90E" lon2="180E" />
    </Region>

The first line "Analysis Now"

The Hour="6;6;3" is for
Hour=
6 hr offset; So it will show 6 hrs later for "now"?
6 hr rounding; So it will increment by 6 hrs and just use 6 hour intervals?
3 hr rounding offset Not so sure about this one. What is it for?

Japan Standard Time (JST) is 9 hours ahead of Coordinated Universal Time (UTC), and we EST =UTC-5

I have another little problem with Singapore server. I've gotten this far:

<Map Url="surfacewinds/surfacewinds_24Hr_%Y%m%d_213000.gif" Contents="Surface Wind Forecast 24hr" Area="1" />
but I don't have a clue what the 213000 is for. Is that hours or something? It seems rather random but seems to be necessary, and it changes.

[rgleason]

Found this reference http://www.cplusplus.com/reference/ctime/strftime/

%Y | Year 2018
%y | Year last two digits (18)
%m| Month two digits (02)
%d| Day with two digits (01...31)
%H | Hour in 24h format (00-23)

There is also http://www.cplusplus.com/reference/ctime/gmtime/
I am looking for rounding offset etc.

[seandepagnier]

The rounding offset for japan is 3 rounding offset of 0: 0, 6, 12, 18 rounding offset of 3: 3, 9, 15, 21

…

On 3/1/18, Rick Gleason ***@***.***> wrote: Found this reference http://www.cplusplus.com/reference/ctime/strftime/ %Y | Year 2018 %y | Year last two digits (18) %m| Month two digits (02) %d| Day with two digits (01...31) %H | Hour in 24h format (00-23) There is also http://www.cplusplus.com/reference/ctime/gmtime/ I am looking for rounding offset etc. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #58 (comment)

[rgleason]

<Map Url="%y%m%d%H.png" Contents="Analysis -6" Area="1" Hour="0;6;3" />
Hour=
"0; no hour offset, so it is the current UTC
6; rounding, rounds to 6 hours
3" rounding offset, shifts the hours used by 3 hours

So I would think this statement would be showing 3 hours but it says "Analysis-6"

[seandepagnier]

The -6 is relative to now

…

On 3/2/18, Rick Gleason ***@***.***> wrote: ` <Map Url="%y%m%d%H.png" Contents="Analysis -6" Area="1" Hour="0;6;3" />` Hour= "0; no hour offset, so it is the current UTC 6; rounding, rounds to 6 hours 3" rounding offset, shifts the hours used by 3 hours So I would think this statement would be showing 3 hours but it says "Analysis-6" -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #58 (comment)

[rgleason]

Sean I must be thick, but isn't "-6" in the past?
Aren't we more interested in the future? So shouldn't it be 6?
Still trying to get this right in my head.

[rgleason]

This "Issue" should become an actively edited page in the manual! I intend to make that happen soon.

BTW Ozolli has updated weatherfaxinternetretreval.xml and bdbcat has fixed AUS BOM downloads by using curl library and identifying the browser. There is still some work needed for flatpak, but it is working to build flatpak now too, needs testing.