windrun calculation

[windymel]

Hi. It appears that the windrun record shows the maximum observed windrun in a 5 minute archive period. I think this is because in versions of weewx <= 3.9.1 the windrun calculation in the archive packet was the sum of the day's windrun. However, this was identified as an issue and inconsistent with other calculated quantities. The calculation was changed in version 3.9.2 to be the windrun for the archive period (refer [https://github.com/weewx/weewx/issues/250]) . Consequently, the code in Belchertown needs to be updated for consistency with this change.

[poblabs]

The code within Belchertown does not calculate wind run. Can you tell me where you see that?

Everything within the skin with reference to wind run is using the weewx built in tags (e.g. $year.windrun.maxtime.raw)

https://github.com/poblabs/weewx-belchertown/search?utf8=%E2%9C%93&q=windrun&type=

[poblabs]

I'm looking at this again and everything I can find is with the weewx tags which are not generated from Belchertown. Belchertown gets that data from weewx directly when using a tag.

Are you saying the MQTT payload contains incorrect data? If so, the MQTT payload data isn't being calculated by Belchertown. It's just showing the data the weewx-mqtt extension is publishing. If the MQTT payload data is wrong, then mwall needs to be made aware of with his extension (post a message to the weewx-user forums)

Let me know what I am missing?

[windymel]

Sorry, I wasn't super clear with my statement above. Basically, since 3.9.2, weewx has changed how it stores windrun in the database. It used to be (3.9.1 and earlier) that the field "windrun" recorded the sum of the daily windrun on each day. So your weewx db entries for windrun would look like 1.2, 2.3, 2.9, 3.7,... Consequently for displaying windrun in Belchertown then you can just take the maximum over the day and compare this for each day to get the record. All good so far.

However, since 3.9.2, the weewx calculation has changed so that it records the windrun for each archive period, rather than the running sum. The values in the DB above would then be 1.2, 0.9, 0.6, 0.8,... This change was made to make it more consistent with other values in the DB such as rain.

I think to fix the implementation in Belchertown, all you need to do is take the "sum" value for windrun for each day, rather than the "max".

I hope that makes sense.

[poblabs]

Ah that does make sense, thanks! The only areas I see windrun max used are on the records page, so I'll get those updated to sum.

Thanks for taking the time to explain it, I was a little lost 😃

[windymel]

no problem. Just to be clear, I think you should use the following two aggregation types:

maxsum	The maximum daily sum in the aggregation period. Aggregation period must be one day or longer.
maxsumtime	The time of the maximum daily sum.

[windymel]

by the way, thanks for the great work on the skin. It's really fantastic. I much prefer it over the Seasons skin.

[poblabs]

Glad you enjoy it!

Interesting; maxsum and not just sum?

[windymel]

I think so. Because it's a record, I think you want the aggregation period to be one year and then take the maxsum (which is the maximum daily sum over the aggregation period). I think the skin grabs the current year daily max as the record and the all time record. Not sure how you get the all time, maybe you have to iterate over all years in the DB and get the maxsum for each year, and then take the max of that?

[poblabs]

The all time is given to me by weewx as well (through some of their example code), and I think that does exactly what you're describing (it's been a while since I read that section of code).

If you'd like to try it out, you can edit your skins/Belchertown/records/index.html.tmpl and search for windrun and update those values. Upon your next archive interval the report should be generated with the new values. I'm going to be remote for a few hours but if you're feeling up to it let me know the results.

[windymel]

No problem. I had a go and it seems to be working well. Will do a pull request and then you can have the code.

[poblabs]

Great! You deserve the credit

[poblabs]

Merged, I'll add it to development too. Thanks again

[poblabs]

How's this looking for you now after letting it sit a few months?

I think it's wrong and I need to roll it back...

Here's my Highest Daily Wind Run

2019	22617.05 miles	February 10, 2019
All Time	January 11, 2016	42664.02 miles

[windcrusader]

It's working perfectly for me. I think your issue is because you probably have archive values in your DB that are cumulative, rather than for the archive period. I have a related issue in that my windrun values from prior to when I changed the DB schema to include windrun are not populated. I think you probably have to run a script over your DB to recalculate the windrun for each archive period.

Also, I note that the change to this type of calculation was only made to 3.9.2 of WeeWX. If you are still using an earlier build, then it will still be adding cumulative values.

In summary, I don't there is a bug, but how to handle changing from earlier builds of WeeWX to later builds, probably needs further consideration.

[poblabs]

Yeah you're probably right. Perhaps I didn't catch this a few months ago. In my preparation to release 1.1 I noticed it.

I am on 3.9.2, so that's good. Guess I'll need to export my database to a dev machine, drop all dailies, rebuild all dailies, and only migrate back the archive_day_windrun table (this way I don't lose the accurate timestamps for all archive tables such as outTemp, appTemp, etc.)

[poblabs]

@windcrusader Dropping dailies and rebuilding them is returning the same sum result. Very large windrun days. What am I missing?

[windcrusader]

The problem is not with your dailies - it is with the actual archive values in the main WeeWX DB. In versions of WeeWX prior to 3.9.2 these were entered as a cumulative value, which was inconsistent with the general philosophy of WeeWX archive records. You can read more about the underlying problem and reason for the change at weewx/weewx#250.

In summary, to fix your problem, what you need to do is go through every archive record in your DB and recalculate windrun. I note this is a bit of a pain to do, and I note through other reading that I was doing that you were working on adapting the "calcMissing()" tool so that it could be called anytime rather than just when exporting and reimporting database records. Maybe a good time to pull that out again?

[poblabs]

You're right. I spaced out on that one. The dailies are derived from the archive table. So much for trying to multitask!

I think I'll try to drop the windrun column in the archive table. Then re-add it and run a calcMissing() and see what that returns. When that's done, then re-run a rebuild of the dailies.

Yes, I had submitted a PR for calcMissing() but was denied as it's going to be part of a larger issue/PR.

[poblabs]

Spent some time on this today and I'm seeing something that seems off.

Reading how calc_windrun is handled, it seems pretty easy.

data['windrun'] = data['windSpeed'] * data['interval'] / 60.0 if data['windSpeed'] is not None else None

A simple MySQL query can accomplish this:

UPDATE weewx.archive SET windrun = (windSpeed * `interval` / 60.0) where windSpeed > 0;

Now my highest windrun based on sum is 9986.71 miles. An improvement over 42,664.02 miles, but that still seems very high.

[windcrusader]

hmmm, curious. As a check, your windrun in miles should be equal to your daily average windspeed * 24. In my case, on my site, the numbers check out perfectly. I can't see anything wrong with your SQL though. It should work. 9986.7 seems way too high. My highest to date is 655 miles, which happened on a very windy day when the average windspeed was 23 knots!

[poblabs]

It gets even more interesting. Feeling a bit lost on this one as to why my results are so high.

According to this, my highest sum windrun is on 1452038400 (Wednesday, January 6, 2016 12:00:00 AM)

MariaDB [weewx]> SELECT * FROM `archive_day_windrun` ORDER BY `sum` DESC LIMIT 1;
+------------+------+------------+---------------+------------+------------------+-------+------------------+---------+
| dateTime   | min  | mintime    | max           | maxtime    | sum              | count | wsum             | sumtime |
+------------+------+------------+---------------+------------+------------------+-------+------------------+---------+
| 1452038400 |    0 | 1452056460 | 32.6649722222 | 1452038460 | 9986.70872221597 |  1418 | 599202.523332966 |   85080 |
+------------+------+------------+---------------+------------+------------------+-------+------------------+---------+
1 row in set (0.00 sec)

To get this day range, it's 1452038400 to 1452142799 (Date and time (your time zone): Wednesday, January 6, 2016 11:59:59 PM GMT-05:00)

So this query below takes that time range and figures out the average wind speed, and the average windspeed across 24 hours.

MariaDB [weewx]> select avg(windSpeed) as avg_windspeed, (avg(windSpeed)*24) as thisDayWindRun from archive where dateTime >= 1452038400 and dateTime <= 1452142799;
+---------------------+-------------------+
| avg_windspeed       | thisDayWindRun    |
+---------------------+-------------------+
| 0.09232844574780066 | 2.215882697947216 |
+---------------------+-------------------+
1 row in set (0.00 sec)

The result is 2.22 (rounded)

Here's the same day's results in wind speed only

MariaDB [weewx]> SELECT * FROM `archive_day_windSpeed` where dateTime = 1452038400;
+------------+------+------------+------+------------+--------+-------+--------+---------+
| dateTime   | min  | mintime    | max  | maxtime    | sum    | count | wsum   | sumtime |
+------------+------+------------+------+------------+--------+-------+--------+---------+
| 1452038400 |    0 | 1452038460 |  3.5 | 1452098700 | 157.37 |  1419 | 9442.2 |   85140 |
+------------+------+------------+------+------------+--------+-------+--------+---------+
1 row in set (0.00 sec)

So I think what I am taking away from this is that:

1. My average wind speed on this day is 0.09 mph
2. Take 0.09 * 24 you get 2.34 miles

Where's 9986.7 coming from?

[poblabs]

Reached out for help here: https://groups.google.com/forum/#!topic/weewx-development/LawxQIAp-mM

[poblabs]

@windymel @windcrusader FYI based on this comment, I think I'm going to revert windrun sum back to sum and not maxsum.

https://groups.google.com/d/msg/weewx-development/LawxQIAp-mM/CZLPzCR0DAAJ

[windcrusader]

maxsum is the correct way to get the max daily "sum" for a variable. As per what tkeffer said, it is used to get the "rainiest" day, or "windiest" day etc.

Stupid question, but after updating your archive records with your SQL above, did you drop all your daily summaries and rebuild as per: here.

[poblabs]

Yes I rebuilt dailies after running the query. That's why the number changes from 42,000 to 9,000.

So the skin is trying to show Highest Daily Wind Run - surely it's not 22,000 or 9,000 miles. So maxsum doesn't seem like the right value?

[poblabs]

I just posted an update. It may seem that my MySQL query was the problem, and I should have used

UPDATE weewx.archive SET windrun = (windSpeed * `interval` / 60.0) where windSpeed is not null;

[poblabs]

I think we're good here as this was a problem with my own database, and a bad investigative query. Closing again. Thanks for the help!