ET values in wxformulas.py returns rate, rather than total

[tkeffer]

ET is a kind of inverse rain. It is an extensive unit, measured in mm or inches. It is not a rate.

Looking through wxformulas.py, I believe it is returning a rate.

See this thread: https://groups.google.com/forum/#!topic/weewx-user/9N7ktsu5cbc

[matthewwall]

according to this, ET is a rate:

http://edis.ifas.ufl.edu/ae459

which definition should we use?

[tkeffer]

The formula in the Davis documentation on derived variables gives "the hourly potential ET in mm." Following that definition, their download protocol emits "ET accumulated over the last hour." In this sense, it's kind of an anti-rain.

Treating it as an extensive quantity (like rain) also simplifies calculations: you just have to add them up.

Finally, templates (and search list) assume it's an extensive quantity.

So, I'd like our definition to be "the amount of ET that happened over the interval," just like rain.

[tkeffer]

Question: the article on calculating ET by Zotarelli, et al, says the final results will be in mm/day.

However, your docstring for evapotranspiration_Metric() says the results will be in mm/day/hour.

Looking through the Zotarelli paper, mm/day seems correct, although there are some mysterious constants that get in the way (such as 0.408 in Eq. 32), which have unknown units.

Am I missing something?

[matthewwall]

the comment should say mm/period, where period can be day or hour

[matthewwall]

the period is a function of the sr_avg and ws_mps. if those are daily, then result is mm/day. if those are hourly, then result is mm/hour.

[tkeffer]

Oh, I get it. I read the solidus as meaning division, not 'or.'

[tkeffer]

I'm finding the Zotarelli paper very frustrating. Why can't they just give a simple equation, defining the terms, without introducing constants with unknown units?

In any case, the complex procedure they outline is to calculate their equation 3. This equation introduced units that make it applicable only for daily calculations. To get to hourly, you have to divide the constant "900" by 24, in addition to expressing the radiation flux in MJ m^-2 h^-1.

In fairness, the Davis doc does the same thing.

[tkeffer]

Fixed in commit 59de935

[RaphaelChochon]

Hello,
I noticed a strange thing in the values of ET since the last update.
I have a Davis VP2, and Weewx records negative values in full day, concordant with the periods where are recorded the highest rates of solar radiation.
For illustrate this :

The site to consult the data (which is under construction) is available here: http://clans-v2.meteo06.fr

Sorry for my bad English, i'm French,
And thank you for all the work !!!

[tkeffer]

Definitely an issue.

The calculation of ET was changed in the last release to use relative humidity if it is available, which, in your case, it is. Obviously, something is not quite right.

Unfortunately, I am away from home right now, so I can't really investigate. I will take a look at this tomorrow.

In the meantime, I'm opening this issue back up.

[RaphaelChochon]

Thank you, and no worries @tkeffer , this is not super urgent. ;)

But it is certain that the values can not be used for the moment. I just re-control it in database, and I have many negative values recorded.

[tkeffer]

Perhaps you could post some values from your database for the input variables. Something like

SELECT dateTime, outTemp, outHumidity, radiation, windSpeed, ET FROM archive WHERE dateTime > x AND dateTime <= y;

where x and y have been chosen to include a period of negative ET values. At least an hour's worth of data.

Oh, and what is your latitude?

[RaphaelChochon]

Here are some lines with positive and negative values
archive.zip

And my latitude : 43° 59.82' N
It is in the hinterland of Nice in France

[RaphaelChochon]

For easier consult :

[tkeffer]

One other thing, what unit system are you using in the database? It looks like it's metric, but I can't tell if it is METRIC or METRICWX.

[RaphaelChochon]

Yes it's METRIC, I forgot to specify it
This may be the problem anyway

[tkeffer]

It should not be a problem: the calculation of ET takes the unit system into account.

Thanks for the data. I'll take a look at them when I get home.

[RaphaelChochon]

If you need a larger sample of data, do not hesitate !

[tkeffer]

So far, I am unable to reproduce these results. For example, for the archive interval at time 1479127500, I am getting an ET rate of 6.45e-03 mm/day, or a total ET of 2.241e-05 over the 5 minute period, which sounds about right.

Two questions:

1. What are you using for altitude?
2. I noticed that you have a repository that references this issue. Are the plots above created by the data gathered by php scripts in that repository?

[RaphaelChochon]

Ah, surprising... !

1. The station was set to 2185 feet, and we just changed the unit setting in meters to 666 meters. The station is well at 666 meters. Perhaps the fact of being in feet could cause a problem?
2. No, the data included above are data produced by Weewx. This repo is the site that I am developing, and that just happen to connect to this database to display them.

[RaphaelChochon]

This site is accessible at this address : http://clans-v2.meteo06.fr

In my repository, this php scripts retrieves the value from the database and multiplies it by 10 to display it in millimeters and not in centimeters since the rain

[tkeffer]

Hang on. I just realized that you have a VantagePro. By default, these stations generate ET in hardware. Did you change any settings in [StdWXCalculate]?

[RaphaelChochon]

Oh, I did not even know it !
My settings in [StdWXCalculate]

        pressure = prefer_hardware
        barometer = prefer_hardware
        altimeter = prefer_hardware
        windchill = software
        heatindex = software
        dewpoint = prefer_hardware
        inDewpoint = prefer_hardware
        rainRate = prefer_hardware

[RaphaelChochon]

However, before the update 3.6 the data of ET calculated by Weewx was strange, and since the update, apart from the negative values, the sum on the day seems coherent with the present conditions.
This means that it is Weewx who calculates them and not the station?
I am wrong?

[tkeffer]

I'm not sure where the ET values are coming from. According to your settings in [StdWXCalculate], they should be calculated by the VantagePro. However, the VantagePro emits a value only once an hour, while your database is showing ET values in every record.

Could you try running weewx from the command line and capture the REC values? You can use this command to do this:

cd /home/weewx
./bin/weewxd weewx.conf weewx.conf | grep "^REC" | tee /var/tmp/output.txt

Let it run for a couple hours, then post the file output.txt.

-tk

[tkeffer]

A modest refinement to the command I gave you. Unfortunately, the pipe buffers the output, so when you terminate the process after a couple of hours, you'll loose whatever was in the pipe. Try this instead:

cd /home/weewx
./bin/weewxd weewx.conf | grep --line-buffered "^REC" | tee /var/tmp/output.txt

-tk

[RaphaelChochon]

Ok, no problem, I just made this second command, and the file looks to be written correctly. I'll let him spin all night and send it to you tomorrow morning.
Yeah, it is 11 o'clock in France ;)

Thanks for your time !

[RaphaelChochon]

Hello !
I ran the script that night and here is the result: output.zip
By it should not rather that I make it work during the day, precisely when there are negative values?

[tkeffer]

True, but my primary objective is to figure out where these numbers are coming from. They are not coming from the VP2 hardware because it omits values only once an hour, while you have ET values in every record.

You say you have a VP2. Do you have the VP2 Plus, which includes a radiation sensor? Because you have been showing plots of solar radiation, I had assumed you do, but perhaps you do not.

It would be helpful to see records for a couple hours during the day.

[RaphaelChochon]

Ok, that's what it seemed to me when reading the file output.

Yes, it's a VP2 Plus cabled with radiation sensor

Ok, I just restarted the writing of the file output.txt, since the current values are negative.

[tkeffer]

Thanks.

Perhaps you could send the log from when you restarted weewx. It will tell us which values weewx will calculate.

[RaphaelChochon]

When I restart Weewx :

Nov 16 13:55:22 RPI-Clans weewx[1491]: engine: Initializing weewx version 3.6.1
Nov 16 13:55:22 RPI-Clans weewx[1491]: engine: Using Python 2.7.9 (default, Sep 17 2016, 20:26:04) #012[GCC 4.9.2]
Nov 16 13:55:22 RPI-Clans weewx[1491]: engine: Platform Linux-4.4.21-v7+-armv7l-with-debian-8.0
Nov 16 13:55:22 RPI-Clans weewx[1491]: engine: Using configuration file /home/weewx/weewx.conf
Nov 16 13:55:22 RPI-Clans weewx[1491]: engine: Loading station type Vantage (weewx.drivers.vantage)
Nov 16 13:55:23 RPI-Clans weewx[1491]: engine: StdConvert target unit is 0x10
Nov 16 13:55:23 RPI-Clans weewx[1491]: wxcalculate: The following values will be calculated: barometer=prefer_hardware, windchill=software, dewpoint=prefer_hardware, appTemp=prefer_hardware, rainRate=prefer_hardware, windrun=prefer_hardware, heatindex=software, maxSolarRad=prefer_hardware, humidex=prefer_hardware, pressure=prefer_hardware, inDewpoint=prefer_hardware, ET=prefer_hardware, altimeter=prefer_hardware, cloudbase=prefer_hardware
Nov 16 13:55:23 RPI-Clans weewx[1491]: wxcalculate: The following algorithms will be used for calculations: altimeter=aaNOAA, maxSolarRad=RS
Nov 16 13:55:23 RPI-Clans weewx[1491]: engine: Archive will use data binding wx_binding
Nov 16 13:55:23 RPI-Clans weewx[1491]: engine: Record generation will be attempted in 'software'
Nov 16 13:55:23 RPI-Clans weewx[1491]: engine: The archive interval in the configuration file (120) does not match the station hardware interval (300).
Nov 16 13:55:23 RPI-Clans weewx[1491]: engine: Using archive interval of 300 seconds (specified by hardware)
Nov 16 13:55:24 RPI-Clans weewx[1491]: engine: Using binding 'wx_binding' to database 'weewx_rpi_clans'
Nov 16 13:55:24 RPI-Clans weewx[1491]: manager: Starting backfill of daily summaries
Nov 16 13:55:24 RPI-Clans weewx[1491]: manager: Daily summaries up to date
Nov 16 13:55:24 RPI-Clans weewx[1491]: restx: StationRegistry: Station will be registered.
Nov 16 13:55:24 RPI-Clans weewx[1491]: restx: Wunderground-PWS: Data for station IPROVENC95 will be posted
Nov 16 13:55:24 RPI-Clans weewx[1491]: restx: PWSweather: Posting not enabled.
Nov 16 13:55:24 RPI-Clans weewx[1491]: restx: CWOP: Posting not enabled.
Nov 16 13:55:24 RPI-Clans weewx[1491]: restx: WOW: Posting not enabled.
Nov 16 13:55:24 RPI-Clans weewx[1491]: restx: AWEKAS: Posting not enabled.
Nov 16 13:55:24 RPI-Clans weewx[1491]: engine: Starting up weewx version 3.6.1
Nov 16 13:55:24 RPI-Clans weewx[1491]: engine: Clock error is -0.39 seconds (positive is fast)
Nov 16 13:55:24 RPI-Clans weewx[1491]: manager: added record 2016-11-16 13:40:00 CET (1479300000) to database 'weewx_rpi_clans'
Nov 16 13:55:29 RPI-Clans weewx[1491]: manager: added record 2016-11-16 13:40:00 CET (1479300000) to daily summary in 'weewx_rpi_clans'
Nov 16 13:55:29 RPI-Clans weewx[1491]: manager: added record 2016-11-16 13:45:00 CET (1479300300) to database 'weewx_rpi_clans'
Nov 16 13:55:33 RPI-Clans weewx[1491]: manager: added record 2016-11-16 13:45:00 CET (1479300300) to daily summary in 'weewx_rpi_clans'

If I understand correctly, we have the parameters in [StdWXCalculate]

[RaphaelChochon]

Ah !

The last record :

REC:    2016-11-16 14:00:00 CET (1479301200) altimeter: 1019.3254913, appTemp: 11.6050728283, barometer: 1021.79886437, cloudbase: 1442.10239821, consBatteryVoltage: 4.77, dateTime: 1479301200.0, dayET: 0.013, dayRain: 0.0, dewpoint: 6.31498129621, ET: -0.000404949779421, extraAlarm1: 0.0, extraAlarm2: 0.0, extraAlarm3: 0.0, extraAlarm4: 0.0, extraAlarm5: 0.0, extraAlarm6: 0.0, extraAlarm7: 0.0, extraAlarm8: 0.0, extraHumid1: None, extraHumid2: None, extraHumid3: None, extraHumid4: None, extraHumid5: None, extraHumid6: None, extraHumid7: None, extraTemp1: None, extraTemp2: None, extraTemp3: None, extraTemp4: None, extraTemp5: None, extraTemp6: None, extraTemp7: None, forecastIcon: 6.0, forecastRule: 63.0, heatindex: 12.5393162393, humidex: 12.5393162393, inDewpoint: 51.8876327818, inHumidity: 45.0, insideAlarm: 0.0, inTemp: 23.6965811966, interval: 5, leafTemp1: None, leafTemp2: None, leafTemp3: None, leafTemp4: None, leafWet1: None, leafWet2: None, leafWet3: None, leafWet4: 0.0, maxSolarRad: 322.746491023, monthET: 0.58, monthRain: 5.207, outHumidity: 65.8076923077, outsideAlarm1: 0.0, outsideAlarm2: 0.0, outTemp: 12.5393162393, pressure: 941.725811864, radiation: 382.538461538, rain: 0.0, rainAlarm: 0.0, rainRate: 0.0, soilLeafAlarm1: 0.0, soilLeafAlarm2: 0.0, soilLeafAlarm3: 0.0, soilLeafAlarm4: 0.0, soilMoist1: None, soilMoist2: None, soilMoist3: None, soilMoist4: None, soilTemp1: None, soilTemp2: None, soilTemp3: None, soilTemp4: None, stormRain: 0.0, stormStart: None, sunrise: 1479275100.0, sunset: 1479318000.0, txBatteryStatus: 0, usUnits: 16, UV: 0.6, windchill: 12.5393162393, windDir: 254.0, windGust: 1.60935200003, windGustDir: 254.0, windrun: 1.45608300005, windSpeed: 0.433287076931, windSpeed10: 1.23796307695, yearET: 29.05, yearRain: 66.6496

ET: -0.000404949779421

So it's the console that sends a negative value?

[RaphaelChochon]

Indeed it goes back quite a lot of negative values...
output.zip

[tkeffer]

It would appear so.

But, let's do one final check. Please substitute your copy of wxservices.py with the attached. It is instrumented to print out intermediate values. Substitute, restart weewx, run for 20 minutes or so, then post the log.

wxservices.zip

[RaphaelChochon]

Okay, I'll do that when I come back tonight and keep you informed.
By cons I will also compare the data in output.txt and what fits into database.
Thank again Tom !

[RaphaelChochon]

Hello!
So, in fact, the values in yesterday's output.txt are identical to those stored in the database.

I substituted the file wxservices.py ant this is the result : output&syslog.zip

In passing, I also noticed that the skin Standard displayed this with a negative value, but rounded

[tkeffer]

I think we have two issues:

1. For whatever reason, your Vantage is not emitting ET. We are going to have to instrument your Vantage driver to figure out why.

2. The calculation of ET has a few errors. Most are minor, but one is not. The formula needs to estimate cooling from outgoing long-wave radiation, but that depends on cloud cover. How to measure that? The formula uses the ratio between the measured incoming radiation and the theoretical solar radiation as a proxy. If there is lots of incoming radiation relative to the expected radiation, the sun must be shining, so there will be lots of outgoing long-wave radiation at night.

   Unfortunately, we are comparing hourly averages of measured incoming radiation to daily averages of theoretical clear sky incoming radiation. Measured radiation should never be higher than theoretical radiation, but because of the differences in averaging periods, it is during the middle of the day. Hence, the formula over-estimates the outgoing (long-wave) radiation and thinks lots of cooling is happening during the middle of the day. This is why you're perversely seeing negative ET (dew) when the sun is shining.

I'll try and come up with some fixes later in the day. In the meantime, if you're facile with Python, you might try watching the driver, vantage.py, and see why it is not emitting ET. Otherwise, wait a bit and I'll send you an instrumented version.

[RaphaelChochon]

Okay, I read some papers yesterday, explaining how to calculate potential ET (ETP).
If I understand correctly, it can be calculated only on a day, and not really between each interval of records, since we need Tn and Tx and others parameters.
This joins what you say, and explaining why there are negative values recorded. It is to "catch up" the difference between the expected ET and the real ET.

• So, is it really possible to calculate the ET between each interval? And is it useful?
• The sum of ET at the end of the day is therefore ok?

I do not know if I am clear in my explanations, my English being approximative (thanks google translate)

I'm not familiar with the Python language, but I'll try to look at it.

[tkeffer]

Raphael,

Please substitute your version of vantage.py with the attached. It will log the raw and unpacked values of ET.

vantage.zip

Substitute, restart weewx, let it run for a couple hours, post the log.

I'm still working on a better version for calculating ET in software.

[RaphaelChochon]

Hello Tom,
I just replaced it and restarted Weewx with the command

./bin/weewxd weewx.conf | grep --line-buffered "^REC" | tee /var/tmp/output.txt

Is it necessary? Or a simple restart of Weewx is enough for you?

Thanks !

[tkeffer]

A simple restart will do.

-tk

Fat-fingered from my Android

On Nov 18, 2016 12:43 PM, "Raph" notifications@github.com wrote:

Hello Tom,
I just replaced it and restarted Weewx with the command

./bin/weewxd weewx.conf | grep --line-buffered "^REC" | tee /var/tmp/output.txt

Is it necessary? Or a simple restart of Weewx is enough for you?

Thanks !

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#160 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADKtq_TZtX7KZw9EM1hpt-bxnjCwE-lOks5q_g3vgaJpZM4KFoxP
.

[RaphaelChochon]

Hello Tom,

I substituted the file vantage.py ant this is the result after many hours : syslog.zip

Apparently no problem !

[tkeffer]

I didn't realize that you were using software record generation. That would explain problem 1. Any particular reason why? Why not use hardware record generation?

As for problem 2, I'm still working on a better version for calculating ET. But, if you use hardware generation, you should not need it.

[RaphaelChochon]

I do not understand, the [StdWXCalculate] section indicate :

        pressure = prefer_hardware
        barometer = prefer_hardware
        altimeter = prefer_hardware
        windchill = software
        heatindex = software
        dewpoint = prefer_hardware
        inDewpoint = prefer_hardware
        rainRate = prefer_hardware

I would have to add :

       ET = hardware

http://www.weewx.com/docs/usersguide.htm#StdWXCalculate

[tkeffer]

Take a look at option record_generation in section [StdArchive]. For hardware generation, it should be set to hardware.

record_generation = hardware

You are also using an unusual database binding. Any reason for that?

[RaphaelChochon]

Oh okay, this option had eluded me !
I just put "hardware".

Yes I am using an external MySQL database on my server.
That causes a problem ?

[RaphaelChochon]

So now I have that in the logs:

Nov 19 14:55:10 RPI-Clans weewx[25958]: vantage: ET raw values=0x00; unpacked value=0.000
Nov 19 14:55:10 RPI-Clans weewx[25958]: manager: added record 2016-11-19 14:55:00 CET (1479563700) to database 'weewx_rpi_clans'
Nov 19 14:55:15 RPI-Clans weewx[25958]: manager: added record 2016-11-19 14:55:00 CET (1479563700) to daily summary in 'weewx_rpi_clans'

And actually I have a value of 0 recorded in database

[tkeffer]

The standard weewx.conf ships with the option record_generation=hardware. You must have changed it.

Using MySQL is not a problem. It's just that you used an unusual name weewx_rpi_clans, instead of the standard weewx.

The vantage only emits an ET value on the hour. For all other records, it emits a zero. The record at 15:00:00 should have an ET value.

[RaphaelChochon]

Yes I had changed this for two other stations that are Oregon WMR200 (http://nice.meteo06.fr and http://gilette-v2.meteo06.fr). Without this setting, weewx could not produce reports.

Yes I changed the name of the database since I have several stations.

I now understand, and it makes the link on what I read about the calculation of Davis (hourly and not at each interval).
At 15:00:00 and at 16:00:00 (now) I had a recording.

Thank you Tom for clarification !
If you need a guinea pig (guinea pig, really !? The term is amazing in English) to improve the calculation of ET, do not hesitate!

[tkeffer]

This thing kicked my butt. There are very few good references on ET calculations: they all seem to hide the units used and make mysterious assumptions about cloud cover, etc. I was able to come up with something that reproduces the test values from Example 19 in the FAO crop reference.

I also tested it against the Davis algorithm, and it gives reasonably close numbers in summer, slightly low for winter.

The algorithm now calculates ET on an hourly basis. It does not allow negative ET values.

Call it fixed in commit 21f55b4