XTypes performance
See the companion document XTypes for a general description of XTypes.
Version 5 introduces the ability to do aggregates of XTypes. That is, one can use a tag such as
$month.myxtype.min
It does this by repeatedly calling get_scalar() over the aggregation period.
In this example, this means calling get_scalar() for every record in the
month. This can be slow.
Here's an example, that builds on the Vapor pressure example. To recap, here's the extension:
# File user/vaporpressure.py
import math
import weewx
import weewx.units
import weewx.xtypes
from weewx.engine import StdService
from weewx.units import ValueTuple
# Add vapor_p to group_pressure
weewx.units.obs_group_dict['vapor_p'] = 'group_pressure'
class VaporPressure(weewx.xtypes.XType):
def __init__(self, algorithm='simple'):
# Save the algorithm to be used.
self.algorithm = algorithm.lower()
def get_scalar(self, obs_type, record, db_manager):
# We only know how to calculate 'vapor_p'. For everything else, raise an exception UnknownType
if obs_type != 'vapor_p':
raise weewx.UnknownType(obs_type)
# We need outTemp in order to do the calculation.
if 'outTemp' not in record or record['outTemp'] is None:
raise weewx.CannotCalculate(obs_type)
# We have everything we need. Start by forming a ValueTuple for the outside temperature.
outTemp_vt = weewx.units.as_value_tuple(record, 'outTemp')
# Both algorithms need temperature in Celsius, so let's make sure our incoming temperature
# is in that unit. Use function convert(). The results will be in the form of a ValueTuple
outTemp_C_vt = weewx.units.convert(outTemp_vt, 'degree_C')
# The attribute ".value" will give us just the value part of the ValueTuple
result = calc_vapor_pressure(outTemp_C_vt.value, self.algorithm)
# Convert to the unit system that we are using and return
return weewx.units.convertStd(result, record['usUnits'])
def calc_vapor_pressure(outTemp_C, algorithm='simple'):
"""Given a temperature in Celsius, calculate the vapor pressure"""
if outTemp_C is None:
return ValueTuple(None, 'mmHg', 'group_pressure')
if algorithm == 'simple':
# Use the "Simple" algorithm.
# We need temperature in Kelvin.
outTemp_K = weewx.units.CtoK(outTemp_C)
# Now we can use the formula. Results will be in mmHg. Create a ValueTuple out of it:
p_vt = ValueTuple(math.exp(20.386 - 5132.0 / outTemp_K), 'mmHg', 'group_pressure')
elif algorithm == 'teters':
# Use Teter's algorithm.
# Use the formula. Results will be in kPa:
p_kPa = 0.61078 * math.exp(17.27 * outTemp_C / (outTemp_C + 237.3))
# Form a ValueTuple
p_vt = ValueTuple(p_kPa, 'kPa', 'group_pressure')
else:
# Don't recognize the algorithm. Fail hard:
raise ValueError(algorithm)
return p_vtNote how get_scalar() uses outTemp from the parameter record to calculate
vapor_p, the vapor pressure.
Let's see how this affects the performance of a simplified version of the Seasons skin. Here's the benchmarking environment:
-
364 MB database
-
NUC 11th gen I7 with 32 GB RAM
-
No NOAA files
-
Generate only
index.htmlandstatistics.html -
Specify only the vapor pressure
vapor_pin the section[DisplayOptions]:[DisplayOptions] # This list determines which types will appear in the "current conditions" # section, as well as in which order. observations_current = vapor_p, # This list determines which types will appear in the "statistics" and # "statistical summary" sections, as well as in which order. observations_stats = vapor_p,
This means that the statistical summaries will only include
vapor_p, and not the usual assortment of types offered by the stock Seasons skin. This allows us to focus on the performance of just the XType.
In this case, we just time how long it takes to generate the two files of the simplified Seasons skin. With v5.0, the benchmark gives 5.97s, which is pretty slow considering how simple the skin is.
The reason is that, by default, the skin will calculate tags such as
$year.vapor_p.min, $year.vapor_p.mintime, etc. Because vapor_p does not
appear in the database, every record for the year must be read and passed on to
the version of get_scalar() in our extension. This must be done for all
aggregation periods, including $day, $week, etc. That's a total of 344,428
database reads and calls!
The simplest optimization is to just put the new type, vapor_p in the
database. This will require adding a new column to the database, then
retro-calculating its values.
First step is to tell WeeWX about the new value. Modify the [StdWXCalculate]
section so it reads:
[StdWXCalculate]
[[Calculations]]
...
vapor_p = prefer_hardware
...Now add the new column, and have its values calculated. This can be a lengthy process (about 10 minutes on my machine), but it's a one-time cost.
weectl database add-column vapor_p
weectl database calc-missingWhen you're done, instead of having to calculate vapor_p for every record, the
tag system can just look it up in the database.
With this optimization, our benchmark takes just 0.11 seconds to run.
One of the reasons the unoptimized version is so slow is because of the
necessity to look up every single record in the aggregation period. We can speed
this up a bit by offering a specialized version of get_aggregate(). With this
version, we do a single database query to get all the values of outTemp we
need to calculate vapor_p. Here's what it looks like:
def get_aggregate(self, obs_type, timespan, aggregate_type, db_manager, **option_dict):
"""Calculate an aggregate value of vapor pressure over a given timespan."""
if obs_type != 'vapor_p':
raise weewx.UnknownType(obs_type)
if aggregate_type not in {'max', 'min', 'mintime', 'maxtime', 'avg'}:
raise weewx.UnknownAggregation(aggregate_type)
# Get outTemp for the timespan. The results will be returned as 3 ValueTuples
starts, stops, values = weewx.xtypes.get_series('outTemp', timespan, aggregate_type=None,
db_manager=db_manager, **option_dict)
# Convert the values to celsius:
values_C = weewx.units.convert(values, 'degree_C')
# Now calculate the aggregate property for vapor pressure
total = 0.0
total_time = 0
minimum = None
maximum = None
mintime = None
maxtime = None
for start, stop, outTemp_C in zip(starts.value, stops.value, values_C.value):
# Calculate the vapor pressure for this record
vapor_vt = calc_vapor_pressure(outTemp_C, self.algorithm)
# Convert to the unit system used by the database and unpack:
vapor_pressure, u, g = weewx.units.convertStd(vapor_vt, db_manager.std_unit_system)
# If it's not None, compile it into the statistics:
if vapor_pressure is not None:
time_interval = stop - start
total += vapor_pressure * time_interval
total_time += time_interval
if minimum is None or vapor_pressure < minimum:
minimum = vapor_pressure
mintime = stop
if maximum is None or vapor_pressure > maximum:
maximum = vapor_pressure
maxtime = stop
if aggregate_type == 'avg':
result = total / total_time if total_time else None
elif aggregate_type == 'mintime':
result = mintime
elif aggregate_type == 'maxtime':
result = maxtime
elif aggregate_type == 'min':
result = minimum
elif aggregate_type == 'max':
result = maximum
else:
# We should never get here.
raise ValueError(f"Unexpected aggregation type {aggregate_type}")
# Convert to the unit system that we are using and return
u, g = weewx.units.getStandardUnitType(db_manager.std_unit_system, 'vapor_p', aggregate_type)
return weewx.units.convertStd(ValueTuple(result, u, g), db_manager.std_unit_system)We must also make a change in class VaporPressureService. We need to prepend
our new XType to the list of XTypes. This is so our version of get_aggregate()
is seen before the default version provided by the XTypes system:
class VaporPressureService(StdService):
def __init__(self, engine, config_dict):
super(VaporPressureService, self).__init__(engine, config_dict)
# Get the desired algorithm. Default to "simple".
try:
algorithm = config_dict['VaporPressure']['algorithm']
except KeyError:
algorithm = 'simple'
# Instantiate an instance of VaporPressure:
self.vp = VaporPressure(algorithm)
# Register it with the XTypes system. Put it at the beginning of types
# so that it is seen first:
weewx.xtypes.xtypes.insert(0, self.vp)With this change, we see a modest speed improvement. The skin generation takes
1.35 s. Our function get_aggregate() is called 40 times.
Providing a specialized version of get_aggregate as shown in the previous
example is general, but it's not particularly fast. The reason is that while we
only had to do 40 database queries (instead of 344,428), those queries still had
to provide outTemp from every record. We could not take advantage of the daily
summaries.
Here's the reason why. Suppose V() represents the vapor pressure function.
To calculate vapor pressure for a given temperature, we use
vp = V(T)
If we want an average value over a time period, we could do
avg_vp = avg(V(T))
That means, calculate V(T) for every record, then average the results.
What we would like to do is
avg_vp = V(avg(T))
because then we could use the daily summaries to calculate avg(T) very
quickly. Unfortunately, that only works if V() is linear, which it definitely
is not — there is an exponential function in the formula.
However, all is not lost. If we are only interested in the minimum and maximum values (and not averages, which is the case for the Seasons skin), then we can use the daily summaries. This is because the vapor pressure formula tells us that there is a monotonic relationship between temperature and pressure: the lowest temperature will result in the lowest pressure, and the highest temperature gives the highest pressure. So, we need only find the minimum and maximum temperature, then calculate the vapor pressure for them.
Here's what get_aggregate() looks like with this optimization:
def get_aggregate(self, obs_type, timespan, aggregate_type, db_manager, **option_dict):
if obs_type != 'vapor_p':
raise weewx.UnknownType(obs_type)
if aggregate_type not in {'min', 'max', 'mintime', 'maxtime'}:
raise weewx.UnknownAggregateType(aggregate_type)
result_vt = weewx.xtypes.get_aggregate('outTemp', timespan, aggregate_type, db_manager)
if aggregate_type in {'mintime', 'maxtime'}:
# Min and max times can be returned as-is.
return result_vt
# A vapor pressure is needed. First convert the temperature to celsius:
outTemp_C = weewx.units.convert(result_vt, 'degree_C')[0]
# Calculate the vapor pressure. Result will be a ValueTuple
vapor_vt = calc_vapor_pressure(outTemp_C, self.algorithm)
# Convert to the unit system that we are using and return
val = weewx.units.convertStd(vapor_vt, db_manager.std_unit_system)
return valWith this change, our version of get_aggregate gets called 40 times, and the
benchmark runs in 0.12 seconds: just as fast as if we had put vapor_p in the
database.
Of course, this solution is not general. You must take a close look at the algorithm used by your XType and decide how you can take advantage of it in order to optimize the run time.
The self-provisioning system of the Seasons skin causes tags to be generated
for everything in observations_current and observation_stats. That's about
40 tags for each type, some of them quite expensive. You might not need that
many results. For example, you may not care about vapor pressure statistics for
each aggregation time period.
Modify skin.conf to read
[DisplayOptions]
# This list determines which types will appear in the "current conditions"
# section, as well as in which order.
observations_current = vapor_p,
# This list determines which types will appear in the "statistics" and
# "statistical summary" sections, as well as in which order.
observations_stats = ,With this change, only the tag $current.vapor_p will be generated. Generation
time drops to 0.11 seconds.
There have been comments that "V5 is so much slower than V4.10!". That's true
only because V5 is doing more: V4.10 did not have the ability calculate
aggregates of XTypes. Here's what the results of the file statistics.html
looks like for V5:
and here's what it looks like for V4.10:
Version 4.10 is faster only because it was unable to calculate any of these values!
Not surprisingly, the time it takes is the same as V5 with no observation statistics: 0.11 seconds.
The following table summarizes the strategies:
| Version | Optimization | Time (2 files) |
|---|---|---|
| 5.0.3 | None | 5.97 s |
| 5.0.3 | In database | 0.11 s |
| 5.0.3 | Specialized get_aggregate
|
1.35 s |
| 5.0.3 | Optimized get_aggregate
|
0.11 s |
| 5.0.3 | No observation stats | 0.11 s |
| 4.10.2 | None | 0.11 s |
