New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
weewx.xtypes.XTypeTable.get_aggregate()
is very very slow
#939
Comments
Thanks, Karen, for this report. It isolates the issue nicely. I will be on the road for the next 5 days, so I will be unable to take a look at this until the end of the week. |
I'm seeing only small increases in generation time. Couple more questions:
|
I started this investigation only because people were complaining to me about the weewx-GTS extension becoming very slow under WeeWX 5.0 causing their report generation to crash. Under WeeWX 4.10 all works fine. To help those people I finally ended up changing the extension. The code now inserts the extension before XTypeTable and handles all the aggregations that XTypeTable handles. So the extension works under 5.0 as well as it worked under 4.10 The weewx-GTS extension provides some derived observation types used in agriculture. Additionally you may want to read Rich's thoughts about XTypeTable. |
I am using a 364 MB database on a NUC 11th gen I7 and 32 GB RAM, so similar processing power, similar database size. I tried to isolate things to just the
I ran the CheetahGenerator only (no image generation). No NOAA reports.
There is no question that V5.0 is much slower, BUT it is doing more. For v4.10.2, the For v5.0.2 (HEAD), there are complete results. Probably the right thing to do is write a wiki page explaining this. If faced with a performance problem, there are several choices:
That's all I can come up with. |
Just tried the test again, this time adding observation type
|
And in reality people do not only have one such observation type but many. And every one adds some time. And in the end people experienced skin creation durations of several minutes, not seconds. The problem is, that the As you say people need more information:
And, may be, some time we find a faster algorithm. With Python sometimes it is like in the early days of computing, when people thought about optimizing some small snippet of assembler code for days. What I made different in my implementation of |
Here are my thoughts and comments on XType performance: https://github.com/weewx/weewx/wiki/XTypes-performance |
Thank you. I added a link to your wiki article to the README of my XTypes. |
Test environment:
test
that has a constant valueTest:
Before installing the test XType, creating the HTML files of the Seasons skins requires 0.43 seconds in the test environment:
I then added
user.test.TestService
toxtype_services
inweewx.conf
and the observation typetest
toobservations_current
andobservations_stats
inskin.conf
. See below for details of the configuration changes.After that creating the HTML files of the Seasons skin requires 14.50 seconds:
That are 3372 percent longer than before.
Why did I perform this test?
With WeeWX 5.0 the weewx-GTS extension consumes a lot of time, forcing WeeWX to restart main loop continuously. With WeeWX 4.10 all works fine. I looked for possible reasons and found out, that there is a new function
weewx.xtypes.XTypeTable.get_aggregate
, that is not available in older versions of WeeWX. Inserting some logging messages revealed that it is indeed this function, that consumes the time.Changes to the configuration:
weewx.conf
:Added
user.test.TestService
toxtype_services
only. This is the whole section:skin.conf
:Added
test
to the end of the list:The text was updated successfully, but these errors were encountered: