Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] FileNotFoundError, wdc tries to pull files from the wrong skin dir? #267

Open
silke-99 opened this issue Apr 9, 2024 · 17 comments
Open
Labels
bug Something isn't working

Comments

@silke-99
Copy link

silke-99 commented Apr 9, 2024

Hi,

I just newly installed your skin on top of a weewx 5.0.2 installation that already runs Belchertown and some other skins. I did not change much yet in the config, apart from changing base_path and HTML_dir and trying to activate forecasting.

Overall I managed to create content, but some stuff (e.g. statistics page) I can't get to generate.

I see these errors below in the log. They are coming from the following 4 templates

statistics.html.tmpl,
celestial.html.tmpl
offline.html.tmpl
about.html.tmpl

and they are all trying to get includes from the Belchertown skin dir! Why is that? Why is it not looking in its own skin dir instead?

Thanks a lot, otherwise really a great looking skin!

weewxd[2173619]: ERROR weewx.cheetahgenerator: Evaluation of template /etc/weewx/skins/weewx-wdc/statistics.html.tmpl failed with exception '<class 'FileNotFoundError'>'
weewxd[2173619]: ERROR weewx.cheetahgenerator: **** Ignoring template /etc/weewx/skins/weewx-wdc/statistics.html.tmpl
weewxd[2173619]: ERROR weewx.cheetahgenerator: **** Reason: [Errno 2] No such file or directory: '/etc/weewx/skins/Belchertown/includes/combined-diagram-tile.inc'
weewxd[2173619]: ERROR weewx.cheetahgenerator: ****  Traceback (most recent call last):
weewxd[2173619]: ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 334, in generate
weewxd[2173619]: ERROR weewx.cheetahgenerator: ****      unicode_string = compiled_template.respond()
weewxd[2173619]: ERROR weewx.cheetahgenerator: ****                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
weewxd[2173619]: ERROR weewx.cheetahgenerator: ****    File "_etc_weewx_skins_weewx_wdc_statistics_html_tmpl.py", line 250, in respond
weewxd[2173619]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 1708, in _handleCheetahInclude
weewxd[2173619]: ERROR weewx.cheetahgenerator: ****      self._CHEETAH__cheetahIncludes[_includeID].respond(trans)
weewxd[2173619]: ERROR weewx.cheetahgenerator: ****    File "_etc_weewx_skins_weewx_wdc_includes_climatological_days_inc.py", line 833, in respond
weewxd[2173619]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 1685, in _handleCheetahInclude
weewxd[2173619]: ERROR weewx.cheetahgenerator: ****      nestedTemplateClass = compiler.compile(source=source,
weewxd[2173619]: ERROR weewx.cheetahgenerator: ****                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
weewxd[2173619]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 738, in compile
weewxd[2173619]: ERROR weewx.cheetahgenerator: ****      fileHash += str(os.path.getmtime(file))
weewxd[2173619]: ERROR weewx.cheetahgenerator: ****                      ^^^^^^^^^^^^^^^^^^^^^^
weewxd[2173619]: ERROR weewx.cheetahgenerator: ****    File "<frozen genericpath>", line 55, in getmtime
weewxd[2173619]: ERROR weewx.cheetahgenerator: ****  FileNotFoundError: [Errno 2] No such file or directory: '/etc/weewx/skins/Belchertown/includes/combined-diagram-tile.inc'
@silke-99 silke-99 added the bug Something isn't working label Apr 9, 2024
@Daveiano
Copy link
Owner

Daveiano commented Apr 9, 2024

Hello and thank you for reaching out!

Very strange behaviour, I have never encountered this before. What changes did you make to base_path and HTML_dir? Could you please post the [WdcReport] section from weewx.conf?

One thing you could do is reset all your changes, does the error also occur with default configs?

@silke-99
Copy link
Author

silke-99 commented Apr 10, 2024

Hi, so this is what I did now:

  • uninstalled weewx-wdc and cleaned up output dir
  • installed it again
  • did the following two changes only:
    weewx.conf:HTML_ROOT = /var/www/weewx/wdc
    skin.conf: base_path = /weewx/wdc/
  • restarted weewx
  • errors appear upon generation as indicated above,

I rely on changing these two things to not get in trouble with my other skin outputs. The other skins (Belchertown, Bootstrap, neowx) are installed in their normal places and output to /var/www/weewx/Belchertown etc.

This is my entire WdcReport config:

    [[WdcReport]]
        skin = weewx-wdc
        enable = true
        HTML_ROOT = /var/www/weewx/wdc

I should also say that I have >10 years of data, so initial output generation takes a looong time.

And after cleaning up everything and starting from scratch, now also I don't see any graphs being created.

(EDIT: the missing graphs topic was solved, I noticed that the dist directory was not created in the output dir. I just did that manually.)

@Daveiano
Copy link
Owner

In your first comment your wrote 4 templates causing this error:

statistics.html.tmpl,
celestial.html.tmpl
offline.html.tmpl
about.html.tmpl

Is it the same error for all four templates? The combined-diagram-tile.inc isn't even used on the about page.

Can you verify that the file skins/weewx-wdc/includes/combined-diagram-tile.inc exists (see SKIN_ROOT, https://www.weewx.com/docs/5.0/usersguide/where/)?

@silke-99
Copy link
Author

silke-99 commented Apr 11, 2024

Hi,

yes, skins/weewx-wdc/includes/combined-diagram-tile.inc exists:

-rw-r--r-- 1 root weewx 6769 Apr 10 12:41 /etc/weewx/skins/weewx-wdc/includes/combined-diagram-tile.inc

And it gets weirder: After some hours and a few restarts and essentially no relevant changes (just tried to reorder skin related lines in weewx.conf and removed a few commented lines) wdc suddenly started to create the missing files.

However, the errors are still there, just not anymore with every wdc run as before. The output files get also updated as they should. And error also appear for other templates now, e.g. year-%Y.html.tmpl

What is even more funny - I now see these skin crossfire errors also for other skin templates directing towards wdc or e.g. Bootstrap. As you can see they are more or less all the same, wildly trying to find files in the wrong skin's directory. I checked, and before the installation of wdc I have no such log entries at all in my syslog.

I'm attaching a longer log snippet with all the log entry types I see (hopefully better to read). By now, as finally all the output files are there, it has turned more into a cosmetic "weird errors" issue, but still really strange.

weewx-syslog.txt

Thank you very much for looking into this! :-)

@Daveiano
Copy link
Owner

Could this be a permission problem? Under which user is weewx running? Does it make a difference if you run the weectl report run WdcReport as root?

I am asking because combined-diagram-tile.inc is owned by the root user:

-rw-r--r-- 1 root weewx 6769 Apr 10 12:41 /etc/weewx/skins/weewx-wdc/includes/combined-diagram-tile.inc

@silke-99
Copy link
Author

well, I had a similar suspicion - weewx runs as user weewx since the v5 upgrade. I chowned the wdc skin dir to weewx now, as all the other skins already were, but this hasn't had any impact. I also noticed that the Belchertown skin was missing enable = true in weewx conf, but adding that also had no impact.

The situation now is as follows:

  • for most of the time, all skins operate normally
  • quite exactly every 12 hours, I start seeing those error messages within a timeframe of 15-60 mins approx.
  • during the time this is happening, I also have experienced visual glitches in the output files, so far mostly in Belchertown (which is my main public facing skin). It seems that the wrong formatting is applied, top menu in different color and font, wind rose missing etc.
  • this will fix itself when skins are back to normal. No more errors until the next cycle.

It looks to me as if twice a day the cheetahgenerator decides to go a little wild and tries to mix things up a little bit. :-)

@Daveiano
Copy link
Owner

quite exactly every 12 hours, I start seeing those error messages within a timeframe of 15-60 mins approx.

This could be the statistics.html page - it gets generated every 12 hours per default config, see: https://github.com/Daveiano/weewx-wdc/blob/3.x/skins/weewx-wdc/skin.conf#L647

Could you try to comment out the [[[statistics]]] section of the skin.conf to disable the generation of it? The statistics (all time) page shows graphs and stats of the whole data available to the station. Generation tooks some time, can you visit the page now and see all the data?

@silke-99
Copy link
Author

I have re-enabled wdc skin and disabled statistics yesterday, and so far no issues/errors. So yes, it looks like the statistics page is the culprit here. (For the record, also no issues when wdc is entirely disabled)

As far as it got generated, the statistics page looks okay to me (no glitches) and also the data make sense. As mentioned before, I have >11 years worth of data in weewx DB, so I assume the processing will be a bit stressful....

Will play with it some more. Thanks a lot! 👍

@Daveiano
Copy link
Owner

Good news, but would be great if all pages could be generated.

I guess it takes too long and generation of the other reports starts in the background and things start to mess up...

WeeWX supports the scheduling of reports via the report_timing option - perhaps this could be an option?
https://weewx.com/docs/5.0/custom/report-scheduling/#the-stale_age-option

stale_age of the stats page is 12 hours, the idea would be to guarantee that no other report runs when wdc generates the stats page.

Another question: On which hardware are you running WeeWX?

@silke-99
Copy link
Author

Hello, sorry for the late response and thanks for your hints.

I have now scheduled all my reports in a way to allow for an 1 hour window at night where only wdc reports are is running and that matches the stale timer of the statistics file (24hours).

This has removed the crossfire into other skins, predictably, the only error messages I see are related to the other wdc templates. See attached, these are all the errors I saw last night.

The timestamp of the output file statistics.html is also showing 02:44 (same as when the errors are happening), but the timestamp shown inside ($alltime.end) is 02:31:00.

I'm running weewx with MySQL on an Intel Xeon E-2276G server (6-Core, 16GB RAM). It also does some other things, but I wouldn't call it resource challenged...

weewx-syslog-1505.txt

@mymaestro
Copy link

Very similar situation here. It loosely coincides with the other problem I posted about statistics.html failing due to database locking issues.
I also have data from 2011 forwards.

@mymaestro
Copy link

Hope there's something here that helps:

Jul 11 07:35:19 cyclone weewxd[3918]: INFO weewx.manager: Added record 2024-07-11 07:35:00 CDT (1720701300) to daily summary in 'weewx.sdb'
Jul 11 07:35:20 cyclone weewxd[3918]: INFO weewx.restx: Wunderground-PWS: Published record 2024-07-11 07:35:00 CDT (1720701300)
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: Evaluation of template /etc/weewx/skins/weewx-wdc/statistics.html.tmpl failed with exception '<class 'FileNotFoundError'>
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: **** Ignoring template /etc/weewx/skins/weewx-wdc/statistics.html.tmpl
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: **** Reason: [Errno 2] No such file or directory: '/etc/weewx/skins/forecast/includes/stat-tile-modals.inc'
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****  Traceback (most recent call last):
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 334, in generate
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      unicode_string = compiled_template.respond()
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "_etc_weewx_skins_weewx_wdc_statistics_html_tmpl.py", line 268, in respond
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 1685, in _handleCheetahInclude
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      nestedTemplateClass = compiler.compile(source=source,
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 738, in compile
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      fileHash += str(os.path.getmtime(file))
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                      ^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "<frozen genericpath>", line 55, in getmtime
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****  FileNotFoundError: [Errno 2] No such file or directory: '/etc/weewx/skins/forecast/includes/stat-tile-modals.inc'
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: Evaluation of template /etc/weewx/skins/weewx-wdc/celestial.html.tmpl failed with exception '<class 'FileNotFoundError'>'
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: **** Ignoring template /etc/weewx/skins/weewx-wdc/celestial.html.tmpl
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: **** Reason: [Errno 2] No such file or directory: '/etc/weewx/skins/forecast/includes/html-head.inc'
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****  Traceback (most recent call last):
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 334, in generate
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      unicode_string = compiled_template.respond()
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "_etc_weewx_skins_weewx_wdc_celestial_html_tmpl.py", line 144, in respond
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 1685, in _handleCheetahInclude
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      nestedTemplateClass = compiler.compile(source=source,
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/CheetJul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      fileHash += str(os.path.getmtime(file))
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                      ^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "<frozen genericpath>", line 55, in getmtime
Jul 11 07:36:12 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****  FileNotFoundError: [Errno 2] No such file or directory: '/etc/weewx/skins/forecast/includes/html-head.inc'
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: Evaluation of template /etc/weewx/skins/weewx-wdc/offline.html.tmpl failed with exception '<class 'FileNotFoundError'>'
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: **** Ignoring template /etc/weewx/skins/weewx-wdc/offline.html.tmpl
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: **** Reason: [Errno 2] No such file or directory: '/etc/weewx/skins/forecast/includes/icons/wdc.svg'
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****  Traceback (most recent call last):
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 334, in generate
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      unicode_string = compiled_template.respond()
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "_etc_weewx_skins_weewx_wdc_offline_html_tmpl.py", line 191, in respond
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 1685, in _handleCheetahInclude
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      nestedTemplateClass = compiler.compile(source=source,
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 738, in compile
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      fileHash += str(os.path.getmtime(file))
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                      ^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "<frozen genericpath>", line 55, in getmtime
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****  FileNotFoundError: [Errno 2] No such file or directory: '/etc/weewx/skins/forecast/includes/icons/wdc.svg'
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: Evaluation of template /etc/weewx/skins/weewx-wdc/about.html.tmpl failed with exception '<class 'FileNotFoundError'>'
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: **** Ignoring template /etc/weewx/skins/weewx-wdc/about.html.tmpl
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: **** Reason: [Errno 2] No such file or directory: '/etc/weewx/skins/forecast/includes/html-head.inc'
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****  Traceback (most recent call last):
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 334, in generate
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      unicode_string = compiled_template.respond()
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "_etc_weewx_skins_weewx_wdc_about_html_tmpl.py", line 135, in respond
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 1685, in _handleCheetahInclude
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      nestedTemplateClass = compiler.compile(source=source,
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 738, in compile
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      fileHash += str(os.path.getmtime(file))
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                      ^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "<frozen genericpath>", line 55, in getmtime
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****  FileNotFoundError: [Errno 2] No such file or directory: '/etc/weewx/skins/forecast/includes/html-head.inc'
Jul 11 07:36:13 cyclone weewxd[3918]: INFO weewx.cheetahgenerator: Generated 14 files for report WdcReport in 602.19 seconds
Jul 11 07:36:13 cyclone weewxd[3918]: INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: Evaluation of template /etc/weewx/skins/forecast/single-table.html.tmpl failed with exception '<class 'FileNotFoundError>
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: **** Ignoring template /etc/weewx/skins/forecast/single-table.html.tmpl
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: **** Reason: [Errno 2] No such file or directory: '/forecast_table.inc'
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****  Traceback (most recent call last):
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 334, in generate
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      unicode_string = compiled_template.respond()
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "_etc_weewx_skins_forecast_single_table_html_tmpl.py", line 107, in respond
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 1685, in _handleCheetahInclude
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      nestedTemplateClass = compiler.compile(source=source,
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 738, in compile
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      fileHash += str(os.path.getmtime(file))
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                      ^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "<frozen genericpath>", line 55, in getmtime
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****  FileNotFoundError: [Errno 2] No such file or directory: '/forecast_table.inc'
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: Evaluation of template /etc/weewx/skins/forecast/multiple-tables.html.tmpl failed with exception '<class 'FileNotFoundEr>
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: **** Ignoring template /etc/weewx/skins/forecast/multiple-tables.html.tmpl
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: **** Reason: [Errno 2] No such file or directory: '/forecast_table.inc'
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****  Traceback (most recent call last):
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 334, in generate
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      unicode_string = compiled_template.respond()
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "_etc_weewx_skins_forecast_multiple_tables_html_tmpl.py", line 127, in respond
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 1685, in _handleCheetahInclude
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      nestedTemplateClass = compiler.compile(source=source,
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 738, in compile
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      fileHash += str(os.path.getmtime(file))
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                      ^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "<frozen genericpath>", line 55, in getmtime
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****  FileNotFoundError: [Errno 2] No such file or directory: '/forecast_table.inc'
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: Evaluation of template /etc/weewx/skins/forecast/single-strip.html.tmpl failed with exception '<class 'FileNotFoundError>
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: **** Ignoring template /etc/weewx/skins/forecast/single-strip.html.tmpl
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: **** Reason: [Errno 2] No such file or directory: '/forecast_strip.inc'
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****  Traceback (most recent call last):
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 334, in generate
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      unicode_string = compiled_template.respond()
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "_etc_weewx_skins_forecast_single_strip_html_tmpl.py", line 109, in respond
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 1685, in _handleCheetahInclude
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      nestedTemplateClass = compiler.compile(source=source,
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 738, in compile
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****      fileHash += str(os.path.getmtime(file))
Jul 11 07:36:13 cyclone weewxd[3918]: ERROR weewx.cheetahgenerator: ****                      ^^^^^^^^^^^^^^^^^^^^^^

My weewx.conf is very straightforward, almost no customization.

@Daveiano
Copy link
Owner

@mymaestro @silke-99

So I did some research and stumbled upon this posts:

Belchertown has a very similar issue: poblabs/weewx-belchertown#924

Also related to weewx: https://github.com/weewx/weewx/wiki/v5-performance-troubleshooting

I currently do not have the time to go through all the info provided but it could be related to the old db schema (v3 and earlier) vs the new db schema (v4 and later) and missing types in the database. When the data is not in the DB, it needs to be calculated at report run time - with a lot of years of data and a lot of types, this can take some time.

Possible fixes from the posts:

From poblabs/weewx-belchertown#924 (comment):

From experience, it seems that the type appTemp usually causes the problem. A solution may be as simple as

weectl database add-column appTemp
weectl database drop-daily
weectl database rebuild-daily

See the documentation for weectl database add-column.

Be sure to do a backup first!

From https://groups.google.com/g/weewx-user/c/6rl2FIbqVp4/m/vNJ_bwjEBwAJ:

weectl database add-column appTemp
weectl database add-column cloudbase
weectl database add-column visibility
weectl database add-column windrun
weectl database add-column cloud_cover
weectl database add-column aqi
weectl database calc-missing 

Would be great if somebody could test this out and report back!

@silke-99
Copy link
Author

I have checked in my database, and most of these columns were existing and populated. The ones missing were:

visibility
cloud_cover
aqi

I have created those columns and tested the backfill for some recent data (complete backfill for all data would take A LONG time).

However, these columns are only filled with NULL at the moment, both by the service and by backfill. So I guess there is no module currently calculating these observations.

@Daveiano
Copy link
Owner

Ok, so this wont make any difference I guess.

What you could do to clarify if this is a general performance issue or related to some observations: You could try to remove observations in skin.conf from stat_tile_observations , table_tile_observations and diagrams: https://github.com/Daveiano/weewx-wdc/blob/3.x/skins/weewx-wdc/skin.conf#L563

Would be interesting if you can get to the point where no more errors appear and all files are generated.

Sorry, but I am missing ideas here on how to solve this. I also think your hardware should be easily capable of handling this.

@mymaestro
Copy link

Adding the columns to the database seems to have resolved the problem for me!
Before adding them, doing weectl report run took several minutes and log many errors. Now it finishes in < 9 seconds!

time sudo weectl report run
Using configuration file /etc/weewx/weewx.conf
All enabled reports will be run.
Generating as of last timestamp in the database.
Done.

real    0m8.622s
user    0m0.006s
sys     0m0.005s

@tongboy
Copy link

tongboy commented Aug 17, 2024

Just an FYI - I followed

@mymaestro @silke-99

So I did some research and stumbled upon this posts:

Belchertown has a very similar issue: poblabs/weewx-belchertown#924

Also related to weewx: https://github.com/weewx/weewx/wiki/v5-performance-troubleshooting

I currently do not have the time to go through all the info provided but it could be related to the old db schema (v3 and earlier) vs the new db schema (v4 and later) and missing types in the database. When the data is not in the DB, it needs to be calculated at report run time - with a lot of years of data and a lot of types, this can take some time.

Possible fixes from the posts:

From poblabs/weewx-belchertown#924 (comment):

From experience, it seems that the type appTemp usually causes the problem. A solution may be as simple as

weectl database add-column appTemp
weectl database drop-daily
weectl database rebuild-daily

See the documentation for weectl database add-column.
Be sure to do a backup first!

From https://groups.google.com/g/weewx-user/c/6rl2FIbqVp4/m/vNJ_bwjEBwAJ:

weectl database add-column appTemp
weectl database add-column cloudbase
weectl database add-column visibility
weectl database add-column windrun
weectl database add-column cloud_cover
weectl database add-column aqi
weectl database calc-missing 

Would be great if somebody could test this out and report back!

Just an FYI - I followed these steps exactly, and then had to run "sudo weectl report run" and it completely fixed this issue on my install after it finished.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants