Error on critical.wav resource

[mariusvw]

It seems that the critical.wav is sometimes missing in the resources directory.

Solution suggestion always copy the default in place when resource is not found.

[HenriWahl]

I have no idea why this happens but can confirm that it does. Looking into this temporary resources folder I found that most of the resource files where missing.

Looks basically like the folder https://github.com/HenriWahl/Nagstamon/tree/master/Nagstamon/resources is copied there at start because all files exist there when Nagstamon has been started freshly but for some unknown reason most of the files vanish after a while.

I have no clue why exactly this /var/folder/* is created at all but assume it has to do something with pyinstaller which is used for packaging of Nagstamon. But this is just a guess.

[pbrunnen]

Hello. I have noticed this error pop-up before all versions. It seems to occur when Nagstamon has been open and running for some time. A quit and restart will always resolve the issue. But in all the versions, the audio alert has still played back appropriately even when the message appears. Is it possibly a sandbox issue where the error is itself in error?

[HenriWahl]

I found out that this problem is related to pyinstaller. Pyinstaller offers 2 ways to package an app: as one file or as directory. I some years ago choose the one file way. When using this mechanism, the one file gets unzipped in the /var/folders/something-directory. This as such is not a bad thing, but macOS from time to time cleans up this folder... the code already running is not involved, but the .wav-files are, because they get simply loaded when needed.
I see 2 solutions:

• either preload the default sounds, so it will be no problem when they vanish
• or get Nagstamon packaged as directory, which I never succeeded with

[HenriWahl]

I was too fast... this solution 30c9604 only works on Linux - where it is not needed.
So the road to running Nagstamon from a directory seems to be the more prospective.

[HenriWahl]

I am not able to get pyinstaller to package Nagstamon as one directory, getting it running.
So both plans are kind of failed.

Another rather mad approach might be to cache the .wav-files and replace them in case they where deleted by macOS...

[HenriWahl]

I managed to store a copy of the .wav-files in memory and when they are not existing, they will be recreated. Pretty mad but works. Maybe you can give it a try with https://nagstamon.ifw-dresden.de/files/incoming/Nagstamon%203.3-20191011.dmg.

[pbrunnen]

Well you beat me to posting my suggestion, but that works. 😀
I have loaded the new version and I can let you know what happens to the files.

[pbrunnen]

Hello HenriWahl,
The new build from the 11th is crashing quite often. I have not been able to tell if this is when the files are missing or not. The crash dump is in the main thread, but the dump doesn't really provide much from what I see.

`Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_CRASH (SIGSEGV)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff5d8e12c6 __pthread_kill + 10
1 libsystem_pthread.dylib 0x00007fff5d99cbf1 pthread_kill + 284
2 libsystem_c.dylib 0x00007fff5d7fed8a raise + 26
3 de.ifw-dresden.nagstamon 0x000000010bfab94c 0x10bfa7000 + 18764
4 de.ifw-dresden.nagstamon 0x000000010bfa95e4 0x10bfa7000 + 9700
5 de.ifw-dresden.nagstamon 0x000000010bfa7cb4 0x10bfa7000 + 3252`

[HenriWahl]

Yes... I see that I see nothing helpful... I will check as soon as possible.
Did it crash alone without any before action or was something else involved?

Edit: the automatic file deletion seems to happen after some days running.

[HenriWahl]

My testing instance is still running. It's a Catalina macOS. Which is your version?

Nagstamon is build on a Sierra host.

[HenriWahl]

Can you please sure it is the very last release you downloaded? There were several versions this day. Maybe reload it from the website just to make sure it is the latest?

[pbrunnen]

Hello HenriWahl,
I am on Mojave (10.14.6) with the latest supplemental updates.
It crashed alone... twice when the computer woke up and the remaining five times while it was just sitting there. No action or change in alert status (we do have two critical alarms right now). Just updated to the latest on the website to be sure. I'll keep testing and update the thread.
Thank you! 😊

[HenriWahl]

@mariusvw can you please check if the latest testing release solves this problem?

[mariusvw]

@HenriWahl
Installed 3.3-20191010, it crashes when a warning or critical state is triggered.
Not able to pinpoint the exact cause for that.

Another thought, why store the wav files in memory and not in the resource location of the application directory? (e.g. /Applications/Nagstamon.app/Contents/Resources)

[HenriWahl]

Please download latest 20191014.

Storage in memory is easier because all other platforms do not use these macOS-folders and pyinstaller isolates the runtime anyway.

[mariusvw]

@HenriWahl tested with 3.3-20191014, crashes at the same moment

[HenriWahl]

What macOS version do you use? Which server do you use? Did any version of Nagstamon work before?

[mariusvw]

@HenriWahl version 3.2.1/20190607 worked just with the wav file warnings. Running Mac OS X 10.15 (Catalina), I have multiple server versions with Nagios Core and Opsview, all result in the same.

[HenriWahl]

And now crashes as soon as you start or when some alert occurs?

[HenriWahl]

What are your displaying settings? There were some changes so let me know: is the popup status window closing when moving the pointer out of it and is Nagstamon running in systray or as floating statusbar?

[mariusvw]

The crash happens when the program is running in the "menubar" of Mac OS X.

Here the current settings tab:

[HenriWahl]

OK... and crashes occur when alarms happen and sounds shall be played?

[mariusvw]

I can't tell which moment it happens, it runs in the background in the menubar, without any sound it crashes and shows a crash report.

[HenriWahl]

So you have sound disabled?

[HenriWahl]

@mariusvw There was a freshly introduced bug in latest testing release, please check the 20191015 version.

[pbrunnen]

From my side I had the crashing narrowed down to when the system was waking from a sleep state.
20191014 did not crash for me when alerts occurred (sound did play) or when the entering/exiting the expanded view from the floating status window.
I will also update to 20191015 now.
Thanks!

[mariusvw]

New day, restarted my macbook and ensured all caches were cleared, solves the crashing on 20191014. Sounds played.

Have not tested yet what happens when the caches are cleared again, takes some days to let it happen, will report back when in 2 days.

[HenriWahl]

@mariusvw @pbrunnen It would be interesting which files and mostly important which directories get deleted when macOS has the idea to cleanup /var/folders. If Nagstamon loses all its files AND its directories like /private/var/folders/7w/hfvrg7v92x3gjt95cqh974240000gn/T/_MEIM1WVB5/Nagstamon/resources it also will have to recreate directories. In case Nagstamon crashes for you, can you please check how empty the temporary folder is or what files and directories are missing?

[pbrunnen]

Good day @HenriWahl
This morning it crashed at Mac wakeup. I looked into the folder and it was empty... To be sure, I have cleared out my private/var of all references to Nagstamon and I have a script watching for the contents.
Thanks!

[pbrunnen]

Good evening @HenriWahl
I can confirm after waking from sleep Nagstamon crashed and the /private/var folder still had all of its contents. No error state at the time.
Thanks!

[HenriWahl]

I added the not-only-restore-sound-files-but-directories-too feature in 3.3-20191016, so this one should be the final solution.

At least regarding the missing-file-problem. If it still crashes after sleep I am not sure what to do then.

[HenriWahl]

@pbrunnen please try to run Nagstamon.app/Contents/MacOS/Nagstamon directly - it will open a terminal window which might help to debug the wakeup issue.

[mariusvw]

@HenriWahl will upgrade to 3.3-20191016 today, so far 20191014 didn't crash, even after a 2 day sleep of my macbook. All contents in /var/folders remained, as soon as those got cleaned out without putting the macbook in sleep mode Nagstamon crashed.

Will run it again with the same test today running from CLI to get most debugging information.

[HenriWahl]

@mariusvw if running with CLI please as described above from Nagstamon.app/Contents/MacOS/Nagstamon because this is the pyinstaller version which creates the sometimes vanishing folders in /var/folders/*.

Thanks for debugging!

[pbrunnen]

Good morning @HenriWahl,
Understood. I'll upgrade to 20191016 and run it from the command line.
Thank you.

[pbrunnen]

Hello @HenriWahl,
So it crashed on me waking. I caught the stdout to a log file, but I didn't get anything useful. All that was logged is:

• QLayout: Attempting to add QLayout "" to StatusWindow "", which already has a layout repeated three times at startup.
• /var/folders/9j/954...0101/T/_MEIjpRZD6/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings repeated once per check on the server.

Unfortunately that was all that was logged. I also attached the crash report as it did seem to have more detail compared to prior.

I will try this again to reproduce capturing all messages to file.
Thank you!

crashdump.txt

[HenriWahl]

Well... reading the crashdump I only can confirm that it... crashed. The error messages you saw in the terminal are of no harm.

Was there an older version which did not crash after wakeup? I updated Qt5 for the latest testing releases because support for newer macOSes should get better.

Does it crash immediately when waking up or is Nagstamon still able to do something?

[mariusvw]

After running it from the console yesterday I noticed one error so far:

qt.svg: Cannot open file '/var/folders/0m/lsmq1_95589cxfx5fds3z16m0000gp/T/_MEIb1ILhQ/Nagstamon/resources/nagstamon_systrayicon_empty.svg', because: No such file or directory

Still couldn't find a good reason for the other errors yet

[HenriWahl]

OK... so this file has to be included into the file backup cache too... I'll take care of it. Did it result in a crash?

[mariusvw]

It didn't crashed

[HenriWahl]

Latest 3.3-20191018 contains fix to avoid #578 (comment)

[HenriWahl]

Now there is a new 3.3-20191018 available at https://nagstamon.ifw-dresden.de/download/#testing which caches every *.wav and *.svg file and restores them on the fly if needed. Worth another check over the weekend...

[HenriWahl]

@pbrunnen @mariusvw any news so far? My long term test runs since days and sound is still played without any error message.

[pbrunnen]

Hi @HenriWahl,
Yes, the sounds have been playing fine.

I have narrowed the crash down to only when I'm waking at the office in my docking station. The most I can get is 'Segmentation fault: 11' when running through the terminal. So this is some strange edge case that is unrelated... It may even a timing issue with the Kerberos change as all the prior versions did not crash.

I would just close this bug as complete. I can open a separate bug if you wish, but this is certainly something I can live with.

Thank you for everything!

[HenriWahl]

Would be fine to just close it... :-) And thanks for debugging too!

[mariusvw]

At my side it works without crashing and the sounds play.

When things change, will let you know.

[HenriWahl]

@mariusvw @pbrunnen can you please check the very latest testing version 3.3-20191112?

After solving this *.wav problem there came up a new one - this time the certifi/cacert.pem file was automatically deleted from /var/folders/*/Nagstamon and missed by the HTTP request machinery as seen in attached screenshot.

Now I let Nagstamon also cache and detect this file.

[pbrunnen]

Hello @HenriWahl ,

can you please check the very latest testing version 3.3-20191112?

Certainly. Does the custom CA parameter need to be set to something and if so, what type of certificate?

Thanks!

[HenriWahl]

No - this certifi/cacert.pem comes with Nagstamon but gets deleted like the *.wav-files. I just managed to restore this file too because it seems to be needed for every HTTP-request.

This problem only could appear because the *.wav-files-deletion-problem was solved. Now both problems are gone and I am interested if another one will raise. Hopefully not.

[pbrunnen]

Hello @HenriWahl ,
I upgraded again to the latest this morning. So far I have not had the problem appear...

[mariusvw]

@HenriWahl I haven't noticed any problems yet.