Web interface fails after several days

[cxselph]

Describe the bug
I leave Companion running 24/7. After about a week, the web interface looks like there is no CSS. When I try to refresh the page, it gives the following error. This has happened on just about every version that has been available.

Error: ENOENT: no such file or directory, open '/var/folders/80/rgd9416d71sbkwzd_yhmgkl80000gn/T/.companion.bitfocus.no.sBN3AJ'

To Reproduce
Steps to reproduce the behavior:

1. Open Companion
2. Wait about a week
3. Try to go to admin interface

Expected behavior
I expect it to run forever. :)

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: OSX 10.14.3 - Same issue with Sierra and High Sierra
• Browser: Chrome Version 73.0.3683.86 (Official Build) (64-bit), Safari 12.0.3

[josephdadams]

This happens to me as well. MacOS Mojave, Chrome. Restarting Companion fixes it, but I too expect it to run forever. :)

[krocheck]

When this occurs, do you notice anything out of the ordinary in term of processor or RAM utilization?

[josephdadams]

I haven't noticed any performance issues, and I never stopped to check. I also recently modified my production Macs to reboot once a week and auto start Companion (among other apps) so I don't see this error hardly at all anymore.

[cxselph]

[Phil0]

I have the same Issue with running 24/7.

Error is
Error: ENOENT: no such file or directory, open '/var/folders/81/rqcz5trx3kb2zlv1zqtk5xgc0000gn/T/.companion.bitfocus.no.suEIqc'

It occurs after arround/little under a week, when I restart it under Admin it then runs for a week but next Sunday it is crashed. I just had the situation that it was running before the Service and after it it was crashed.

The issue is just with the web interface, everything else seems to run well for longer.

Maybe the logs get to full or something because not all devices run 24/7 and the Modules throw errors or not reachable messages into the log.

[josephdadams]

I'm still getting it too. @Phil0 which version of Companion are you running?

[Phil0]

Was using 1.4 on Mojave's but we have updated on Catalina now. I'm inspecting and will give feedback soon!

[cxselph]

I have been using 2.0 for a while now on mojave and still have the issue.

[cxselph]

I use another node app on github that had a similar issue and it has been fixed. It may not be related but could be helpful. karlcswanson/micboard#12

[onfire4g05]

This can be duplicated by running the compiled version (I've only gotten it to do it on the compiled version -- see note at the bottom) and then setting your system clock ahead by a year (be sure to disable all running apps when doing this as it may cause havoc on some apps). Some observations I made are:

• Once it fails, it always fails until restarted.
• Once you've gone so far in the future you seem to be "ok" until a computer reboot when the process starts over. So, if you use year 2021, next time you run the app you won't have the problem until you get to 2022, etc.
• Only the public static path is affected (/press/bank, /js/jquery/jquery.js, etc, are not).
• Moving the base static path around has no affect (like moving it to the bottom of all others).
• This may be Mac specific, but I don't know.
• Here's a stack trace of what happens, which doesn't seem to be useful at all (relavant lines from nodes timers.js):

Error: ENOENT: no such file or directory, open '/var/folders/m5/rpghj_d901z9_t5r2bxyj4n4hq2sy2/T/.companion.bitfocus.no.v7LdWo'
--------------
Error
    at Function.logerror (/Users/josborne/Source/companion/electron-output/mac/Companion.app/Contents/Resources/app.asar/node_modules/express/lib/application.js:630:43)
    at runCallback (timers.js:795:20)
    at tryOnImmediate (timers.js:751:5)
    at processImmediate [as _immediateCallback] (timers.js:722:5)

• Could be a bug in the node version used?
• Modules don't affect the issue (loading no modules still causes the failure).
• Disabling everything except the server_http still causes the problem (basically commenting out io - help in app.js).
• Older versions of express also have the issue.
• The file/folder that's attempted to be loaded never exists on the system.

So in dev mode... when running the app in dev for over 12 hours and then changing the date, I did have it lock up and require a force quit. It's a little bit of a different issue though, and I couldn't get it to always do it so I'm not sure if the issues are related at all or if it was just a fluke.

[MichaelAllen]

/var/folders/.../T is where Mac stores user-specific temporary files. (You can run /usr/bin/getconf DARWIN_USER_TEMP_DIR to see what the specific path is for your user account.) Under that folder, there are several .companion.bitfocus.no.___ files. One of those files contains the html for the web interface. (Manually rm that file and you get the same error)

My best guess is that express doesn't like serving files out of the asar archive, so it is extracting the files for the web interface to a temp location before serving them, and sometime later the OS is whacking them once they are older than a certain date.

[onfire4g05]

Hmmm, I compiled with it specifying a hard coded path in my home directory and it still failed after the date changes. I suppose some magic could happen to duplicate the folder, so I can easily check that by editing some files in my hard coded path. I didn't see the folder in the temp folder (/var/folder), but I probably missed it. I'll check again later.

That doesn't explain, in my opinion, why the file/folder it is attempting to access changes the last bit for separate web paths (for example, index.html might be xhslf while a css file may be kfiebc). Moreover, I couldn't get the error middleware function to even get called, which is very strange... unless there's some further logic that would prevent the middleware from running in some weird case like that.

[willosof]

This smells like «upgrade Electron». The version we’re running is very old. If anyone manages to upgrade it (and keep everything working), I’ll buy them a beer. Or a whole case.

[onfire4g05]

I do stand corrected on the path thing, I looked into it again and you are right. That might solve all the issues, actually. 😅 Or at least makes me feel less crazy.

[onfire4g05]

This smells like «upgrade Electron». The version we’re running is very old. If anyone manages to upgrade it (and keep everything working), I’ll buy them a beer. Or a whole case.

So, I got it to build with Electron 9.2 and Node 12.14.1 so that I could test if that'd fix this issue, and, unfortunately, this particular bug remains. :(

I'm going to do more digging, since there must be a config/diff location/something to keep this from happening.

[bryce-seifert]

This was fixed for me for a while, but with the latest dev builds it seems to have popped back up:

Error: ENOENT: no such file or directory, open '/var/folders/p0/r9w5l1216v98z6r7138y4zrh0000gn/T/.companion.bitfocus.no.wBUBNM'

[josephdadams]

@onfire4g05 any thoughts?

[Julusian]

Could you send a screenshot of the errors in the chrome console.
You can get to there by right clicking the page and selecting inspect, then switch to the console tab.
That will help us see what it is trying and failing to load

[bryce-seifert]

[Julusian]

that did not provide the detail I thought it would. my fault not yours
switch to the network tab then refresh the page. screenshot of anything shown red in that please

[bryce-seifert]

No problem, here's the network tab:

Request URL: http://192.168.10.4:8000/
Request Method: GET
Status Code: 404 Not Found
Remote Address: 192.168.10.4:8000
Referrer Policy: strict-origin-when-cross-origin
Accept-Ranges: bytes
Cache-Control: public, max-age=0
Connection: keep-alive
Content-Length: 262
Content-Security-Policy: default-src 'none'
Content-Type: text/html; charset=utf-8
Date: Mon, 01 Feb 2021 21:05:19 GMT
ETag: W/"8a74-1774987f368"
Last-Modified: Thu, 28 Jan 2021 15:06:22 GMT
X-App: Bitfocus AS
X-Content-Type-Options: nosniff
X-Powered-By: Express
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cache-Control: max-age=0
Connection: keep-alive
Host: 192.168.10.4:8000
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36

[Julusian]

huh, so its the root page that is failing. I was expecting one of the libraries or something to be the problem..

[Julusian]

so this is surprisingly easy to replicate by after running companion, finding and deleting all the temp files created under /var/folders.
Using find /var/folders | grep bitfocus to find the folders, then rm -R for each

What I am finding is that when I do that, then I get the 404 errors. Looking at the build, the files are included inside the electron asar file, which they probably shouldnt. But by forcefully removing those, then it doesnt work at all.

So I suspect that either the electron update broke this, or less likely maybe it was never truly fixed.
The fix should be simple enough to:

• exclude the public folder from the asar file on mac (or does every os need these fixes?)
• in the http server, if we are in electron and on mac (or we could be clever and check if it exists on disk first?), modify the path given to express.static to replace /app.asar/ with /app.asar.unpacked/

[Julusian]

or perhaps we should look into why this happens only for the public folder. various other files are served via express in almost the exact same way, but are not extracted like this. why is that?

[Julusian]

A further update, this is a bigger problem than initially thought. All of the libraries used also break.
From what I can see is happening here, is that the files are being cached on disk as they are first served to a client. Once I delete these, those files will instead 404 error.

So the same fix will need to be done for the exposed libraries too. This feels messy, but it will be short-lived code until #1402 is ready

[onfire4g05]

What I did, was just exclude them from asar. Since when they are in the asar The files will eventually stop working after Companion has been running for so many days. It's been a while since I looked at this, but I don't think the files were actually deleted (deleting the files will obviously cause the error). You can set the time in the future (about 10 days) and it'll always just stop working.

[onfire4g05]

Take a look at #1193, which fixed the issue by excluding these files from the asar.

[Julusian]

Yeah, the files are still being excluded from the asar, but that version of them simply isnt being used.

I somehow missed the fact that the libraries were excluded from the asar too.
I just tried with setting the clock forward, and that does also trigger this.

[Julusian]

This should be fixed in build 3096 (not 3095, I broke that in another way...) 5e11519

The previous fix was to unpack them from the asar file.
This new fix does a similar thing but makes it more explicit by adding the files as 'resources'. meaning they live in a folder on disk that can be referenced explicitly. This lives inside the app bundle on mac so will not be cleaned up by the os.

Please give it a test, and if noone reports an issue within the next week or so I shall close this