Stops working after a week #1065
Comments
|
Yep, same on my side. It seems, that the scheduled task is still runnning, but I also get an HTTP 500 on the frontend. |
|
Same here, only started the container yesterday (sat) around 15.00 (UTC+1), last run was at 23.00 (UTC+1), this morning I also saw the error 500. Something happening on sat/sun night? @michaelmittermair can you also confirm similar behavior? |
|
Yes, look similar. I checked my container. It was running longer than just 1 day, but the last run was yesterday (SAT) 23:47pm (UTC +1). |
|
Same here, no logs that give any useful information. Last successful run was (SAT) 23:42 (UTC+1). Next run would have been at (SUN) 00:12 (UTC+1). |
|
Same here |
|
same here. logs don't indicate anything has gone wrong. simply after about a week, server crashes, and web ui only loads showing |
|
@te5s3rakt could you do one additional check (as I believe this could narrow down the probleem a bit). Could you check if this happens after midnight on Saturdays? |
sure thing. my local time? |
|
Yes, what UTC offset are you in? |
UTC + 8 I've just restarted and updated my container. I'll check next weekend how the container is going. |
|
I had this too this morning. A restart of my docker container restored access for me. I noticed in the logs that Laravel was throwing an error saying that no application encryption key has been specified. |
|
same here |
I'm seeing this same error in the logs |
|
i dont think that the app key is the reason for the server crash. I have
a key generated and dont have a warning on start.
My guess is that there is something wrong in: /app/Console/Kernel.php
/**
* Perform system maintenance weekly on Sunday morning,
* start off the week nice and fresh.
*/
$schedule->command(SystemMaintenance::class)->weeklyOn(0)
->timezone($settings->timezone ?? 'UTC');
|
|
I have also a 500 on one of my instances, got the following error: 192.168.176.23 - - [24/Jan/2024:11:17:44 +0100] "GET /api/speedtest/latest HTTP/1.1" 200 386 "-" "-" |
|
My instance (Docker, using Postgres) just stopped responding at 00:00 EST (05:00 GMT) on Sunday morning. The same thing happened last Sunday. Logs seem to indicate some kind of issue connecting to Postgres. However, the Postgres server is running fine and connected to other apps. A restart of the Docker container fixes the issue. The logs show the following: |
Was right on time. Sunday morning container sh!t itself. |
|
I get the same error after midnight Sunday, 500 | SERVER ERROR |
|
Same again for me this Sunday. Error once again this one (when browsing to the server): |
|
same problem here, running on RPi 4 with Postgres DB. 2024-01-28T07:58:15.918491042Z [previous exception] [object] (PDOException(code: 2002): SQLSTATE[HY000] [2002] Connection refused at /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php:65) 2024-01-28T07:58:22.570684825Z [2024-01-28 07:58:22] production.ERROR: No application encryption key has been specified. {"exception":"[object] (Illuminate\Encryption\MissingAppKeyException(code: 0): No application encryption key has been specified. at /var/www/html/vendor/laravel/framework/src/Illuminate/Encryption/EncryptionServiceProvider.php:79) I have an app key specified, if I recreate the container it will work for a few days. restarting the container produces this error: |
I think mine also stops working every Sunday. |
|
Same here. My logs:
|
|
yep i get this sunday morning problem. restarting the speedtest and db containers gets it running again. |
I only restart my speedtest container. my db one never presents any issues with anything else I have connected to it. |
|
same here (stopps with Error 500) Database is mariadb - restarting web container fixes the issue |
i first tried restarting the speedtest container, but no joy. when i restarted both it worked again. |
Same here. First week with this running, discovered it stopped last night. Restart of the Speedtest container fixed it. |
|
As a temporary workaround, you can add the following to your /etc/crontab on the docker host system to restart the container every sunday morning at 00:02 local time:
|
I agree this is probably the source of the issue. SystemMaintenance is sourced from I can confirm that I can replicate the crash at a specific time by editing /app/console/Kernel.php and then restarting the container:
This causes a crash at 12:12 local time. If I comment out both calls to Artisan in refreshCache () (SystemMaintenance.php), the server stays up even through a scheduled maintenance. Leaving either one uncommented will still result in a crash. My guess is that the optimize command somehow clears out the app key, even for people who have specified one, leading to a crash. |
|
+1 me too... |
|
I still have the problem. Stopped exactly at midnight. Front end worked. Just no tests run. |
|
oh, i just noticed, although neither of my instances stopped working, the scheduled behaviour changed after midnight. One of my instances is scheduled to run 4 tests each day: at 2:01AM, 3:01AM, 4:01AM and 5:01AM As you can see here, this was happening fine all week, but then today (sunday) the first instance only ran one test at 5:01:20AM 
And the second instance ran at the correct times, but only ran two tests instead of the scheduled four.. 
|
|
Another 'me too', stopped working at midnight. Using mariadb 10. |
|
#1097 and Let's see if this helps 🤞 |
|
Don't think this helped, unfortunately. Like before I added I don't think the try/catch is the issue here, I think something about the Artisan cache clear is destroying some cached data that the app expects to still have. |
|
@mmomjian from your |
|
@alexjustesen here is some info to help. I had already changed the Fair warning this is a super long comment, my apologies. docker-compose.yaml Run a a fresh docker compose down and up The website works fine. I edit app/Console/Kernel.php to change the time of SystemMaintenance to 14:22 local time. At that time, the website returns a 500 error. Docker logs show: |
|
@mmomjian can you share your |
|
Docker compose is below. The env file is just the postgres connection parameters. Here is the .env file from within the container: |
|
Thanks I'll dig into this more later. In the meantime does your |
|
Just updated my comment. There is a 31 character app key in the container's |
|
@mmomjian that's normal, make sure you generate your key correctly. |
|
@alexjustesen I didn’t generate the key, that’s the automatically created one. |
|
@mmomjian is there a problem with the key? If so want to keep on topic to make sure I've addressed the scheduler stopping. @ReVoLt112 is OP so if they say it's resolved or others can confirm I'd like to get this issue closed. |
|
@alexjustesen I'm not sure what you mean by a problem with the key? As my last checking, the web portal stops replying at 00:00 local time every Sunday morning. Anything beyond that is just our guesses at what the problem could be. You had asked me about the |
i will report back on Sunday morning |
is there a way to "force" it to maintenance mode so we can try before Sunday morning ? |
|
@Churator @alexjustesen I have been simulating a maintenance by editing Kernel.php to change the time of the cron job to whenever I want (outlined above). I just did a test with v0.15.0 and the crash behavior (500 error, unable to connect to DB) is exactly the same as it was in the earlier version. After the test, I always destroy the old container and start with a fresh install. We will see what happens on Sunday but I highly suspect it will be the same crash behavior. |
|
I've extensively triggered the error over the last few days manually. The conclusion I arrived at is that as soon as the system maintenance runs all settings that are set by docker environment variables are no longer used. Steps I took: This leads me to the conclusion that with system maintenance running only the default settings from the source code and the settings from the .env-file come into play - docker environment variables are no longer used by the app (but still set - I checked with "set"-command). Strange is that all this only happens when the maintenance commands are triggered by the source code. If I run php artisan optimize oder optimize:clear in the console the error never happens. |
|
@Starfiresg1 this is an excellent writeup and it tracks exactly with my observations. I can't quite seem to figure out what about the source code is causing this, since the same commands run in the console do not cause it. Could it be that running it from the console persists the env vars through the Shell and this allows artisan to retain the otherwise-lost variables through an interactive shell? |
|
@Starfiresg1 this is excellent, thanks for digging into the issue. I'm using |
|
Once |
|
@alexjustesen of course, no problem |
|
Pulled v0.15.2 and reset my .env to the default (no DB-settings, no APP_KEY) and run the system maintenance twice by modifying Kernel.php - the web GUI survived both times without displaying an error. |
|
Fixed on my end as well, using env vars, and the modified cron job time. Thanks @alexjustesen ! |
|
Putting a pin in it then! Thanks all |
and others
Hi, after one week my speedtest-tracker stops working. Sometimes it just doesnt execute scheduled tests, sometimes i get a HTTP500. Has someone also noticed same behaviour?
Unfortunately no helpful logs are produced... I am watching 3 instances and all 3 crash nearly same moment from saturday to sunday
The text was updated successfully, but these errors were encountered: