-
Notifications
You must be signed in to change notification settings - Fork 10.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Laravel 5.3 Job fails without calling failed() method #15192
Comments
failed()
method
failed()
method
PHP doesn't work like that. Adding another parameter won't cause it to suddenly start getting called. PHP would still call it, but with just too many parameters, which is the same as calling it with no parameters for all intense and purposes. |
Normally I would agree, but from what I can tell, this alone resolved the issue. I'll attempt to recreate with some sample code. |
Could not reproduce with a I understand this is unlikely the issue. But now I'm curious what I changed that resolved my original issue. |
Okay, so I think this is related to My job queue did not behave this way when running 5.2. So I still think there's something wonky here. I'll keep digging. But for now, I'll just set my |
We're doing an upgrade to 5.3 right now and coming across some weirdness around failed jobs with the database queue driver. If I have an email job sitting in So confused! Not sure if our problems are related, but it seems something is funky with failed job handling and database queues right now. |
@jesseleite that sounds exactly like the issue I was having at first as well. Be sure to restart your queue after upgrading, I think that was one of the things that added to the weirdness. Nonetheless, I still think there's something here. I planned to look at this closer over the weekend to provide some code samples to reproduce the issue between 5.2 and 5.3. |
Currently my job is failing because of the issue @jasonmccreary raised in the first place
I can find this in my Making sure that the @GrahamCampbell The
|
@jasonmccreary sorry for delayed reply, was camping all week ⛺
As mentioned in my original post, we're running into this problem when manually running
Curious if you had a chance to dig in? |
@crynobone and I were chatting about this commit he made a while back to InteractsWithQueue. He suggested the That said, we're still having problems with jobs not being released to either the |
@jesseleite sorry for the delay. I have not. Just as you have, I spent a few hours trying to determine the root cause before moving on. I definitely think there is something here. Whether it's the config, PHP 7, the daemon, the driver, or some combination therein, I don't know. But somethings up. Hopefully we'll get some more participation on this issue. |
Check this commit: e1d60e0 The method was renamed to |
@themsaid, thank you I saw @taylorotwell's reply and commit. However, even if we add our own I'm not 100% sure, but I believe it might be something that changed in this commit, which was a pretty big overhaul of queues. |
@jesseleite Not sure if your issue is the same as @jasonmccreary, I'm kind of confused here :) I feel like they are two different issues, no? |
@themsaid possibly. I feel like we're both pointing at different possible causes, but neither of us are really sure what's really going on. @jasonmccreary, correct me if I'm wrong, but what I believe we both have in common is this:
UPDATE: It seems the exception is visible in my log file, but as mentioned no valuable output from |
Yes, @themsaid my original issue seems to be the same as what @jesseleite is now describing. I backed off as I did not have time to reproduce the error. Since, I have noticed something else unrelated to this specific issue, but related to the queue behavior which is the combination of |
@jesseleite I've tried to replicate the issue using the database driver, I throw an exception in the
The Job runs 3 times and then gets removed from the jobs table and a new record is added to the failed_jobs table. Are you experiencing the issue with the database driver too? @jasonmccreary couldn't replicate using the sync driver and I'm unable to replicate using the DB driver. |
I had the same problem that the failed method was not called, and the job disappeared and was not moved to the failed_jobs table. I changed my job´s failed method to also accept the event, like this: and now it works. According to this code
in this file Sorry if I am way off here, I am quite new to Laravel. |
Hi, we came across an issue this morning that sounds like its related to other posts here after upgrading to 5.3 yesterday (specifically 5.3.9 and the We had multiple jobs that hit the default timeout of 60 seconds over night, and the Supervisor was set up to run After digging through the code and trying to replicate locally, I think the expected behaviour would be that after the The jobs in question don't have a Let me know if you want any more information. |
@philbates35 I've tried to replicate locally too with no luck, that's one tricky issue to trace. |
@philbates35's issue is more inline with what I'm now experiencing. However, I'm still curious what the original issue was. I agree with @GrahamCampbell that the missing parameter isn't how PHP normally behaves, but I'm curious with PHP 7's new Error exception, if certain error reporting configurations would cause this behavior. Anyway, I'm curious the stack for people having this issue. Maybe there's a clue there. I was running Laravel 5.3.2, PHP 7.0.10, and supervisor was managing my job queue. |
Here's our stack: Laravel 5.3.9 |
@themsaid Yesterday I created a fresh 5.3 project and tried to reproduce as well, but can't seem to be able to. I would think it has something to do with my code, but then it seems quite a few people are having issues with queues lately as well. I'm just not sure where to look. It worked in 5.2, and nothing has changed on this 5.3 upgrade branch apart from upgrade guide related commits. I'll keep playing around to see if I can reproduce in a fresh 5.3 app.
Laravel 5.3.9 and upgraded to 5.3.10 I'm skeptical of it being a stack issue, since everything worked in same environment with 5.2. But who knows, I'm lost! |
Are you all using the database driver? or is it failing on different drivers? |
@themsaid reading through this issue thread again, it seems everyone that has reported issues here was using the database driver (references to |
That doesn't rule out that there issue didn't occur with other drivers too of course, just that no one has reported an issue with the other drivers. |
Yeah that's what I suspect, I'm thinking if maybe some sql error happens in the middle and for some reason doesn't get logged. |
Honestly this is a bit frustrating. I've unknowingly been using Queues with the database driver since Laravel 5.1. It's worked fine until 5.3. Now, from my perspective, it doesn't work in 5.3, so I open an issue and the answer is basically - you shouldn't have been using that. Fine, but let's at least document that if there are no plans to make the database driver production-ready. I doubt I'm the only person using the database driver in production. Clearly, there's at least 4 of us 😉 |
I have tried to reproduce, but have spent hours and can't nail down an exact situation where it happens, and where it doesn't happen. Not even sure it's JUST the database driver, but it could be.
100% agree with @jasonmccreary. If the database is worthy of storing production data, dealing with password resets tokens, etc., why should it not be worthy of queue job handling? I don't think shunning the database driver is the solution. |
The described issue seems to happen in the The issue I see is that it does not log to the I have been using The queue's also vanish if they timeout (although above it stated as when PHP restarts/fails), like described for many using the EDIT: EDIT 2: The source code shows that the timeout only logs an exception and does not mark the job as failed. If this is changed it could likely fix this if the issue is queue's timing out and causing the jobs to just vanish. Also noticed this happening with the Sorry for being long winded. I just have fought with this for a couple weeks now so I want to note all my findings. |
Can someone briefly summarize the issue in like 1 sentence? "The failed() method is never called for jobs which hit the timeout limit N number of tries" Is that correct? It's hard to parse out what the issue is in all these replies. |
@taylorotwell I think there is various issues that all seem similar as the result is lost queues that don't end up in the failed job queue. Anyway in one sentence mine would be "Using no timeout (0) results in the default timeout (60) being used sometimes." (don't know the exact trigger for it, if its every time or just sometimes) The queue when timing out does not trigger the failed function, so it took a lot of digging to figure it out. I temp solved it by using a number other then 0 in the timeout, matching it up with the I use SQS, with a 15 min visibility timeout, and I have jobs that take over 60 seconds to complete (though i have 0 as the timeout). That should be enough to replicate at least my version of the issue. |
Sorry, this does seem to have become a catch-all for queue issues. I have opened a tries vs retry_after issue (#15696). While it might be related, I consider that a separate issue than what I originally opened here. For me, this issue was - failed jobs are silently lost to the ether |
Yeah this thread has basically become an unusable mass of confusion with no clear steps on what to do to reproduce anything. |
Fair. However, I think there's something here. I had tried to come up with reproduction steps, but my focus became getting my app working again. I'll try to reinvestigate early next week. Feel free to tag this accordingly to relay it's on hold. |
Just for the record, if I manually set the timeout high, my failed job is still
Since upgrading to 5.3, jobs with errors are sometimes failing silently and are not being sent back to the jobs or failed_jobs table (lost forever). Whether because of database driver, retry_after, timeout, etc., and whether these issues are related, I think we're all equally confused. @taylorotwell I apologize for the confusion, but I think if any of us could clearly replicate we create a proper PR instead of an issue. |
Even if someone could give me a list of say 5 steps to reproduce the issue on a fresh Laravel install that would be helpful. |
https://github.com/jordanramstad/queue-validator I have made a simple package for my own use to help mitigate any damage from queue's vanishing. This is no fix but at the very least might help to diagnose / reproduce issues related to queue's. No unit tests or anything and just kind of pulled out of my application so might not work in all environments but I tried to add a clear readme anyway if anyone wants to try it out. I don't claim this is the best way to do it, just the way I am currently leaning on, feel free to open issues in it to let me know what you think. |
I had same issue that job wasn't written into failed_jobs table after it falied.
I had to delete migration file and entry in migrations table for failed_jobs and rerun php artisan queue:failed-table command. Hope this helps at solving the issue. Was jobs table structure changed as well? 2016_08_08_115250_create_failed_jobs_table.php.txt |
@mediley, these changes (and now a sample migration) are listed in the Upgrade Guide under the Database Driver Changes section. |
@jesseleite Exactly! I just had to roll back a production update to 5.3 this morning because our jobs seem to work 4 out of every 5 times, using the DB driver and latest 5.3. It's the same as you, they mysteriously vanish with no error or trace... Which obviously is the worst for anyone trying to fix this! I even have Log::info(xxxx) throughout the calling function, and all the Log entries get logged without issue when the jobs disappear. Php7/Postgres |
I seriously would never use the DB driver for queues in production. It is no intended to be a production solution. |
Now, that's interesting. I never knew that the DB driver was not a production solution. We're not processing millions of jobs, and it is nice to have a central repository for jobs without having to pay for Amazon. We can drop/spin up servers on a whim and all the jobs stay put in the db. |
@timothyallan ensure you restart your queues after pushing then new code. I experienced this same problem and in retrospect, I believe it was due to the old queue process still running. |
@jasonmccreary Yup this is all on docker, so every time I make a code change, everything is restarted. I still remember I got stung a year ago, for half a day with the old "queue processes still using old code" issue :) |
@timothyallan roger that. Regardless, I still believe there is a ghost in the code that seems to be appearing in 5.3. I've managed to tweak my app's code/configuration enough to continue to use the database driver. However, it does sound it is officially not recommended for production. |
To be fair, if failed jobs are unexpectedly lost to the ether, it's not stable enough for any environment :P |
Give me a fresh application that recreates that problem. |
That, I 100% understand. You need steps to reproduce 👍 , otherwise this conversation isn't going anywhere. |
@taylorotwell I work with @jesseleite and I have been able to reproduce a use case where jobs will not make it to the failed jobs table. Use Case
SummaryThe problem as far as I can tell is that the
If there is something wrong with the payload then an exception could be thrown. For example a In our case we were overriding functionality in Since an error gets thrown in the How to ReplicateThe following can be replicated on a fresh install of Laravel:
|
I believe @brianmclachlin's above PR fixes the disappearing failed jobs issue 👌 Maybe we can close this issue now? |
After hours of tests and research, I just removed the "Exception" word from the function and it worked for me. Laravel 5.4. Command "php artisan queue:work --tries=1" Before (not working):
After (working)
Could this be a bug? |
@fredsvanelli you probably had the wrong |
Just to keep this up to date, the Database driver is now acceptable for production use cases alongside MySQL 8.0: https://divinglaravel.com/a-production-ready-database-queue-diver-for-laravel |
Will need to investigate alternative queue drivers for production with multiple workers: laravel/framework#15192 (comment)
In Laravel 5.3, a Job will silently fail (no method invocation and nothing in the Laravel log) if you have not updated the jobs
failed()
method to receive anException
.For example, the following code is acceptable in Laravel 5.2, but will not be called in Laravel 5.3.
It needs to be updated to:
While I would not consider the update a bug, it was rather time consuming to track down since no error was reported. After determining the error, part of me thinks the unexpected method signature should throw an exception.
If nothing else, this was not listed in the Upgrade Guide, so opening this issue for future readers.
The text was updated successfully, but these errors were encountered: