Jobs randomly result in composer load class fail without even trying to process it

[JesterIruka]

• Laravel Version: 9.5.1
• PHP Version: 8.1.3
• Database Driver & Version: PGSQL (12.9)
• Queue driver: Redis (with 8 workers on queue "low")

Description:

Random instances of the same Job lead to composer autoload error. (Failed to open stream: No such file or directory).
I use pm2 to create the workers, the job doesn't even appear at the screen, just print directly in the storage/logs/laravel.log

Job code: https://prnt.sc/pCtZMRf8Er9f
(The backoff seems to work)

First error: https://prnt.sc/om8PCthikwEs
Second error (after some tries to fix it): https://prnt.sc/o7EIa2qW6wYO

Storage/logs: https://prnt.sc/P5CL-MkAl5jq

Steps To Reproduce:

Unfortunately I can't reference a repository.
The error seems too much random to replicate, sometimes I can run 20 jobs in a row flawless, I tried to test it on my local machine, but the WSL always give a curl timedout.

(The error also happens when I dispatch multiple empty jobs)
https://prnt.sc/7z9myHScPrTh
After running 3000 jobs on 8 workers, I got 8140 lines of logs of:

include(/home/ubuntu/backend/vendor/composer/../facade/ignition/src/QueryRecorder/Query.php): Failed to open stream: No such file or directory 

[driesvints]

Hi there,

Thanks for reporting but it looks like this is a question which can be asked on a support channel. Please only use this issue tracker for reporting bugs with the library itself. If you have a question on how to use functionality provided by this repo you can try one of the following channels:

• Laracasts Forums
• Laravel.io Forums
• StackOverflow
• Discord
• Larachat
• IRC

However, this issue will not be locked and everyone is still free to discuss solutions to your problem!

Thanks.

[JesterIruka]

Please, reconsider your response, since there isn't any class of my project in the stacktrace, I tried the discord before opening an issue, got nothing so far.

[2022-03-22 18:35:23] production.ERROR: include(/home/ubuntu/backend/vendor/composer/../facade/ignition/src/QueryRecorder/Query.php): Failed to open stream: No such file or directory {"exception":"[object] (ErrorException(code: 0): include(/home/ubuntu/backend/vendor/composer/../facade/ignition/src/QueryRecorder/Query.php): Failed to open stream: No such file or directory at /home/ubuntu/backend/vendor/composer/ClassLoader.php:571)
[stacktrace]
#0 /home/ubuntu/backend/vendor/composer/ClassLoader.php(571): Illuminate\\Foundation\\Bootstrap\\HandleExceptions->handleError()
#1 /home/ubuntu/backend/vendor/composer/ClassLoader.php(571): include()
#2 /home/ubuntu/backend/vendor/composer/ClassLoader.php(428): Composer\\Autoload\\includeFile()
#3 /home/ubuntu/backend/vendor/facade/ignition/src/QueryRecorder/QueryRecorder.php(41): Composer\\Autoload\\ClassLoader->loadClass()
#4 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(404): Facade\\Ignition\\QueryRecorder\\QueryRecorder->record()
#5 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(249): Illuminate\\Events\\Dispatcher->Illuminate\\Events\\{closure}()
#6 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Database/Connection.php(889): Illuminate\\Events\\Dispatcher->dispatch()
#7 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Database/Connection.php(732): Illuminate\\Database\\Connection->event()
#8 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Database/Connection.php(687): Illuminate\\Database\\Connection->logQuery()
#9 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Database/Connection.php(502): Illuminate\\Database\\Connection->run()
#10 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Database/Connection.php(454): Illuminate\\Database\\Connection->statement()
#11 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2980): Illuminate\\Database\\Connection->insert()
#12 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Queue/Failed/DatabaseUuidFailedJobProvider.php(64): Illuminate\\Database\\Query\\Builder->insert()
#13 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(214): Illuminate\\Queue\\Failed\\DatabaseUuidFailedJobProvider->log()
#14 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(161): Illuminate\\Queue\\Console\\WorkCommand->logFailedJob()
#15 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(404): Illuminate\\Queue\\Console\\WorkCommand->Illuminate\\Queue\\Console\\{closure}()
#16 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(249): Illuminate\\Events\\Dispatcher->Illuminate\\Events\\{closure}()
#17 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(195): Illuminate\\Events\\Dispatcher->dispatch()
#18 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(581): Illuminate\\Queue\\Jobs\\Job->fail()
#19 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(527): Illuminate\\Queue\\Worker->failJob()
#20 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(455): Illuminate\\Queue\\Worker->markJobAsFailedIfWillExceedMaxAttempts()
#21 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(432): Illuminate\\Queue\\Worker->handleJobException()
#22 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(378): Illuminate\\Queue\\Worker->process()
#23 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(172): Illuminate\\Queue\\Worker->runJob()
#24 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(117): Illuminate\\Queue\\Worker->daemon()
#25 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(101): Illuminate\\Queue\\Console\\WorkCommand->runWorker()
#26 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\\Queue\\Console\\WorkCommand->handle()
#27 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#28 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure()
#29 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\\Container\\BoundMethod::callBoundMethod()
#30 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Container/Container.php(653): Illuminate\\Container\\BoundMethod::call()
#31 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Console/Command.php(136): Illuminate\\Container\\Container->call()
#32 /home/ubuntu/backend/vendor/symfony/console/Command/Command.php(298): Illuminate\\Console\\Command->execute()
#33 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Console/Command.php(121): Symfony\\Component\\Console\\Command\\Command->run()
#34 /home/ubuntu/backend/vendor/symfony/console/Application.php(1015): Illuminate\\Console\\Command->run()
#35 /home/ubuntu/backend/vendor/symfony/console/Application.php(299): Symfony\\Component\\Console\\Application->doRunCommand()
#36 /home/ubuntu/backend/vendor/symfony/console/Application.php(171): Symfony\\Component\\Console\\Application->doRun()
#37 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Console/Application.php(94): Symfony\\Component\\Console\\Application->run()
#38 /home/ubuntu/backend/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(129): Illuminate\\Console\\Application->run()
#39 /home/ubuntu/backend/artisan(37): Illuminate\\Foundation\\Console\\Kernel->handle()
#40 {main}
"}

Edit: The composer autoload is dumped correctly, my backend works just fine, this is a random error that occour within the same job, depending on the instance and god knows what.

[JesterIruka]

(I forgot to mention you)
@driesvints

Edit: in the meanwhile, I'm processing my webhooks jobs with javascript, using a dedicated redis set key with scores, since this may take a couple of days to analyze.

[JesterIruka]

Probably related, the dispatch()->beforeCommit() doesn't work on the first time.
https://i.imgur.com/lGeuEhh.png

[JesterIruka]

Can you please review this issue? @driesvints

[mikeydevelops]

Maybe something failed while composer was dumping the classes?
Have you tried reinstalling composer dependencies i.e. delete vendor and running composer install?

[JesterIruka]

Maybe something failed while composer was dumping the classes? Have you tried reinstalling composer dependencies i.e. delete vendor and running composer install?

@mikeydevelops Yes, I did everything, install, dump autoload, everything you can possible imagine with composer, it's random as hell, sometimes < 30% of the jobs actually work, sometimes is around >= 90%, even without applying any update to the project, just regular dispatchs and queue retries.

[mikeydevelops]

Sorry for the dumb question, but do you restart your queue workers after you dump the autoload?

[JesterIruka]

Sorry for the dumb question, but do you restart your queue workers after you dump the autoload?

@mikeydevelops Also yes, since I use pm2 + octane, every worker is in the same namespace, when I restart the webserver, every worker also restart.

[JesterIruka]

Everything synchronized

Edit: (The scheduler is on the wrong namespace but it doesn't dispatch any sendwebhookjobs)

[mikeydevelops]

can you see currently, does vendor/facade/ignition/src/QueryRecorder/Query.php exist. does the whole folder exist vendor/facade/ignition?

[JesterIruka]

can you see currently, does vendor/facade/ignition/src/QueryRecorder/Query.php exist. does the whole folder exist vendor/facade/ignition?

It does not exists, only vendor/facade/ignition-contracts/
This (/vendor/facade/ignition) also doesn't exists on my local machine. But ignition-contracts does

[mikeydevelops]

Do you have any other projects running on the same server, maybe it is a memory leak where composer is using a different class loader?

[JesterIruka]

Do you have any other projects running on the same server, maybe it is a memory leak where composer is using a different class loader?

This is the only project running in PHP. Every other project is JavaScript, the server also has 32Gb, I'm currently using 2.65Gb