php artisan queue:retry fails

[kajgies]

Bug description

Hey everyone,

Whenever I have a failed job, calling php artisan queue:retry all fails.

The error message I am getting is:

>php artisan queue:retry all

   InvalidArgumentException 

  Database connection [tenant] not configured.

  at \vendor\laravel\framework\src\Illuminate\Database\DatabaseManager.php:152
    148▕         // If the configuration doesn't exist, we'll throw an exception and bail.
    149▕         $connections = $this->app['config']['database.connections'];
    150▕
    151▕         if (is_null($config = Arr::get($connections, $name))) {
  ➜ 152▕             throw new InvalidArgumentException("Database connection [{$name}] not configured.");
    153▕         }
    154▕
    155▕         return (new ConfigurationUrlParser)
    156▕                     ->parseConfiguration($config);

  1   \vendor\laravel\framework\src\Illuminate\Database\DatabaseManager.php:115
      Illuminate\Database\DatabaseManager::configuration("tenant")

  2   \vendor\laravel\framework\src\Illuminate\Database\DatabaseManager.php:86
      Illuminate\Database\DatabaseManager::makeConnection("tenant")

Steps to reproduce

Setup an empty Laravel project with this package.
Create a job.
Make sure the job fails.
Try running php artisan queue:retry all

Expected behavior

Calling php artisan queue:retry all should properly retry the job.

Laravel version

8.75.0

stancl/tenancy version

3.4.5

[stancl]

Can you show the code of the job?

[stancl]

This should be fixed in 73a4a30. Could you try running composer require stancl/tenancy:3.x-dev to see if the change fixes this?

[stancl]

Fixed in 73a4a30

[evilchis94]

I'm still having the issue in stancl/tenancy version 3.5.0 and Laravel 8.38.0.

My config is multi database. I have QueueTenancyBootstrapper and DatabaseTenancyBootstrapper in tenancy.php config.

I have forced the database queue driver to use the central connection with

'connection' => 'central',

in config/queue.php

Note that my job got dispatched from the tenant context.

I don't know what else to do. Thanks!

[stancl]

Can you show the code of the job?

[evilchis94]

Sure

<?php

namespace App\Jobs;

use App\Models\Tenant\Order;
use App\Models\Tenant\Route;
use App\Models\Tenant\RouteRegistry;
use App\Models\Tenant\Visit;
use App\StateMachines\OrderStatusStateMachine;
use Carbon\Carbon;
use Exception;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldBeUnique;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Str;

class RoutePendingVerificationOrder implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    /**
     * The order instance.
     *
     * @var \App\Models\Tenant\Order
     */
    protected $order;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct(Order $order)
    {
        $this->order = $order;
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        $this->order->load('customerAddress.suburb.zipcode');

        $zipcode = null;

        // Obtenemos un arreglo de zipcodes unicos de todos las ordenes
        if ($this->order->customerAddress) {
            $zipcode = $this->order->customerAddress->suburb->zipcode->name;
        }

        if (is_null($zipcode)) {
            $this->fail(new Exception('La dirección del contrato no tiene colonia', 500));
        }

        $centralDatabase = config('tenancy.database.central_db_name');

        $route = Route::where('route_type_id', 2)->whereHas('zones.zipcodes', function ($query) use ($zipcode, $centralDatabase) {
            $query->from("{$centralDatabase}.zipcodes")->where('name', $zipcode);
        })->with('zones.zipcodes')->first();

        if (is_null($route)) {
            $this->fail(new Exception('No hay rutas que tengan ligadas una zona con el CP correspondiente.'));
        }

        $routeRegistry = RouteRegistry::where('route_id', $route->id)->whereDate('date', Carbon::now())->first();

        if (empty($routeRegistry)) {
            $routeRegistry = RouteRegistry::create([
                'routeId' => $route->id,
                'date' => Carbon::now(),
                'userId' => $route->inchargeUserId,
                'transportId' => $route->transportId
            ]);
        }

        $visit = new Visit();
        $visit->routeRegistryId = $routeRegistry->id;
        $visit->creatorId = $route->inchargeUserId;
        $visit->date = Carbon::now();
        $visit->uniqueId = (string) Str::uuid();
        $visit->visitTypeId = 3;
        $visit->orderId = $this->order->id;
        $visit->customerId = $this->order->customerId;
        $visit->saveQuietly();

        $this->order->status()->transitionTo(OrderStatusStateMachine::VERIFIED_PENDING);

        // TODO send push notification
    }

    public function fail($exception = null)
    {
        Log::error('job failed');
    }
}

[stancl]

Hmm, could I see your:

• queue connection config
• database connections (the ones you're using)
• and the exact code that dispatches this?

And does the issue happen on queue:work, or only on queue:retry?

[evilchis94]

config/queue.php

'database' => [ 'driver' => 'database', 'table' => 'jobs', 'queue' => 'default', 'retry_after' => 90, 'after_commit' => false, 'connection' => 'central' ]

config/database.php

'connections' => [
     'central' => [
            'driver' => 'mysql',
            'url' => env('CENTRAL_DATABASE_URL'),
            'host' => env('CENTRAL_DB_HOST', '127.0.0.1'),
            'port' => env('CENTRAL_DB_PORT', '3306'),
            'database' => env('CENTRAL_DB_DATABASE', 'forge'),
            'username' => env('CENTRAL_DB_USERNAME', 'forge'),
            'password' => env('CENTRAL_DB_PASSWORD', ''),
           ... (same as defaults)
        ],

        'mysql' => [
            'driver' => 'mysql',
            'url' => env('DATABASE_URL'),
            'host' => env('DB_HOST', '127.0.0.1'),
            'port' => env('DB_PORT', '3306'),
            'database' => env('DB_DATABASE', 'forge'),
            'username' => env('DB_USERNAME', 'forge'),
            'password' => env('DB_PASSWORD', ''),
          ... (same as defaults)
        ] 
],

.env

CENTRAL_DB_HOST=mysql
CENTRAL_DB_PORT=3306
CENTRAL_DB_DATABASE=****
CENTRAL_DB_USERNAME=****
CENTRAL_DB_PASSWORD=****

DB_CONNECTION=central
DB_HOST=mysql
DB_PORT=3306
DB_DATABASE=****
DB_USERNAME=****
DB_PASSWORD=****

The code that dispatches the job is a StateMachine linked to a model property (status). When I transition the status some hooks are triggered.

In some controller:

$order->status()->transitionTo(OrderStatusStateMachine::PENDING_VERIFICATION);

In OrderStatusStateMachine.php

<?php

namespace App\StateMachines;

use App\Jobs\RoutePendingVerificationOrder;
use App\Models\Tenant\Order;
use Asantibanez\LaravelEloquentStateMachines\StateMachines\StateMachine;
use Carbon\Carbon;

class OrderStatusStateMachine extends StateMachine
{

.......

public function afterTransitionHooks(): array
    {
        return [

            self::PENDING_VERIFICATION => [
                function ($from, $model) {
                    if ($from === self::PROSPECT && $model->order_origin_id == 1) {
                        // -------------- Dispatch the job
                        RoutePendingVerificationOrder::dispatch($model); 
                    }
                }
            ],
        ];
    }
}

The issue happens only on queue:retry

[stancl]

Hmm, that's strange. Could you add some dd()s to the QueueTenancyBootstrapper to see what code gets executed? Specifically if this listener is executed, and what the initializeTenancyForQueue() method does afterwards.

Also, what makes the job fail?

[stancl]

I can reproduce the issue in 3.4.6, with 3.5.0 fixing it.

➜ samuel@MacBook-Pro  ~/Sites/queue-test  art queue:retry 3f1943c8-2c50-4730-9eb7-6e393a742565

   InvalidArgumentException 

  Database connection [tenant] not configured.

  at vendor/laravel/framework/src/Illuminate/Database/DatabaseManager.php:152
    148▕         // If the configuration doesn't exist, we'll throw an exception and bail.
    149▕         $connections = $this->app['config']['database.connections'];
    150▕ 
    151▕         if (is_null($config = Arr::get($connections, $name))) {
  ➜ 152▕             throw new InvalidArgumentException("Database connection [{$name}] not configured.");
    153▕         }
    154▕ 
    155▕         return (new ConfigurationUrlParser)
    156▕                     ->parseConfiguration($config);

      +12 vendor frames 
  13  [internal]:0
      App\Jobs\TestJob::__unserialize()

      +16 vendor frames 
  30  artisan:37
      Illuminate\Foundation\Console\Kernel::handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

➜ samuel@MacBook-Pro  ~/Sites/queue-test  composer update 
Loading composer repositories with package information
Updating dependencies
Lock file operations: 0 installs, 18 updates, 0 removals
...
  - Upgrading stancl/tenancy (v3.4.6 => v3.5.0)
...
Use the `composer fund` command to find out more!
> @php artisan vendor:publish --tag=laravel-assets --ansi --force
No publishable resources for tag [laravel-assets].
Publishing complete.


➜ samuel@MacBook-Pro  ~/Sites/queue-test  art queue:retry 3f1943c8-2c50-4730-9eb7-6e393a742565
The failed job [3f1943c8-2c50-4730-9eb7-6e393a742565] has been pushed back onto the queue!

➜ samuel@MacBook-Pro  ~/Sites/queue-test  art queue:work 
[2022-01-05 11:46:33][10] Processing: App\Jobs\TestJob
[2022-01-05 11:46:33][10] Failed:     App\Jobs\TestJob

➜ samuel@MacBook-Pro  ~/Sites/queue-test  art queue:failed                                    
+--------------------------------------+------------+---------+------------------+---------------------+
| ID                                   | Connection | Queue   | Class            | Failed At           |
+--------------------------------------+------------+---------+------------------+---------------------+
| 3f1943c8-2c50-4730-9eb7-6e393a742565 | database   | default | App\Jobs\TestJob | 2022-01-05 11:46:33 |
+--------------------------------------+------------+---------+------------------+---------------------+

➜ samuel@MacBook-Pro  ~/Sites/queue-test  art queue:retry 3f1943c8-2c50-4730-9eb7-6e393a742565
The failed job [3f1943c8-2c50-4730-9eb7-6e393a742565] has been pushed back onto the queue!

[evilchis94]

Hey @stancl: it was Laravel version.

The JobRetryRequested listener was added in version 8.64.0 and my previous version was 8.38.0, so the listener wasn't executed.

Thank you so much for this awesome library you made and for your quick replies! Sorry for the inconvenience this may have caused you.

[stancl]

No worries, appreciate that you tested it! It was a bit half-blind debugging for me as I haven't encountered this myself before partially reproducing it in some tests, so the feedback is helpful.