Host via Laravel Vapor

[viemic]

Hi,

we have all our services running via Laravel Vapor.

Since the standalone-app specifically is depending on Horizon I am wondering which steps are necessary to host Mailcoach through Vapor as well.

Do any of you have ideas to accomplish this case?

[freekmurze]

Currently this is not supported. Mailcoach requires Horizon and Horizon is not easy to use on Vapor.

[lahivee]

Sorry to pick this up again, but I would also be really interested in this as the company I work for also uses Vapor only for Laravel projects

[lahivee]

@freekmurze do you mind giving a quick outline what is the way to achieve this. Possibly swapping Redis/Horizon with SQS

[freekmurze]

I'm thinking that the biggest hurdle to take would be rate limiting. You need to be able to rate limit jobs so that you don't go over the sending limits set by the email service providers.

Currently the package use this package to rate limit: https://github.com/spatie/laravel-rate-limited-job-middleware

The package relies on a Redis specific function to throttle stuff.

In order to make Mailcoach run on Vapor you'll need to remove that functionality and make sure you rate limit in some other way.

Like already said above, Mailcoach does support Vapor, but you're free to take a stab at making it run. Should you succeed, let me know. If it's easy to make it work on Vapor, I'd consider making the necessary adjustments and support it officially.

[lahivee]

Thanks for the elaboration! I now see where the issue is.

I'll try to play around with possible solutions and see if I can get something to work.

One solution I could think of is by setting a default rate limit that fits all of the supported mail providers and remove the dependency to Redis/horizon all together.
This default rate could be very generous (in terms of high) since the person sending the email will most likely not care if sending out a campaign takes 5, 10, 20 or 30 seconds.

Is this a viable though to build upon @freekmurze ?

[freekmurze]

Sounds good, try it out, and let us know how it went! 👍

[ziming]

Hmm personally I "rate limit" by doing something hacky like this

$now = Carbon::now();

foreach ($items as $item) {
    $when = $now->addMilliseconds(200); // addMicroseconds, addSeconds.etc
    $item->notify((new Reminder)->delay($when));
}

[Francois-Francois]

@ziming I think you should do (I may be wrong) :

$now = Carbon::now();
$when = $now->addMilliseconds(200);

foreach ($items as $item) {
    $when = $when->addMilliseconds(200); // addMicroseconds, addSeconds.etc
    $item->notify((new Reminder)->delay($when));
}

//or

$now = Carbon::now();

foreach ($items as $i => $item) {
    $when = $when->addMilliseconds(200 * $i); // addMicroseconds, addSeconds.etc
    $item->notify((new Reminder)->delay($when));
}

//or

foreach ($items as $item) {
    $when = now()->addMilliseconds(200); // addMicroseconds, addSeconds.etc
    $item->notify((new Reminder)->delay($when));
}

[ziming]

@rikless Thanks for the suggestions

Carbon is mutable so I think my approach is still fine. Unless you did something like this tweet to force CarbonImmutable

https://twitter.com/taylorotwell/status/1047578025263607809?lang=en

Your 3rd approach may have a problem as now() in 2nd iteration is probably just nanoseconds away from now() in 1st iteration.

But I think your code, which appears to assume Carbon is Immutable is less confusing :)

[sandervanhooft]

Has anyone tried this rate limiting approach yet? I'm still on the fence regarding Vapor - if Mailcoach can't run on it Vapor remains a no go.

[viemic]

@sandervanhooft unfortunately no, but maybe I just lack knowledge on how to do this. In the meanwhile I keep using Mailchimp and hope someone smart is able to conquer this obstacle

[lahivee]

I have done a few experiements in my limited time but without success. However, I can't imagine that it is too hard to achieve. Did anyone else have success on Vapor migration?

Same as @viemic I am also using Mailchimp for now

[nsouto]

Hello everyone,

So I just got Mailcoach working on Vapor, the package version I didn't try out the full app one yet, I believe the installed version uses file based storage for some settings and that would be a problem without updating that to use S3 or a DB/cache for storage.

Since Mailcoach package relies on Laravel to provide all the configuration, small adaptations to the config file make it easy to use env vars to make it easy to change configs on the fly (just by redeploying it)

To be able to use both a dev / prod / staging whatever envs at the same time this is needed:

    'perform_on_queue' => [
        'calculate_statistics_job' => env('MAILCOACH_CALCULATE_STATISTICS_JOB_QUEUE', 'mailcoach'),
        'send_campaign_job' => env('MAILCOACH_SEND_CAMPAIGN_JOB_QUEUE', 'send-campaign'),
        'send_mail_job' => env('MAILCOACH_SEND_MAIL_JOB_QUEUE', 'send-mail'),
        'send_test_mail_job' => env('MAILCOACH_SEND_TEST_MAIL_JOB_QUEUE', 'mailcoach'),
        'process_feedback_job' => env('MAILCOACH_PROCESS_FEEDBACK_JOB_QUEUE', 'mailcoach-feedback'),
    ],

This will allow you to have multiple ses configurations per env as well:

    'ses_feedback' => [
        'configuration_set' => env('MAILCOACH_SES_CONFIGURATION_SET', 'mailcoach'),
    ],

The current issue is with file uploads which need to go into S3 so for now I'm relying on the text/monaco editors and looking into options for that so I'm not reinventing the wheel. I do believe it should be possible to get the current Mailcoach adapted to use it, the problem for example relies on uploading a subscriber import which will be processed by a job, but at that point the local uploaded file no longer exists in the new container running the Job.

On the vapor.yml file you basically need this:

    scheduler: true
    runtime: php-7.4
    queue: false
    queue-timeout: 600
    queue-concurrency: 50
    queues:
      - default
      - send-campaign
      - send-mail
      - mailcoach-feedback
      - mailcoach

Adapting each queue name to the proper environment and whatever you setup on the env vars.

The queue-concurrency, I still need to run more tests but that's basically how many jobs you want to run at the same time, together with Mailcoach's Redis throttling you can get it under control for your own account limits whether using with SES or another sender, at that time it's about math.

Also make sure you have a Redis cache on it, as Mailcoach depends on that, Vapor by default will use a dynamoDB cache but it won't work well with the built in throttling, didn't think about whether or not this would be a possibility to update within Mailcoach, but a Redis cache works well although a bit more costly.

The one quick update needed on Mailcoach is to make sure to use the asset() helper for any static images so they load through the S3 bucket automatically (eg the horse when sending isn't working)

For now I was able to get it working, except for the one big thing that needs to be solved which is the uploads, it's not difficult but int the Mailcoach context would have to be solved universally no matter where it's running.

I hope this clears it out, I'm available if anyone has any questions or difficulty with running it on Vapor, I can look into getting the full application running in there as well, but hopefully this will help and together we can make this better.

[freekmurze]

@nsouto Thank you so much for your research;

If you have the time for it, I'd appreciate a diff with non breaking changes, that add these env helpers to the config file, together with other pain points you solved.

The one quick update needed on Mailcoach is to make sure to use the asset() helper for any static images so they load through the S3 bucket automatically (eg the horse when sending isn't working)

I'll take a look soon at where we don't use that asset() helper and add it. Again, if you have already fixed this and have the time, I'd appreciate a diff with the necessary changes.

For now I was able to get it working, except for the one big thing that needs to be solved which is the uploads, it's not difficult but int the Mailcoach context would have to be solved universally no matter where it's running.

I think on the short I'll disable uploads when we detect that we are on Vapor. In a later release we'll make sure that uploads work on (via S3) on Vapor too.

The queue-concurrency, I still need to run more tests but that's basically how many jobs you want to run at the same time, together with Mailcoach's Redis throttling you can get it under control for your own account limits whether using with SES or another sender, at that time it's about math.

Did you set up Redis on Vapor? My thinking is that on ideally we don't need this when using mailcoach on Vapor. I think on AWS it's possbible to set throttling on the SQS queue itself.

[WDC]

When using Mailcoach with Laravel Vapor, we are running into the following error:

Redis connection [horizon] not configured.

We are deploying to AWS Lambda and AWS Elasticache (Redis) with the following config/connections.php configuration:

‘redis’ => [
        'default' => [
            'url' => env('REDIS_URL'),
            'host' => 'redis.redacted.cache.amazonaws.com',
            'password' => env('REDIS_PASSWORD', null),
            'port' => env('REDIS_PORT', 6379),
            'database' => env('REDIS_DB', 0),
        ],
…
]

And this config/queue.php:

‘connections’ => [
        'horizon' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue' => env('REDIS_QUEUE', 'default'),
            'retry_after' => 90,
            'block_for' => null,
        ],
...
]

The error message is very odd, because the horizon connection is most certainly defined, and when running the same code locally (without Vapor, of course), the connection is found and I can manipulate the Elasticache resource as expected. Only when being deployed via Vapor does Laravel complain about the connection's configuration.

Any help would be greatly appreciated!

@nsouto we are running the full Laravel Vapor stack, not just the package. Do you have any insight into what might cause the Redis connection [horizon] not configured.? I do not believe that S3 is involved in the error message that we are seeing; the campaigns that we are sending do not include images or attachments (at the moment).

[freekmurze]

Mailcoach now supports Vapor. Take a look at this discussion for more info: https://github.com/spatie/laravel-mailcoach/discussions/454