Asynchonrously send to sendmail?

[JasonBarnabe]

As I understand it, the sendmail delivery method uses popen, which is synchronous, blocking until the process completes. While some sendmail interfaces are asynchronous (queuing the message to be delivered later), others like postfix are not, so the process only completes once the mail has been received by the remote server.

We send a lot of large emails so we can see multi-second waits for the sendmail command to complete. Even though these emails are being processed by a sidekiq process and not a web server, it's still a significant delay to the sidekiq process.

What if this gem asynchronously ran the sendmail command instead (as a configuration option or at least a monkeypatch)? It seems like it would not be able to raise an exception if the command fails. Are there any other drawbacks?

[zarqman]

@JasonBarnabe Correct me if I'm missing something, but hasn't Sidekiq already effectively made this async for you? It sounds like you've already done the right thing by moving mail delivery to a background job.

If mail had its own async queue, then if the Ruby process dies or is shutdown for a deploy, you'd lose email. For reliable delivery, mail would need to persist those messages somewhere, handle retries, inform you of permanent failures, and make it all observable with metrics. That sounds quite a bit like a background queue system. 😄

Sidekiq does a great job of handling retries and exceptions and has a persistence layer, so it sounds like a good choice for all of that. (It is worth remembering though that Sidekiq's free edition doesn't do particularly well with handling Ruby shutdowns. Other queues address that better, but usually at a cost of lower overall throughput.)

As you've already noted, you could also switch to a sendmail-compatible interface that queues mail itself. (Or reconfigure postfix to do so, if possible.)

[JasonBarnabe]

It's async from the perspective of the original web request that triggered the action. It's synchronous from the perspective of a sidekiq job: the job will not complete until it's gone (in our case) to postfix and gotten a response from the remote server. Like I said, we send a lot of email and we have a significant percentage of sidekiq time waiting on the I/O from the remote server.

I'm not familiar with the behaviour of the various ways to run a system command. If sendmail was called with spawn/fork/detach, does the Ruby process dying stop the sendmail command? Would a sidekiq shutdown affect it? A system shutdown?

[JasonBarnabe]

Note I'm not proposing/asking about mail having its own queue - just that it spawns async threads/processes whatever for the sendmail calls.

[j1wilmot]

The mail gem is responsible for delivering a message to an MTA. It makes sense for this delivery to be synchronous so that you know if there are errors sending data to the MTA. The MTA is responsible for handling mail delivery (queueing etc). Is Sendmail's delivery mode set to queue? See man page on delivery mode.

[JasonBarnabe]

While some sendmail interfaces are asynchronous (queuing the message to be delivered later), others like postfix are not, so the process only completes once the mail has been received by the remote server.

It seems like it would not be able to raise an exception if the command fails. Are there any other drawbacks?

[zarqman]

@JasonBarnabe Are you positive postfix cannot enable an outbound mail queue? A super quick search shows articles discussing how to unstick a message in the outbound queue, which suggests that postfix has one.

Otherwise, if you can afford the extra processes that spawn/fork would require, why not just bump up the number of Sidekiq threads? Those would be lighter-weight on your servers than forking anyway.

[JasonBarnabe]

I am not positive that postfix cannot run asynchronously, but from what I've read, the normal sendmail way of doing it (using the DeliveryMode argument/setting) does not work with postfix. I have also added logging to this gem around the call to sendmail which shows this can often take several seconds, but I'm not 100% sure it's the normal operation of sendmail that is causing the delay and not something else.

Our bottleneck is memory usage. I would assume that spawning/forking the sendgrid command would be less memory intensive then a whole other sidekiq thread.

[sebbASF]

Seems to me that this is out of scope for this Gem.

[zarqman]

@JasonBarnabe Spawning/forking would likely make your memory usage worse, not better. It would also reduce observability into what's going on, which I'd suggest is a net negative when you're already resource constrained.

I think your best bet is to address this by one of: a) changing your Postfix config to queue first, deliver second; b) replacing Postfix with another SMTP server that can take on the queuing; c) moving to a 3rd party service like AWS SES, Sendgrid, Sparkpost, Mailgun, or Postmark; or (d) upgrading your servers.

BTW, some of the multi-second delays in sending you're seeing are, some of the time, intentional on the part of the destination SMTP servers. It's a common tactic to make spamming more expensive by consuming sender resources. 😄 For legit mail, this is just a cost of doing business.

As others have mentioned, this is out of scope for the mail gem, so perhaps you'd consider closing this Issue?

[JasonBarnabe]

I don't understand how considering the way mail invokes sendmail is "out of scope" given that it already invokes sendmail today, but I can understand how this could be a bad idea at least in the general sense, so closing.