"batch" processing at the end of connection: does Haraka allow for this?

[ThomasEhardt-smarsh]

My use case is to use Haraka to receive archivable emails and then upload them to S3 for additional processing. Assuming we have this workflow:

1. connection made to Haraka
2. messages uploaded to Haraka by caller
3. messages uploaded to S3 (asynchronously)
4. client disconnect
5. after all messages uploaded to S3 kick off some process on all messages (e.g. creation of a manifest, reconciliation process, etc.)

Now the last step can happen after the client disconnects or before; that is something that is still being discussed, but the gist is thatI want to take an action on all of the messages at once at some point.

I do not see a hook for such a scenario, so my question is whether or not Haraka has a means to do this. I imagine it can be done via standard node code, but ideally, there is something that Haraka exposes to do the same.

TIA

[msimerson]

Yes.

You can do this with a plugin. Your plugin would need to validate the recipient. When it does so, for the messages that want future uploading to S3, set the queue.wants property:

• connection.transaction.notes.set('queue.wants', 'myPlugin')

Assure your plugin hooks both queue and queue_outbound. Then you're assured that your plugin will handle delivery. You can employ any logic you wish within the queue hook. You probably want something like this:

// myPlugin
exports.register = function () {
  this.register_hook('queue', 'my_queue_function')
  this.register_hook('queue_outbound', 'my_queue_function')
}
exports.my_queue_function = function (next, connection) {
  // do your magic. Queue the message to disk, rabbitmq, mongodb, etc..
}
exports.hook_disconnect = function (next, connection) {
  // the connection is ending, upload all the queued messages to S3
}

[ThomasEhardt-smarsh]

this seems to just allow the messages to be handled one after another synchronously; i'm looking for something that happens on ALL of the messages received at once, once they are all handled

e.g. my_queue_function writes the files to disk, then after all of the files from the session are written to disk, take an action on all of them (e.g. bulk upload to a bucket). In other words, I would like to know

• when all the messages in a connection have been processed (say, written to disk)
• then take some action on them such as bulk upload to a bucket

[msimerson]

SMTP connections can have many transactions, and each transaction is a message and each message can have many recipients. So, you need to queue every message from each transactions. After the client signals bye, the disconnect hook will fire and that can be your trigger to do the S3 upload. Please read Plugins.md