New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC][Messenger] Buffered Message Bus Implementation Considerations #29157

Closed
kiler129 opened this Issue Nov 9, 2018 · 4 comments

Comments

Projects
None yet
5 participants
@kiler129
Contributor

kiler129 commented Nov 9, 2018

Intro

First of all let me start by saying I'm creating this issue as a followup of #28983

Background

Currently in our application we batch-process large sets of records. Some of the operations (e.g. geocoding or creation of thumbnails) are very resource-intensive and slow but not crucial for the application functionality. For these reasons every time we process a record we dispatch various messages handled by separate servers which execute these slow operations asynchronously.

However, there is a catch - none of the operations can be executed before records land in a database. We dispatch messages with IDs of entities, so that they are small and each worker fetches the records and processes it further. In order to make speed up the processing we flush data into database in chunks. This causes a problem where message is dispatched into a queue and entity is still sitting in memory waiting to be flush()-ed.

Solution

The solution we developed was very simple - BufferedMessageBus. How it works it simply composes real bus and stores all messages passed on dispatch() in memory cache. It adds additional flush() which when called does RealBus::dispatch() on every message.

BC consequence

Currently the return signature of dispatch() was changed from void to Envelope. This really does not allow buffering of messages, since they are not immediately processed by middlewares.

I cannot think about any sensible implementation of delayed-dispatch pattern here if dispatch() is to return Envelope.

  1. Am I doing something horribly wrong?
  2. Maybe the returned Envelope should be optional?
  3. Any thoughts?

ping @nicolas-grekas @sroze

@nicolas-grekas

This comment has been minimized.

Member

nicolas-grekas commented Nov 9, 2018

I don't understand the conflict: what prevents you from returning the passed Envelope and be done with it?

@nicolas-grekas

This comment has been minimized.

Member

nicolas-grekas commented Nov 10, 2018

So, I found the discussion on Slack about that.
The Envelope can be used to tell what happened behind dispatch.
If you buffered it, that's perfectly legal to return the envelope you just received as argument.
At this moment, there is nothing more to tell about routing.
The key point is that if the caller expects a specific stamp in return then it may be incompatible with buffering. But that's not at all an issue with returning an Envelope: that's just natural, envelope or not. Actually, Envelope allows deciding these situations if one wants to build the infrastructure to do so in their code.
You could even stamp the Envelope with your own BufferedStamp if that makes sense.

So, TL;DR, this is a misunderstanding to me :)

@Tobion

This comment has been minimized.

Member

Tobion commented Nov 17, 2018

I also don't see why returning Envelope should not allow you to do the same buffering as before.

@Tobion Tobion closed this Nov 17, 2018

@sroze

This comment has been minimized.

Member

sroze commented Nov 18, 2018

@kiler129 did these answer make sense to you? I agree with @nicolas-grekas & @Tobion, this has nothing to do with the return value indeed.

Also, my 5cts regarding your use-case would be to say that you should write your own middleware for this, not another implementation of the message bus.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment