Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Messenger] Add a redis stream transport #30917

Merged
merged 2 commits into from
Apr 27, 2019

Conversation

alexander-schranz
Copy link
Contributor

@alexander-schranz alexander-schranz commented Apr 6, 2019

Q A
Branch? master
Bug fix? no
New feature? yes
BC breaks? no
Deprecations? no
Tests pass? Yes
Fixed tickets #28681
License MIT
Doc PR symfony/symfony-docs#11341

As discussed in #28681 this will refractor @soyuka implementation of redis using the redis stream features so we don't need to handle parking the messages ourself and redis is doing it for us.

Some interesting links about streams:

+-----------R
|    GET    | -> XREADGROUP
+-----------+
      |
      | handleMessage
      V
+-----------+  No
|  failed?  |---------------------------+
+-----------+                           |
      |                                 |
      | Yes                             |
      V                                 |
+-----------+  No                       |
|   retry?  |---------------------------+
+-----------+                           |
      |                                 |
      | Yes                             |
      V                                 V
+-----------R                     +-----------R
|   REJECT  | -> XDEL             |    ACK    | -> XACK
+-----------+                     +-----------+

GET: Will use XREADGROUP to read the one message from the stream
REJECT: Reject will just remove the message with XDEL from the stream as adding it back to the stream is handled by symfony worker itself
ACK: Will use the XACK Method to ack the message for the specific group

The sender will still be simple by calling the XADD redis function.

#EU-FOSSA

@curry684
Copy link
Contributor

curry684 commented Apr 6, 2019

Needs [Messenger] prefix ;)

@lyrixx lyrixx changed the title Add a redis stream transport [Messenger] Add a redis stream transport Apr 6, 2019
Copy link
Member

@chalasr chalasr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, I want it for 4.3 :)
The framework integration is missing, we need to register the redis transport factory in FrameworkBundle's messenger.xml (and add a test for it)

@alexander-schranz
Copy link
Contributor Author

@chalasr thank you for your review. did fix it and register the factory in the framework bundle.

@chalasr
Copy link
Member

chalasr commented Apr 25, 2019

Adding a test in

as for amqp would be nice.

@alexander-schranz alexander-schranz force-pushed the redis-messenger branch 4 times, most recently from d0b63af to 4ed23bd Compare April 25, 2019 22:59
Copy link
Member

@chalasr chalasr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried this on a real project (will allow me to drop enqueue/messenger-adapter 🎉 ), works just fine.
I think implementing MessageCountAwareInterface can be done later, not a blocker.

@fabpot
Copy link
Member

fabpot commented Apr 27, 2019

Thank you @alexander-schranz.

@fabpot fabpot merged commit ff0b855 into symfony:master Apr 27, 2019
fabpot added a commit that referenced this pull request Apr 27, 2019
…ander-schranz)

This PR was merged into the 4.3-dev branch.

Discussion
----------

[Messenger] Add a redis stream transport

| Q             | A
| ------------- | ---
| Branch?       | master
| Bug fix?      | no
| New feature?  | yes
| BC breaks?    | no
| Deprecations? | no
| Tests pass?   | Yes
| Fixed tickets | #28681
| License       | MIT
| Doc PR        | symfony/symfony-docs#11341

As discussed in #28681 this will refractor @soyuka implementation of redis using the redis stream features so we don't need to handle parking the messages ourself and redis is doing it for us.

Some interesting links about streams:

 - https://redis.io/topics/streams-intro
 - https://brandur.org/redis-streams

```
+-----------R
|    GET    | -> XREADGROUP
+-----------+
      |
      | handleMessage
      V
+-----------+  No
|  failed?  |---------------------------+
+-----------+                           |
      |                                 |
      | Yes                             |
      V                                 |
+-----------+  No                       |
|   retry?  |---------------------------+
+-----------+                           |
      |                                 |
      | Yes                             |
      V                                 V
+-----------R                     +-----------R
|   REJECT  | -> XDEL             |    ACK    | -> XACK
+-----------+                     +-----------+
```

**GET**: Will use `XREADGROUP` to read the one  message from the stream
**REJECT**: Reject will just remove the message with `XDEL` from the stream as adding it back to the stream is handled by symfony worker itself
**ACK**: Will use the `XACK` Method to ack the message for the specific group

The sender will still be simple by calling the `XADD` redis function.

#EU-FOSSA

Commits
-------

ff0b855 Refractor redis transport using redis streams
7162d2e Implement redis transport
@alexander-schranz alexander-schranz deleted the redis-messenger branch April 27, 2019 17:09
@soyuka
Copy link
Contributor

soyuka commented Apr 28, 2019

Nice work thanks @alexander-schranz !

@alexander-schranz
Copy link
Contributor Author

@soyuka thank you for initial work!

@nicolas-grekas nicolas-grekas modified the milestones: next, 4.3 Apr 30, 2019
@fabpot fabpot mentioned this pull request May 9, 2019
wouterj added a commit to symfony/symfony-docs that referenced this pull request May 11, 2019
…chranz)

This PR was submitted for the master branch but it was squashed and merged into the 4.3 branch instead (closes #11341).

Discussion
----------

Add documentation for the Redis transport

This will add documentation how to configure and using the redis transport with the messenger component.

symfony/symfony#30917

#EUFOSSA

Commits
-------

c22fade Add documentation for the Redis transport
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.