Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Messenger][RFC] Improve Messenger to support other app consuming/recieving the message #33912

Open
tristanbes opened this issue Oct 8, 2019 · 12 comments

Comments

@tristanbes
Copy link
Contributor

@tristanbes tristanbes commented Oct 8, 2019

Description

Hello,

I'm a new user of Messenger; Thank you to all contributors of this promising component; Yet, it gave me a hard time to solve some basic problematics.

Example

Let's say you have 2 different Symfony applications.

The application A (big, monolithic) is responsible of publishing the message to the queue using messenger. Until here, it's ok, everything is covered by the doc.

Then the queue (Amazon SQS/SNS) triggers a new Lambda execution, passing the message to the function (which is a SF Application B, responsible of generating a PDF); This behaviour is specific to Amazon, but it can be generalized of "messages can be pulled or pushed".

bP6zZi8m48HxFyKerL7m0XGeeJwA8nNbGZOXuhms_k4kz_Gi5CW297IDLvPtlfdrrZ5HIDb-O-k3-yKgnr4uHcpOkv2YgTh9zaucDDwZgJ1Vecei_Jb1-zLM2xGitIW3hAfeO5m10zPwOnau6DgdyKo9yK0ofkJSk9zaTnsXpw8QVVWe5QlP2xFa2luxkJg-Qy7yQRxx9D3IHUneVopySSgS6MCFaYaThk5-WjFpGDwpH6DC

The problematic

A. On all examples on the doc, you assume that the consumer is the application that put the message in the queue itself;

But what if it's not the case ? What if it is another Symfony (smaller) project, responsible of generating the PDF only ? What if it's another language ? (JS ?).

B. The Messenger component sends the messages with some namespace inside;

So if your message class lives in App\AwesomeBundle\Message\PdfMessage in application A, and App\Message\PdfMessage in application B, then application B won't be able to decode the Message;

C. What if the message is not pulled by the messenger component itself, but somhow is recieved (pull vs pushed) ?


I achieved this scenario, and here's what helped me:

Do you think some of these problematic are generic enough to be covered by the component code or by the documentation ?

Thank you :)

@tristanbes tristanbes changed the title [Messenger][RFC] Improve the component to handle separate application [Messenger][RFC] Improve the component to support other app consuming/recieving the message Oct 8, 2019
@tristanbes tristanbes changed the title [Messenger][RFC] Improve the component to support other app consuming/recieving the message [Messenger][RFC] Improve Messenger to support other app consuming/recieving the message Oct 8, 2019
@dunglas

This comment has been minimized.

Copy link
Member

@dunglas dunglas commented Oct 9, 2019

Isn't the Symfony Serializer integration enough for these cases? When using Serializer, the message is encoded in JSON, and can easily be decode by any app able to deal with JSON.

@weaverryan

This comment has been minimized.

Copy link
Member

@weaverryan weaverryan commented Oct 10, 2019

We’re covering this in the coming days in SymfonyCasts:

A) symfony Serializer for turning messages into JSON if the worker will be another app. That’s one line of config.

B) custom Messenger transport Serializer is your app is consuming messages that originate from another app. That’s one class and one line of config (and if you want, you can use the Serializer component from within the this custom transport Serializer).

Could the problem be just a lack of documentation to point this out?

Thanks for the convo!

@ragboyjr

This comment has been minimized.

Copy link
Contributor

@ragboyjr ragboyjr commented Oct 10, 2019

So, I actually was able to successfully get messenger working with magento 1.x to a symfony application over redis. Both applications send and receive messages from the other.

What we ended up doing was just create a third repository for storing our shared message classes between the two, and then installed the repository as a git submodule and used composer path type repositories to include into each of the applications.

I think using shared classes is safer than maintaining two separate versions of the class, but, you still have to be careful that any message class changes are done in a backwards compatible way to remove the tight coupling of those apps being deployed together.

@Matth--

This comment has been minimized.

Copy link

@Matth-- Matth-- commented Oct 10, 2019

We have an application that is synced up with a .net process. We use a custom Messenger Serializer (described by @weaverryan) for inbound and outbound messages.

@rjwebdev

This comment has been minimized.

Copy link
Contributor

@rjwebdev rjwebdev commented Oct 10, 2019

I am struggling with this issue myself currently at work where I have to publish messages onto a queue where another agency will pick up the messages.

I tried to create a small package that contains interfaces which would be implemented by the classes that will be published so both of us would end up installing this package. In my messenger config, I bound the interfaces to the different queues but this did not work since the class names were added to the envelope and not the interface names.

@pounard

This comment has been minimized.

Copy link
Contributor

@pounard pounard commented Oct 10, 2019

@rjwebdev

but this did not work since the class names were added to the envelope and not the interface names

That's one of my main concern with messenger (and symfony/serializer component as well): you cannot give a map of names or aliases that would map to PHP class. In a project we wrote this: https://gist.github.com/pounard/25d2a8d4502a6bb61e660957196a5773 (which is then filled with an alias to PHP class array map, from a custom bundle configuration).

Then you specify your messages with business oriented names and not technical ones. One missing bit in my example code is that we also have a custom messenger's serializer that maps the names the other way around to send them instead of PHP class names (PHP classes got converted to business name when published, then converted back into PHP class names when consumed).

@pounard

This comment has been minimized.

Copy link
Contributor

@pounard pounard commented Oct 10, 2019

@ragboyjr

I think using shared classes is safer than maintaining two separate versions of the class, but, you still have to be careful that any message class changes are done in a backwards compatible way to remove the tight coupling of those apps being deployed together.

If you have more than two applications (currently we have only one but we are targeting a client environment where we'll have at least 5, each with their own maintainers and release plan, plus one end-user centric dashboard that will arbitrarily consume events to populate a client dashboard).

In this future, having other technos than PHP is considered, and having stalling application for 1 or 2 year is also a possibility: we have to specify our messages as a high-level business-oriented specification unbound to the technical aspects. Which means, backward compatibility when possible, having the most recent upgrades able to talk using more than one version of the protocol, etc...

In that regard, sharing code with every application is not possible.

@pounard

This comment has been minimized.

Copy link
Contributor

@pounard pounard commented Oct 10, 2019

@tristanbes

Do you think some of these problematic are generic enough to be covered by the component code or by the documentation ?

I think it should, I opened #33373 #33368 (messenger) and #33399 #33397 (serializer) for discussion some time ago. You might also want to participate in #32049

@ragboyjr

This comment has been minimized.

Copy link
Contributor

@ragboyjr ragboyjr commented Oct 10, 2019

@pounard for sure, you gotta do what works best for your team and roadmap. A custom serializer might be the best way to go if you end up sharing messages across various languages. If you can keep in php, a shard repository that can be updated at its own cadence in the individual repos might be less work.

@pounard

This comment has been minimized.

Copy link
Contributor

@pounard pounard commented Oct 10, 2019

@ragboyjr

That's the usual answer I get :) I do not always agree though. Regarding incoming content type I still think that the messenger component should handle this natively. For the rest, every bit is subject to subjectivity and one's need. But all remain interesting topics to discuss.

To go further in this direction, there's still a probability that we'll switch from the messenger to a more bare-metal AMQP implementation instead one day, if the messenger doesn't easily answer our needs; that's why all our business code sends its messages via a custom interface, and messenger, in our project, is completely hidden behind it. It's replaceable.

@weaverryan

This comment has been minimized.

Copy link
Member

@weaverryan weaverryan commented Oct 10, 2019

Are you suggesting that the message type would be set on the content type and then a system would allow you to map each type to a class in your code (that the Serializer works then use fire deserialization)? Or do I misunderstand?

For us, our custom Serializer is a basic switch case statement: we determine what kind is message is coming in (e.g. by reading some info from a header or just looking at the field), then use that in a switch case to create and return different message objects for each.

@pounard

This comment has been minimized.

Copy link
Contributor

@pounard pounard commented Oct 11, 2019

@weaverryan

Are you suggesting that the message type would be set on the content type and then a system would allow you to map each type to a class in your code

content-type and message type are both very different things, I think they can be handled very differently.

Message type may or may not be exposed by the bus implementation, for example in AMQP you can retrieve messages with the 'type' property (as of now, the messenger stores it alongside the message header, in the form of the PHP class name). Converting from an arbitrary type to a PHP class could be a feature of component/serializer, or could be handled by the messenger serializer (which I think makes more sense).

Regarding content-type (i.e. format in component/serializer semantic) should be handled by the messenger serializer class.

For us, our custom Serializer is a basic switch case statement: we determine what kind is message is coming in (e.g. by reading some info from a header or just looking at the field), then use that in a switch case to create and return different message objects for each.

I think that a basic version of that based upon configuration could be in messenger core, all it needs is a map to be injected, using the dic or annotations (either one or both). You then wouldn't even need to implement your own.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.