Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
AMQP/HTTP bridge #3415
Title: AMQP/HTTP bridge for envoy
AMQP (http://www.amqp.org/) is a popular standard protocol for messaging middleware. It supports many patterns of asynchronous message exchanged (brokered queues, peer-to-peer, routed etc.) including the request-response pattern used by HTTP. The idea is to bridge between AMQP request/response and HTTP. Other messaging patters possible in AMQP are out of scope.
The bridge would be in 2 halves:
Each half can be deployed independently allowing Envoy to behave as an AMQP to HTTP bridge or a HTTP to AMQP bridge. Both halves could be deployed in the same Envoy instance to make it behave like a limited (request/response only) AMQP router, since there are full-featured AMQP routers already out there that probably isn't a major use case.
The bridge will be implemented as a pair of filters, or a single filter that can be configured to work in either mode. The filter will do a simple, fixed translation from AMQP to HTTP and back. Users of the bridge will rely on Envoy's existing HTTP features to get interesting routing and transformation behaviour on the HTTP side of the bridge.
There is an incomplete work in progress at https://github.com/alanconway/envoy-amqp
referenced this issue
May 29, 2018
This is an inconsistency I don't love, but I would just follow along w/ the others for now.
@mattklein123 The envoy doc suggests it supports pipelining but my first attempt didn't work. I have multiple request messages coming in on the AMQP side of a connection, and I simply concatenated the corresponding HTTP requests on the HTTP side. Envoy only processed the first one and dropped the rest. It's very likely I'm doing something wrong, should that work?
The AMQP side expects to be able to send as many requests as it likes and correlate the responses as they flow back, so it ends up heavily pipelined on the HTTP side. For now I've throttled it back to a single request-response at a time but that's not ideal performance-wise. I'm not (yet) using envoy's codec so I'm doing HTTP 1.1 directly - not ideal and maybe part of my problem.
Gotcha. At the back-end of an AMQP network, my amqp_server filter does potentially do a bunch of illegal pipelining because it's aggregating requests from many sources and not bothering to check if they're idempotent, I'm seeing occasional problems that may be related to that. Sounds like I should slow the flow with AMQP credit and pass requests up to envoy one-by-one.…
On Mon, Jun 25, 2018 at 9:45 PM, Matt Klein ***@***.***> wrote: The HTTP/1.1 client does not perform pipelining. The server also does not actual pipeline, rather it applies back pressure while processing requests to avoid head of line blocking resource constraint situations. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3415 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHa6XrzRXN-iUcvVlRGnpeLNhSluIZltks5uAZJHgaJpZM4UDCGA> .
Not forgotten, but on hold, I'm in the process of presentations & discussions with other interested parties to gather feedback before taking it further, but it is still a going concern. I'm always interested in feedback from the envoy perspective.…
On Tue, Jul 31, 2018 at 4:42 PM, Jim Min ***@***.***> wrote: This is useful. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3415 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHa6XpZKHYrzvNMxp6L8oxgI-9Hk8i4vks5uMMFQgaJpZM4UDCGA> .