Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AMQP/HTTP bridge #3415

Closed
alanconway opened this issue May 17, 2018 · 12 comments

Comments

Projects
None yet
5 participants
@alanconway
Copy link

commented May 17, 2018

Title: AMQP/HTTP bridge for envoy

Description:

AMQP (http://www.amqp.org/) is a popular standard protocol for messaging middleware. It supports many patterns of asynchronous message exchanged (brokered queues, peer-to-peer, routed etc.) including the request-response pattern used by HTTP. The idea is to bridge between AMQP request/response and HTTP. Other messaging patters possible in AMQP are out of scope.

The bridge would be in 2 halves:

  1. Receiving AMQP requests, forwarding as HTTP, translating HTTP responses back to AMQP
  2. Receiving HTTP requests, forwarding as AMQP translating AMQP responses back to HTTP

Each half can be deployed independently allowing Envoy to behave as an AMQP to HTTP bridge or a HTTP to AMQP bridge. Both halves could be deployed in the same Envoy instance to make it behave like a limited (request/response only) AMQP router, since there are full-featured AMQP routers already out there that probably isn't a major use case.

The bridge will be implemented as a pair of filters, or a single filter that can be configured to work in either mode. The filter will do a simple, fixed translation from AMQP to HTTP and back. Users of the bridge will rely on Envoy's existing HTTP features to get interesting routing and transformation behaviour on the HTTP side of the bridge.

Goals

  • Envoy can accept and initiate AMQP connections (act as AMQP client or server)
  • Allow all existing Envoy HTTP routing/rewriting features on the HTTP side of the bridge.
  • Flexible mapping/routing of AMQP addresses to HTTP host + path using Envoy's existing URL rewriting and routing abilities
  • Run-time sync between incoming AMQP links and Envoy service discovery (needs more thought)

Use cases

  • HTTP tunneling over a routed AMQP network using http://qpid.apache.org/components/dispatch-router
  • HTTP client access to AMQP requset/response services
  • AMQP client access to HTTP servers
  • (AMQP request/response tunneling over HTTP - possible but not clear that anyone wants that.)

Relevant Links

There is an incomplete work in progress at https://github.com/alanconway/envoy-amqp

@alanconway

This comment has been minimized.

Copy link
Author

commented Jun 11, 2018

@mattklein123 question - the extension source is all under source/extensions, but the configuration is under api/ Should new extensions follow this pattern, or bundle their configuration .proto files with their source?

@mattklein123

This comment has been minimized.

Copy link
Member

commented Jun 11, 2018

@mattklein123 question - the extension source is all under source/extensions, but the configuration is under api/ Should new extensions follow this pattern, or bundle their configuration .proto files with their source?

This is an inconsistency I don't love, but I would just follow along w/ the others for now.

@alanconway

This comment has been minimized.

Copy link
Author

commented Jun 25, 2018

@mattklein123 The envoy doc suggests it supports pipelining but my first attempt didn't work. I have multiple request messages coming in on the AMQP side of a connection, and I simply concatenated the corresponding HTTP requests on the HTTP side. Envoy only processed the first one and dropped the rest. It's very likely I'm doing something wrong, should that work?

The AMQP side expects to be able to send as many requests as it likes and correlate the responses as they flow back, so it ends up heavily pipelined on the HTTP side. For now I've throttled it back to a single request-response at a time but that's not ideal performance-wise. I'm not (yet) using envoy's codec so I'm doing HTTP 1.1 directly - not ideal and maybe part of my problem.

@mattklein123

This comment has been minimized.

Copy link
Member

commented Jun 26, 2018

The HTTP/1.1 client does not perform pipelining. The server also does not actual pipeline, rather it applies back pressure while processing requests to avoid head of line blocking resource constraint situations.

@alanconway

This comment has been minimized.

Copy link
Author

commented Jun 28, 2018

@stale

This comment has been minimized.

Copy link

commented Jul 28, 2018

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Jul 28, 2018

@jmin911

This comment has been minimized.

Copy link

commented Jul 31, 2018

This is useful.

@stale stale bot removed the stale label Jul 31, 2018

@alanconway

This comment has been minimized.

Copy link
Author

commented Aug 1, 2018

@stale

This comment has been minimized.

Copy link

commented Aug 31, 2018

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Aug 31, 2018

@cmluciano

This comment has been minimized.

Copy link
Member

commented Sep 4, 2018

Can we add the "no stalebot" label to this issue @envoyproxy/maintainers

@stale stale bot removed the stale label Sep 4, 2018

@junr03 junr03 added the no stalebot label Sep 4, 2018

@stale

This comment has been minimized.

Copy link

commented Mar 23, 2019

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Mar 23, 2019

@stale

This comment has been minimized.

Copy link

commented Mar 30, 2019

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.

@stale stale bot closed this Mar 30, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.