Feature: AMQP over Websockets (with proxy) #264

pgbhagat · 2018-02-22T04:50:34Z

Issue Description

I am trying out sending messages using 'azure-event-hubs-java'. It uses AMQP protocol which needs TCP ports '5671/5672' to be opened,
We tried having an HTTP proxy, but it won't work as AMQP traffic is TCP and not HTTP/HTTPS,
But we do NOT want to create firewall exception for these ports, we have to use some proxy which will allow amqp traffic, may be SOCKS,
We tried setting up Java code to setup these parameters but its not working.
System.getProperties().put("proxySet", "true"); System.getProperties().put("socksProxyHost", <proxy hostname>); System.getProperties().put("socksProxyPort", <proxy port>);
Another way of setting socks proxy through java invocation
java -DsocksProxyHost=<socks proxy hostname> -DsocksProxyPort=<port>
It does not work, amqp traffic still hits firewall directly where it gets blocked.

Few questions:
Has anyone faced this problem?
Is there any other way to allow this traffic through some proxy?
How does a typical deployment outside AZURE environment works?

Appreciate your help.

The text was updated successfully, but these errors were encountered:

SreeramGarlapati · 2018-02-23T19:30:48Z

Hi @pgbhagat
We have an implementation of this library which talks AMQP over websockets - on https port 443.
Appreciate if you can try out and see if it works over proxy:
https://github.com/Azure/azure-event-hubs-java/tree/websocket.with.1_0

I am not sure - if this will work though. I think there are 2 technical limitations:

Unknown - and will appreciate if you can try and confirm - Reactor library creates sockets using SocketChannel - I don't know if SocketChannel supports or honors the DsocksProxy flags.
The websockets library that we wrote - to implement transport layer - doesn't support proxy.

if there is a limitation at SocketChannel - since it provides - non-blocking IO - which is very crucial, we will have to implement in our websocket.impl layer.

pgbhagat · 2018-02-28T08:42:01Z

We will try it out. Before that, few questions.

How is amqp over websockets performance? Does it performs as comparable to direct amqp?
Since it goes through http, is it still a better choice than http level https://docs.microsoft.com/en-us/rest/api/eventhub/generate-sas-token#using-the-shared-access-signature-at-http-level
and https://docs.microsoft.com/en-us/rest/api/eventhub/send-batch-events

I will try out AMQP over websockets and will see if it works.
Thanks

pgbhagat · 2018-02-28T14:56:40Z

I am trying out this branch, i am facing two problems;

Adding maven dependency: Despite of having below entry in my pom.xml, i am not able to link to correct library from branch websocket.with.1_0. This i come to know while debugging and actually seeing if MessagingFactory.createConnection is not using port 443.

<dependency> <groupId>com.microsoft.azure</groupId> <artifactId>azure-eventhubs-eph</artifactId> <version>0.15.1</version> </dependency>

To resolve above problem for now, i switched to branch websocket.with.1_0, compiled the code on my machine, added a pom dependency as below
<dependency> <groupId>com.microsoft.azure</groupId> <artifactId>azure-eventhubs-eph</artifactId> <version>1.0.0-SNAPSHOT</version> </dependency>

Now correct library code got picked. i.e. MessagingFactory.createConnection with port 443.

I tried to send messages with some more code changes as per library APIs and got the below error.
I think we need to fix two issues, first -> i mentioned in 1. maven dependency with 'azure-eventhubs-eph' should download library from branch 'websocket.with.1_0' and second -> below error :)

19:42:03.690 [pool-13-thread-1] WARN com.microsoft.azure.eventhubs.impl.MessagingFactory [] - messagingFactory[MessagingFactory7dbce4], hostName[mynamespace-ns.servicebus.windows.net], error[scheduling reactor failed Task com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor@2c9814b5 rejected from java.util.concurrent.ThreadPoolExecutor@750a15a6[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 362] java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:454) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:748)]

pgbhagat · 2018-02-28T15:04:51Z

Also I raised an issue #270

SreeramGarlapati · 2018-02-28T19:46:02Z

hi @pgbhagat - websocket.with.1_0 is under development - and I only enabled plain send and receive (the code under this: https://github.com/Azure/azure-event-hubs-java/tree/websocket.with.1_0/azure-eventhubs).
I did not fix the eph parts of it, yet. So, lets ignore issue (1) for now and focus on just getting plain send example running.

using amqp over websockets - should offer very similar perf characteristics as that of using amqp directly over tcp.

for http - every request goes through our authentication layer - but for websockets - its done only when you are creating the sender or receiver. So, websockets will perform better than http.

I didn't understand where you are facing the problem. This is where we create the connection on port 443: https://github.com/Azure/azure-event-hubs-java/blob/websocket.with.1_0/azure-eventhubs/src/main/java/com/microsoft/azure/eventhubs/impl/MessagingFactory.java#L116

your dependency should look something like this (don't include azure-eventhubs-eph yet):

<dependency>
			<groupId>com.microsoft.azure</groupId>
			<artifactId>azure-eventhubs</artifactId>
			<version>1.0.0-SNAPSHOT</version>
			<scope>system</scope>
			<systemPath>/Users/<---path---->/azure-event-hubs-java/azure-eventhubs/target/azure-eventhubs-1.0.0-SNAPSHOT.jar</systemPath>
		</dependency>

pgbhagat · 2018-03-05T08:36:05Z

Shutting down executor after sometime (say 60 seconds) post EventHubClient.send(event) resolved error 'rejected from java.util.concurrent.ThreadPoolExecutor'. It looks 'amqp over websocket' works. I can see data reaching azure. When websocket library will be released so that i can use it?

SreeramGarlapati · 2018-03-06T03:04:57Z

Hi @pgbhagat - we don't have concrete dates yet. I will update you with the release dates for websocket support.

Did you try websockets from behind a proxy ? or did you try plain websockets ?
we currently are still trying to figure out websockets with proxy - are you interested in websockets with proxy ? authenticated proxy or plain (without proxy).

Thanks a lot for trying a reporting back.!

pgbhagat · 2018-03-06T12:08:39Z

@SreeramGarlapati , Sorry, it does not work.
One of my test box is configured to allowed all network traffic and so 'amqp over websocket' works on it.
But now i am using a box where only http traffic is allowed through http proxy, here i am getting below error:

Exception in thread "main" com.microsoft.azure.eventhubs.EventHubException: connection aborted
at com.microsoft.azure.eventhubs.impl.ExceptionUtil.toException(ExceptionUtil.java:59)
at com.microsoft.azure.eventhubs.impl.MessagingFactory.onConnectionError(MessagingFactory.java:249)
at com.microsoft.azure.eventhubs.impl.ConnectionHandler.onTransportError(ConnectionHandler.java:114)
at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
at com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:452)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Our Firewall preventing outgoing TCP connection.

We wanted 'amqp over websocket' to work with http proxy server.

pgbhagat · 2018-03-14T17:15:05Z

@SreeramGarlapati any update on release date for websocket support?

SreeramGarlapati · 2018-03-15T19:36:55Z

Hi @pgbhagat, Thanks a lot checking back!

We don't have any crisp dates yet. I am trying to get basic amqp over websocket with proxy and no authorization - to work. Once, I have a POC - I will notify you. I will most likely have one by Mid next week.

…es (#293) SNAPSHOT jars for `websocket proxy support` #264

SreeramGarlapati · 2018-03-20T01:16:16Z

Hi @pgbhagat - may I request you to validate the POC for websockets over proxy using these instructions and ack. back.
https://github.com/SreeramGarlapati/azure-event-hubs-java/tree/websocket.with.1_0/snapshots/com/microsoft/azure/azure-eventhubs/1.1.0-SNAPSHOT

pgbhagat · 2018-03-21T05:13:13Z

@SreeramGarlapati - yes, we will try this POC library, but will be able to do so by next week.

ChrisCollinsIBM · 2018-05-01T12:32:56Z

We have many end users looking for some type of proxy support that doesn't have a requirement to allow 5671/5672 to be directly open, we're also using the azure-eventhubs-eph code and would be anxious to know when a test version that the Websockets code extends to would be available to test out!

balistof · 2018-07-04T17:38:43Z

When will the WS enhancement be merged back to master and be released?

1. Supports max_frame_size of 4k (current limitation of <a href="https://github.com/Azure/qpid-proton-j-extensions">qpid-proton-j-extensions</a> library) 2. doesn't support PROXY on websockets - this will follow. Amqp over WebSockets is particularly used when enterprise policies (ex: firewall outbound port rules) doesn't allow traffic on the default Amqp secure port (`5671`). To send or receive over websockets - which uses port `443`, set the `TransportType` on `ConnectionStringBuilder`, like this: ``` connectionStringBuilder.setTransportType(TransportType.AmqpWebSockets) ``` related: #264

SreeramGarlapati · 2018-08-15T17:36:39Z

@dak0ta / @balistof - the current snapshot version on dev branch is ready for Websockets. Please validate the bits per your requirement and ack/nack on this thread.

balistof · 2018-08-16T14:05:16Z

@SreeramGarlapati - looks good

SreeramGarlapati · 2018-09-12T04:31:29Z

@dak0ta @pgbhagat - here's the preview package of proxy support for websockets

<repositories>  
  <repository>  
    <id>azure eventhubs</id>  
    <url>https://oss.sonatype.org/content/repositories/commicrosoftazure-2584</url>  
  </repository>  
</repositories>  

  <dependency> 
    <groupId>com.microsoft.azure</groupId> 
    <artifactId>azure-eventhubs</artifactId> 
    <version>1.2.0-PREVIEW</version> 
  </dependency>

Code Sample to set proxy:

ExecutorService threadPool = Executors.newCachedThreadPool();
try {
    EventHubClient.setProxyHostName("NN.NN.NN.NN"); // ip address of proxy
    EventHubClient.setProxyHostPort(8888);
    EventHubClient.setProxyUserName("1"); // these usernames passwords work for fiddler as proxy!
    EventHubClient.setProxyPassword("1");

    final ConnectionStringBuilder connectionStringBuilder = new ConnectionStringBuilder(EVENTHUB_CONNECTION_STRING);
    connectionStringBuilder.setTransportType(TransportType.AMQP_WEB_SOCKETS); // PROXY is enabled only on this TransportType
    final EventHubClient ehClient = EventHubClient.createSync(connectionStringBuilder.toString(), threadPool);
    ehClient.sendSync(EventData.create("Test Message".getBytes()));

    ehClient.closeSync();
} finally {
    threadPool.shutdown();
}

Appreciate your (n)ack.!!

ChrisCollinsIBM · 2018-09-13T13:41:09Z

We'll give it a look - thanks!

ChrisCollinsIBM · 2018-09-21T18:37:47Z

So we've done some experimenting and haven't been able to get things working.

We setup a Dante SOCKS5 proxy and pointed the client at it but got errors.

In /var/log/socks.log

Sep 21 11:52:18 (1537541538.423074) sockd[8103]: info: pass(1): tcp/accept [: X.X.X.X.46622 Y.Y.Y.Y.8888
Sep 21 11:52:18 (1537541538.426758) sockd[8103]: info: block(1): tcp/accept ]: X.X.X.X.46622 Y.Y.Y.Y.8888: error after reading 1 byte in 0 seconds: unknown SOCKS version 67 in client request

And from the Azure SDK:

[pool-1-thread-1] INFO com.microsoft.azure.eventhubs.impl.ProxyConnectionHandler - addProxyHandshake: hostname[XXXXXXXXXXXX.servicebus.windows.net:443]
[pool-1-thread-2] WARN com.microsoft.azure.eventhubs.impl.ConnectionHandler - onTransportClosed: hostname[XXXXXXXXXXXX.servicebus.windows.net:443], error[org.apache.qpid.proton.engine.TransportException: connection aborted]
[pool-1-thread-2] INFO com.microsoft.azure.eventhubs.impl.ConnectionHandler - onConnectionUnbound: hostname[XXXXXXXXXXXX.servicebus.windows.net:443], state[ACTIVE], remoteState[UNINITIALIZED]
Exception in thread "main" com.microsoft.azure.eventhubs.EventHubException: org.apache.qpid.proton.engine.TransportException: connection aborted
at com.microsoft.azure.eventhubs.impl.ExceptionUtil.toException(ExceptionUtil.java:60)
at com.microsoft.azure.eventhubs.impl.MessagingFactory.onConnectionError(MessagingFactory.java:265)
at com.microsoft.azure.eventhubs.impl.ConnectionHandler.onTransportError(ConnectionHandler.java:163)
at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
at com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:491)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:785)

=====================================

So what type of Proxy should we be using? An HTTPS proxy doesn't seem relevant for AMQP over Websockets.

Any help is appreciated!

SreeramGarlapati · 2018-09-21T20:36:10Z

@dak0ta
Use HTTP proxy - please refer here for more help: https://github.com/Azure/azure-event-hubs-java/releases/tag/1.2.0.
Also, while you try that please, let me know if you have any more suggestions for this release description.

ChrisCollinsIBM · 2018-09-24T16:53:31Z

Thanks. So we had an older squid that wasn't working great with HTTP CONNECT tunnelling so I setup a newer one.

New one works fine with CURL to test, but when trying out the sample EventHubs code we get

Exception in thread "main" com.microsoft.azure.eventhubs.TimeoutException: Opening MessagingFactory timed out.
    at com.microsoft.azure.eventhubs.impl.MessagingFactory$1.run(MessagingFactory.java:125)
    at com.microsoft.azure.eventhubs.impl.Timer$ScheduledTask.onEvent(Timer.java:49)
    at com.microsoft.azure.eventhubs.impl.DispatchHandler.onTimerTask(DispatchHandler.java:9)
    at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:233)
    at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
    at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
    at com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:491)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(Thread.java:785)

Squid logs show a request to the eventhubs on 443, but I don't think that's what this request here is.

1537807898.328 59999 X.X.X.X TCP_TUNNEL/200 4011 CONNECT XXXXXXXXXXXX.servicebus.windows.net:443 - HIER_DIRECT/40.86.225.142 -

Is this first request succeeding then a subsequent AMQP request failing? Or is this first one actually failing?

amitnswain · 2019-08-16T10:50:37Z

Hi All,

I have a solution for the Servicebus Exception issue where The firewall is restricting to use the PORT:5671 for some of the client network VMs.

If your Firewall is set ON by the Administrator which is blocking the AMQP request then use the below steps to set your firewall OFF and try to test the Service bus Connection again.

Search for Command --> Right click and Run as Administrator

Then run the below command to check the Firewall status for different profiles in the system.

netsh advfirewall show allprofiles

Then run the below commands to turn the firewall off.

netsh advfirewall set domainprofile state off
netsh advfirewall set privateprofile state off
netsh advfirewall set publicprofile state off

Now you can check the firewall status again.

Please reply here if the solution works. It worked for me for my client network.

1. Supports max_frame_size of 4k (current limitation of <a href="https://github.com/Azure/qpid-proton-j-extensions">qpid-proton-j-extensions</a> library) 2. doesn't support PROXY on websockets - this will follow. Amqp over WebSockets is particularly used when enterprise policies (ex: firewall outbound port rules) doesn't allow traffic on the default Amqp secure port (`5671`). To send or receive over websockets - which uses port `443`, set the `TransportType` on `ConnectionStringBuilder`, like this: ``` connectionStringBuilder.setTransportType(TransportType.AmqpWebSockets) ``` related: #264

SreeramGarlapati self-assigned this Feb 23, 2018

SreeramGarlapati added a commit that referenced this issue Mar 20, 2018

POC SNAPSHOT jar for EventHubs client to talk - websockets over proxi…

d867419

…es (#293) SNAPSHOT jars for `websocket proxy support` #264

SreeramGarlapati changed the title ~~AMQP proxy setting~~ Feature: AMQP over Websockets (with proxy) Mar 26, 2018

SreeramGarlapati added the enhancement label Mar 26, 2018

SreeramGarlapati mentioned this issue Aug 9, 2018

Websockets support #362

Merged

This was referenced Sep 11, 2018

Transport layer for proxy handshake Azure/qpid-proton-j-extensions#8

Merged

WebSockets Proxy support #378

Merged

SreeramGarlapati closed this as completed in #378 Sep 21, 2018

SreeramGarlapati mentioned this issue Sep 21, 2018

Followup: websockets proxy support #381

Merged

SreeramGarlapati mentioned this issue Sep 24, 2018

EventHubClient (with transportType=Websockets) creation times out when running on java-x86_64 (mixed mode) #385

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: AMQP over Websockets (with proxy) #264

Feature: AMQP over Websockets (with proxy) #264

pgbhagat commented Feb 22, 2018 •

edited by SreeramGarlapati

Loading

SreeramGarlapati commented Feb 23, 2018 •

edited

Loading

pgbhagat commented Feb 28, 2018

pgbhagat commented Feb 28, 2018 •

edited

Loading

pgbhagat commented Feb 28, 2018 •

edited

Loading

SreeramGarlapati commented Feb 28, 2018

pgbhagat commented Mar 5, 2018 •

edited

Loading

SreeramGarlapati commented Mar 6, 2018

pgbhagat commented Mar 6, 2018 •

edited

Loading

pgbhagat commented Mar 14, 2018

SreeramGarlapati commented Mar 15, 2018

SreeramGarlapati commented Mar 20, 2018

pgbhagat commented Mar 21, 2018

ChrisCollinsIBM commented May 1, 2018

balistof commented Jul 4, 2018

SreeramGarlapati commented Aug 15, 2018 •

edited

Loading

balistof commented Aug 16, 2018

SreeramGarlapati commented Sep 12, 2018

ChrisCollinsIBM commented Sep 13, 2018

ChrisCollinsIBM commented Sep 21, 2018

SreeramGarlapati commented Sep 21, 2018

ChrisCollinsIBM commented Sep 24, 2018 •

edited

Loading

amitnswain commented Aug 16, 2019 •

edited

Loading

Feature: AMQP over Websockets (with proxy) #264

Feature: AMQP over Websockets (with proxy) #264

Comments

pgbhagat commented Feb 22, 2018 • edited by SreeramGarlapati Loading

Issue Description

SreeramGarlapati commented Feb 23, 2018 • edited Loading

pgbhagat commented Feb 28, 2018

pgbhagat commented Feb 28, 2018 • edited Loading

pgbhagat commented Feb 28, 2018 • edited Loading

SreeramGarlapati commented Feb 28, 2018

pgbhagat commented Mar 5, 2018 • edited Loading

SreeramGarlapati commented Mar 6, 2018

pgbhagat commented Mar 6, 2018 • edited Loading

pgbhagat commented Mar 14, 2018

SreeramGarlapati commented Mar 15, 2018

SreeramGarlapati commented Mar 20, 2018

pgbhagat commented Mar 21, 2018

ChrisCollinsIBM commented May 1, 2018

balistof commented Jul 4, 2018

SreeramGarlapati commented Aug 15, 2018 • edited Loading

balistof commented Aug 16, 2018

SreeramGarlapati commented Sep 12, 2018

ChrisCollinsIBM commented Sep 13, 2018

ChrisCollinsIBM commented Sep 21, 2018

SreeramGarlapati commented Sep 21, 2018

ChrisCollinsIBM commented Sep 24, 2018 • edited Loading

amitnswain commented Aug 16, 2019 • edited Loading

pgbhagat commented Feb 22, 2018 •

edited by SreeramGarlapati

Loading

SreeramGarlapati commented Feb 23, 2018 •

edited

Loading

pgbhagat commented Feb 28, 2018 •

edited

Loading

pgbhagat commented Feb 28, 2018 •

edited

Loading

pgbhagat commented Mar 5, 2018 •

edited

Loading

pgbhagat commented Mar 6, 2018 •

edited

Loading

SreeramGarlapati commented Aug 15, 2018 •

edited

Loading

ChrisCollinsIBM commented Sep 24, 2018 •

edited

Loading

amitnswain commented Aug 16, 2019 •

edited

Loading