Issue/544 reactive streams support #963

dotta · 2015-09-01T10:42:59Z

Here is an initial proposal to fix #544.

I'm entirely new to both Netty and AHC, so if you notice anything weird it's very likely that I haven't understood something :-)

Other than that, something is puzzling me with the tests I wrote. Basically, if I change the CHUNK_SIZE in https://github.com/AsyncHttpClient/async-http-client/compare/AsyncHttpClient:master...dotta:issue/544-reactive-streams-support?expand=1#diff-5133f4e3fed98f7a99081a1ae643c4c0R104 to a value around 20000 (or lower), the tests start to fail because there is a mismatch between the received bytes and the expected ones. I honestly have no good explanation for this, but I thought it may have something to do with the internals of Netty (or maybe a config flag?). I'd really appreciate it if someone more experienced with Netty could have a look.
EDIT: This issue is now fixed after using RxJava instead of providing my own implementation of Publisher in the test :-)

This PR also includes @jroper commit to add support for handling response bodies.

slandelle · 2015-09-01T11:24:57Z

api/src/main/java/org/asynchttpclient/request/body/generator/ReactiveStreamBodyGenerator.java

+
+        private static final Logger LOGGER = LoggerFactory.getLogger(QueueBackedSubscriber.class);
+
+        private final BlockingQueue<ByteBuffer> queue;


Blocking, really?

The alternative is queuing up elements in the subscriber, but the risk with that is overflowing the queue (dropping elements is even worse) when the consumer is much slower than the producer. Also, note this implementation is only used when the provider doesn't support reactive streams.

Also, note this implementation is only used when the provider doesn't support reactive streams.

I'd rather throw an UnsupportedOperationException then. :)

Mmm, I'm looking at the implementation of FeedableBodyGenerator and realise I probably don't need blocking. Forgive me :-)

slandelle · 2015-09-01T11:30:34Z

Hi Mirko,

I think your ReactiveStreamBodyGenerator is broken. Are you sure it cannot be implemented on top of FeedableBodyGenerator?

slandelle · 2015-09-01T11:31:44Z

api/src/main/java/org/asynchttpclient/request/body/generator/ReactiveStreamBodyGenerator.java

+                // don't overflow the passed buffer
+                if(buffer.remaining() <= lastElement.remaining()) {
+                    int bufferSize = buffer.remaining();
+                    buffer.put(lastElement.array(), lastElement.position(), bufferSize);


What if the buffer doesn't have an array?

Oh, is that possible? Any idea how I should handle that? (should I create the array myself, or fail?)

Oh, is that possible?

Of course: DirectBuffers, composites, etc. That's what hasArray() is for.

See FeedableBodyGenerator, you can put a ByteBuffer into another.

@dotta the netty in action cover this:).composites could have array if it have only one element and the element have array or it was empty.

@hepin1989 Don't confuse Netty's ByteBuf and Java's ByteBuffer :)

@slandelle haha,sorry,I don't see the whole context:).

dotta · 2015-09-01T12:01:44Z

Hi @slandelle, thanks a lot for the feedback!

I think your ReactiveStreamBodyGenerator is broken. Are you sure it cannot be implemented on top of FeedableBodyGenerator?

Do you mean the implementation of ReactiveStreamBodyGenerator.StreamedBody? After looking at the FeedableBodyGenerator.PushBody, I feel like I should be able to reuse it. I'll have a deeper look now.

slandelle · 2015-09-01T12:06:31Z

Do you mean the implementation of ReactiveStreamBodyGenerator.StreamedBody? After looking at the FeedableBodyGenerator.PushBody, I feel like I should be able to reuse it. I'll have a deeper look now.

Yeah, they serve the same purpose, so I guess existing code could be reuse/extended/wrapped. FeedableBodyGenerator.PushBody currently has a known issue that the reading buffer has to be larger than the payload, but that's something that can/should be fixed.

dotta · 2015-09-01T16:00:50Z

api/src/main/java/org/asynchttpclient/request/body/generator/ReactiveStreamBodyGenerator.java

+                executor.execute(new Runnable() {
+                    @Override
+                    public void run() {
+                      subscription.request(1);


@jroper I'm not so convinced about requesting only one element at a time. Any thought?

Considering this is a fallback implementation, I think it's fine. Another option, seeing as the provider is going to end up buffering infinitely anyway, is you can invoke subscription.request(Long.MAX_VALUE) in onSubscribe, this is allowed according to reactive streams spec 3.17.

dotta · 2015-09-01T16:04:15Z

@slandelle I think you should be more pleased now (I for sure am :-)).

slandelle · 2015-09-01T16:10:37Z

api/src/main/java/org/asynchttpclient/request/body/generator/IFeedableBodyGenerator.java

@@ -0,0 +1,11 @@
+package org.asynchttpclient.request.body.generator;
+
+public interface IFeedableBodyGenerator {


For consistency, I'd rather remove the I prefix and prefix the default implementation with Simple (or better if you can think of something).

slandelle · 2015-09-02T07:36:01Z

...rs/netty4/src/test/java/org/asynchttpclient/netty/request/body/NettyReactiveStreamsTest.java

@@ -0,0 +1,68 @@
+/*
+ * Copyright (c) 2010-2012 Sonatype, Inc. All rights reserved.


Please change copyright to "the AsyncHttpClient Project".

dotta · 2015-09-02T15:01:28Z

@slandelle I believe I've addressed all comments. The only pending one is #963 (comment), which you are discussing with @jroper.

Let me know if you have other comments, or if I should rebase and squash commits.

slandelle · 2015-09-04T16:57:35Z

@dotta @jroper Sorry, busy day. Will have a look this week-end.

dotta · 2015-09-07T08:40:41Z

@slandelle I'm happy to squash commits and rebase if you give me your bless :-) In the meanwhile, I'll update Play master to take advantage of the changes in this PR. Whohoo!

slandelle · 2015-09-07T12:50:51Z

@dotta Sorry, had some unexpected things to deal with first... Yes, could you please squash and rebase? Let's merge this so everyone can move forward and get everything stable in the next weeks so we can release 2.0.0.

dotta · 2015-09-07T12:51:44Z

Whohooo! I'm on it, thanks! :-)

AsyncHttpClient#544

Issue/544 reactive streams support

slandelle · 2015-09-07T14:16:28Z

@jroper @dotta Thanks guys!

dotta · 2015-09-07T14:21:52Z

@slandelle Thank you for merging!

I'm updating the Play codebase to use AHC master, will let you know if I hit any issue.

slandelle · 2015-09-07T14:25:24Z

Great!
What's your timeline for Play 2.5?

dotta · 2015-09-07T14:31:37Z

I believe it's November 2015, but I'll let @jroper confirm.

Once I've verified that things work properly on the Play side, I'd really appreciate if you could release a 2.0.0 milestone (in case you would like to wait a bit more before cutting a 2.0.0 final).

slandelle · 2015-09-07T14:33:39Z

I can easily cut an alpha.

I'm still considering some refactorings (maybe Proxies) + I might want to break a few things to improve DNS handling in Gatling.

dotta · 2015-09-07T14:35:04Z

Sounds good. I'll ping you soon :-)

dotta · 2015-09-08T16:22:01Z

@slandelle I've a branch in Play that is now using the new StreamedAsyncHandler, and it seems to be working just fine (and I'm really pleased with the result, the code is much, much, much easier to read now!). Could you cut a release so that I can push that work in Play?

slandelle · 2015-09-08T17:55:22Z

I just released 2.0.0-alpha9.
Cheers!

jroper · 2015-09-08T23:32:30Z

Thanks @slandelle!

dotta · 2015-09-09T14:47:43Z

@slandelle Thank you!

slandelle reviewed Sep 1, 2015
View reviewed changes

dotta reviewed Sep 1, 2015
View reviewed changes

slandelle reviewed Sep 1, 2015
View reviewed changes

slandelle reviewed Sep 2, 2015
View reviewed changes

dotta force-pushed the issue/544-reactive-streams-support branch from a7b7b08 to f91bac1 Compare September 2, 2015 14:58

Reactive streams support for handling HTTP request/response bodies. Fix

71c658c

AsyncHttpClient#544

dotta force-pushed the issue/544-reactive-streams-support branch from a373a21 to 71c658c Compare September 7, 2015 13:49

slandelle added Enhancement Netty labels Sep 7, 2015

slandelle added this to the 2.0.0 milestone Sep 7, 2015

slandelle added a commit that referenced this pull request Sep 7, 2015

Merge pull request #963 from dotta/issue/544-reactive-streams-support

fc870d8

Issue/544 reactive streams support

slandelle merged commit fc870d8 into AsyncHttpClient:master Sep 7, 2015

dotta deleted the issue/544-reactive-streams-support branch September 7, 2015 14:15

dotta mentioned this pull request Sep 8, 2015

Reimplement NingWS.executeStream using Akka Stream playframework/playframework#4952

Closed

sanjams2 mentioned this pull request Mar 11, 2021

Closing channels on request streaming failures #1777

Open


		private static final Logger LOGGER = LoggerFactory.getLogger(QueueBackedSubscriber.class);

		private final BlockingQueue<ByteBuffer> queue;

		@@ -0,0 +1,11 @@
		package org.asynchttpclient.request.body.generator;

		public interface IFeedableBodyGenerator {

		@@ -0,0 +1,68 @@
		/*
		* Copyright (c) 2010-2012 Sonatype, Inc. All rights reserved.

Issue/544 reactive streams support #963

Issue/544 reactive streams support #963

Uh oh!

Conversation

dotta commented Sep 1, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

slandelle commented Sep 1, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dotta commented Sep 1, 2015

Uh oh!

slandelle commented Sep 1, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dotta commented Sep 1, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dotta commented Sep 2, 2015

Uh oh!

slandelle commented Sep 4, 2015

Uh oh!

dotta commented Sep 7, 2015

Uh oh!

slandelle commented Sep 7, 2015

Uh oh!

dotta commented Sep 7, 2015

Uh oh!

slandelle commented Sep 7, 2015

Uh oh!

dotta commented Sep 7, 2015

Uh oh!

slandelle commented Sep 7, 2015

Uh oh!

dotta commented Sep 7, 2015

Uh oh!

slandelle commented Sep 7, 2015

Uh oh!

dotta commented Sep 7, 2015

Uh oh!

dotta commented Sep 8, 2015

Uh oh!

slandelle commented Sep 8, 2015

Uh oh!

jroper commented Sep 8, 2015

Uh oh!

dotta commented Sep 9, 2015

Uh oh!

Uh oh!