Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix to call onRemoval() immediately when using StreamMessage.collect() #3670

Merged
merged 2 commits into from
Jul 2, 2021

Conversation

ikhoon
Copy link
Contributor

@ikhoon ikhoon commented Jun 30, 2021

Motivation:

DecodedHttp{Request,Response} removes an object from the inbound
traffic when onRemoval() is called. The current implementation of
StreamMessage.collect() in DefaultStreamMessage calls onRemoval()
when a stream is closed via close(). That means the inbound traffic
will be removed only after the messages of a stream are fully received.

Modifications:

  • Immediately drain the objects from the queue of
    DefaultStreamMessage whenever they are written to the queue
    when collect() is called so that onRemoval() is called right away.

Result:

You no longer see a ResponseTimeoutException when collecting a large number of
message using Http{Request,Response}.aggreagte().

…ct()`.

Motivation:

`DecodedHttp{Request,Response}` remove an object from the inbound
traffic when `onRemoval()` is called. The current implementation of
`StreamMessage.collect()` in `DefaultStreamMessage` calls `onRemoval()`
when a stream is closed via `close()`. That means the inbound traffic
will be removed only when the messages of a stream is fully received.

Modifications:

- Immediately drain the objects from the `queue` of
  `DefaultStreamMessage` whenever they are written to the `queue`
  when `collect()` is called so that `onRemoval()` is called right away.

Result:

You no longer see a `ResponseTimeoutException` when collecting a large number of
message using `Http{Request,Response}.aggreagte()`.
@ikhoon ikhoon added the defect label Jun 30, 2021
@ikhoon ikhoon added this to the 1.10.0 milestone Jun 30, 2021
@ikhoon ikhoon modified the milestones: 1.10.0, 1.9.2 Jun 30, 2021
Copy link
Member

@minwoox minwoox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some nits. Thanks!

} else {
// We don't need to invoke onSubscribe() for NoopSubscriber.
invokedOnSubscribe = true;
notifySubscriber();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to call this method here? 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though a stream is open, we need to drain queue so that onRemoval() should be called immediately.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I meant was that if this thread fails to set the state as State.CLEANUP, then there might be another thread that handles the elements.
So I was wondering in which case we need to call this method. 🤔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I realized that the State can be State.OPEN so we need to call that. 😅

@minwoox
Copy link
Member

minwoox commented Jul 1, 2021

Could you check the test failure? https://github.com/line/armeria/pull/3670/checks?check_run_id=2950644605
I saw the same failure from multiple CI builds so I was wondering if it's related to this change. 🤔

@ikhoon
Copy link
Contributor Author

ikhoon commented Jul 1, 2021

Could you check the test failure? #3670 (checks)
I saw the same failure from multiple CI builds so I was wondering if it's related to this change. 🤔

Oops... It is possible.

@codecov
Copy link

codecov bot commented Jul 1, 2021

Codecov Report

Merging #3670 (582e153) into master (369428d) will increase coverage by 0.03%.
The diff coverage is 96.55%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #3670      +/-   ##
============================================
+ Coverage     73.39%   73.42%   +0.03%     
- Complexity    14594    14600       +6     
============================================
  Files          1283     1283              
  Lines         56001    56010       +9     
  Branches       7171     7173       +2     
============================================
+ Hits          41103    41127      +24     
+ Misses        11269    11268       -1     
+ Partials       3629     3615      -14     
Impacted Files Coverage Δ
...rp/armeria/common/stream/DefaultStreamMessage.java 88.43% <96.42%> (+2.77%) ⬆️
...p/armeria/common/stream/AbstractStreamMessage.java 83.33% <100.00%> (+3.89%) ⬆️
...rp/armeria/server/tomcat/ManagedTomcatService.java 50.84% <0.00%> (-22.04%) ⬇️
...nternal/server/tomcat/Tomcat90ProtocolHandler.java 50.00% <0.00%> (-15.39%) ⬇️
...rp/armeria/server/tomcat/TomcatServiceBuilder.java 36.36% <0.00%> (-10.91%) ⬇️
...com/linecorp/armeria/server/saml/SamlEndpoint.java 62.50% <0.00%> (-2.50%) ⬇️
...corp/armeria/common/logging/DefaultRequestLog.java 76.88% <0.00%> (-0.26%) ⬇️
...a/com/linecorp/armeria/client/HttpChannelPool.java 77.92% <0.00%> (+0.27%) ⬆️
...com/linecorp/armeria/server/HttpServerHandler.java 76.65% <0.00%> (+0.31%) ⬆️
.../linecorp/armeria/client/Http1ResponseDecoder.java 65.11% <0.00%> (+0.58%) ⬆️
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 369428d...582e153. Read the comment docs.

Copy link
Member

@minwoox minwoox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 🎉 🎉

Copy link
Member

@trustin trustin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @ikhoon!

@trustin trustin merged commit 7ca9b24 into line:master Jul 2, 2021
@ikhoon ikhoon deleted the collect-bug branch August 31, 2021 08:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants