Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove memory copy when extract frame in LengthFieldBasedFrameDecoder … #3704

Closed
normanmaurer opened this issue Apr 29, 2015 · 2 comments
Closed

Remove memory copy when extract frame in LengthFieldBasedFrameDecoder … #3704

normanmaurer opened this issue Apr 29, 2015 · 2 comments
Assignees
@normanmaurer
Labels
improvement
Milestone
4.0.28.Final

Comments

@normanmaurer
Copy link
Member

Motivation:

We are currently doing a memory cop to extract the frame in LengthFieldBasedFrameDecoder which can be eliminated.

Modifications:

Use buffer.slice(...).retain() to eliminate the memory copy.

Result:

Better performance.
@normanmaurer normanmaurer added this to the 4.0.28.Final milestone Apr 29, 2015
@normanmaurer
Copy link
Member Author

Fixed by 4.0 (4c70a99) , 4.1 (891be30) and master (4c70a99)

@normanmaurer normanmaurer self-assigned this Apr 29, 2015
@slandelle
Copy link
Contributor

👍

asfgit pushed a commit to apache/flink that referenced this issue Jan 13, 2016
@uce
[hotfix, runtime] Downgrade to Netty version 4.0.27.Final
d92e422 
Netty versions >= 4.0.28.Final contain an improvement by Netty, which slices
a Netty buffer instead of doing a memory copy (netty/netty#3704)
in the LengthFieldBasedFrameDecoder. In some situations, this interacts badly
with our Netty pipeline leading to OutOfMemory error.

To reproduce, run batch WordCount with a range partitioner and reasonably large
data.
asfgit pushed a commit to apache/flink that referenced this issue Jan 13, 2016
@uce
[FLINK-3151] [runtime] Downgrade to Netty version 4.0.27.Final
66dd32d 
Netty versions >= 4.0.28.Final contain an improvement by Netty, which slices
a Netty buffer instead of doing a memory copy (netty/netty#3704)
in the LengthFieldBasedFrameDecoder. In some situations, this interacts badly
with our Netty pipeline leading to OutOfMemory error.

To reproduce, run batch WordCount with a range partitioner and reasonably large
data.
@NicoK NicoK mentioned this issue Feb 23, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@normanmaurer normanmaurer

Labels
improvement
Projects
None yet
Milestone
4.0.28.Final
Development

No branches or pull requests
2 participants
@normanmaurer @slandelle