New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve BasicIStreamWrapper performance by supplying a buffer (like FileReadStream) #1365

Open
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
4 participants
@tcShadowWalker

tcShadowWalker commented Sep 18, 2018

Significantly improves performance over unbuffered access to the underlying istream. (factor 10)
With these changes, the performance is essentially the same as for FileReadStream.

Jan-Philip Stecker
Improve BasicIStreamWrapper by supplying a buffer (like FileReadStream).
Significantly improves performance over unbuffered access to the underlying istream. (factor 10)
With these changes, the performance is essentially the same as for FileReadStream.
@tencent-adm

This comment has been minimized.

Show comment
Hide comment
@tencent-adm

tencent-adm Sep 18, 2018

CLA assistant check
Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Jan-Philip Stecker seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

tencent-adm commented Sep 18, 2018

CLA assistant check
Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Jan-Philip Stecker seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Sep 18, 2018

Coverage Status

Coverage decreased (-0.003%) to 99.919% when pulling da36212 on tcShadowWalker:faster_istreamwrapper into c0ca05f on Tencent:master.

coveralls commented Sep 18, 2018

Coverage Status

Coverage decreased (-0.003%) to 99.919% when pulling da36212 on tcShadowWalker:faster_istreamwrapper into c0ca05f on Tencent:master.

@miloyip

This comment has been minimized.

Show comment
Hide comment
@miloyip

miloyip Sep 19, 2018

Collaborator

I think MemoryStream is what you are trying to deal with.

Collaborator

miloyip commented Sep 19, 2018

I think MemoryStream is what you are trying to deal with.

@tcShadowWalker

This comment has been minimized.

Show comment
Hide comment
@tcShadowWalker

tcShadowWalker Sep 19, 2018

No, I very much want to use the istream-API. In my real use-case the istream is a boost::iostream with a gzip decompressor.
However, currently the istream API is so incredibly slow that it's just plain unusable because of the high overhead for each single character.
I dont have the entire Json-document in-memory at once (it's hundreds of MB in size uncompressed), so I don't believe MemoryStream is a good fit?

tcShadowWalker commented Sep 19, 2018

No, I very much want to use the istream-API. In my real use-case the istream is a boost::iostream with a gzip decompressor.
However, currently the istream API is so incredibly slow that it's just plain unusable because of the high overhead for each single character.
I dont have the entire Json-document in-memory at once (it's hundreds of MB in size uncompressed), so I don't believe MemoryStream is a good fit?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment