-
Notifications
You must be signed in to change notification settings - Fork 194
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory usage #297
Comments
You can use techniques like streaming a Vector instead of a single value.
conduit-combinators provides many helper functions for that case, as well
as Text and Bytestring.
…On Wed, Feb 8, 2017, 5:51 PM snap9 ***@***.***> wrote:
It looks like Conduit, in the case of lists, focuses on streaming data
through a pipe one element at a time.
- Does this mean the max memory use is the memory size of each element
in the list?
- Do most people using Conduit just use the default 1-element approach
and find it fast enough?
- Can we customize the memory usage? For instance, make it stream 10
values at once. This would increase memory usage 10x but could improve
execution time.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#297>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AADBB4YT5ZFCuqHrULTU1yagwT4iuwkIks5raeSXgaJpZM4L6_aO>
.
|
@snoyberg Thanks for the tips. Just to clarify, you're suggesting I use [Vector a] instead of [a]. It would now be my responsibility to pack as many values as I want (for instance 10 values) into these vectors. So it's not like conduit has some magic to pack more values. I'd have to do it myself using Vector, adjust the number of elements myself, and benchmark for myself which works best. Am I understanding correctly? |
Essentially, yes. Some convenience functions like foldMapCE will choose a
size for the vector based on other constraints.
…On Sun, Feb 12, 2017, 7:22 PM snap9 ***@***.***> wrote:
@snoyberg <https://github.com/snoyberg> Thanks for the tips. Just to
clarify, if streamed a vector (a list of vectors, I presume), it would now
be my responsibility to pack as many values as I want (for instance 10
values) into these vectors, wouldn't it?
So it's not like conduit has some magic to pack more values. I'd have to
do it myself using Vector and adjust the number of elements myself and
benchmark for myself which works best.
Am I understanding correctly?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#297 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AADBBygVa2Mu9TKafpmb_YhAE9lVL28dks5rbz_igaJpZM4L6_aO>
.
|
It looks like Conduit, in the case of lists, focuses on streaming data through a pipe one element at a time.
The text was updated successfully, but these errors were encountered: