Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider specializing std::vec::from_elem to calloc when possible #38723

Closed
sfackler opened this issue Dec 30, 2016 · 5 comments
Closed

Consider specializing std::vec::from_elem to calloc when possible #38723

sfackler opened this issue Dec 30, 2016 · 5 comments
Labels
C-enhancement Category: An issue proposing an enhancement or a PR with one. I-slow Issue: Problems and improvements with respect to performance of generated code. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.

Comments

@sfackler
Copy link
Member

This will require adding calloc to the global allocator API, but can significantly speed up situations where a large buffer is allocated but not entirely used.

@brson
Copy link
Contributor

brson commented Dec 31, 2016

Vec::from for [T] probably, not from_slice (I don't see a from_slice).

What's the benefit you are expecting? When would zeroed pages help in Vec?

@brson brson added I-slow Issue: Problems and improvements with respect to performance of generated code. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. C-enhancement Category: An issue proposing an enhancement or a PR with one. labels Dec 31, 2016
@sfackler
Copy link
Member Author

This would be targeting things like vec![0; 40960] which uses from_elem.

Calloc can avoid explicitly zeroing pages if it knows they're already zeroed out (e.g. if they just came from an mmap call). If much of the buffer isn't actually used, this can be way faster. See for example https://bitbucket.org/cffi/cffi/issues/295/cffinew-is-way-slower-than-it-should-be-it

@abonander
Copy link
Contributor

abonander commented Jan 2, 2017

As a practical example, this could improve performance of both std::io::BufReader::new() and Read::read_to_end()/read_to_string() (at least for reading into an empty vector). The box operator could also be specialized to use this for the same reasons.

I asked about this before on one of the easy question threads on Reddit: one of the responses I got said that it probably wouldn't improve performance under jemalloc for various esoteric reasons (which I found hard to believe, but whatever); however, it would definitely help under the system allocator.

Edit: consolidated two comments and removed redundant statements

@bluss bluss changed the title Consider specializing vec::from_slice to calloc when possible Consider specializing std::vec::from_elem to calloc when possible Jan 3, 2017
@bluss
Copy link
Member

bluss commented Jan 3, 2017

std::vec::from_elem is the function behind the vec![x; n] repetition. source for from_elem

mbrubeck added a commit to mbrubeck/rust that referenced this issue Mar 15, 2017
frewsxcv added a commit to frewsxcv/rust that referenced this issue Mar 17, 2017
Specialize Vec::from_elem to use calloc

Fixes rust-lang#38723.  This specializes the implementation for `u8` only, but it could be extended to other zeroable types if desired.

I haven't tested this extensively, but I did verify that it gives the expected performance boost for large `vec![0; n]` allocations with both alloc_system and jemalloc, on Linux.  (I have not tested or even built the Windows code.)
frewsxcv added a commit to frewsxcv/rust that referenced this issue Mar 17, 2017
Specialize Vec::from_elem to use calloc

Fixes rust-lang#38723.  This specializes the implementation for `u8` only, but it could be extended to other zeroable types if desired.

I haven't tested this extensively, but I did verify that it gives the expected performance boost for large `vec![0; n]` allocations with both alloc_system and jemalloc, on Linux.  (I have not tested or even built the Windows code.)
mbrubeck added a commit to mbrubeck/rust that referenced this issue Mar 17, 2017
frewsxcv added a commit to frewsxcv/rust that referenced this issue Mar 18, 2017
Specialize Vec::from_elem to use calloc

Fixes rust-lang#38723.  This specializes the implementation for `u8` only, but it could be extended to other zeroable types if desired.

I haven't tested this extensively, but I did verify that it gives the expected performance boost for large `vec![0; n]` allocations with both alloc_system and jemalloc, on Linux.  (I have not tested or even built the Windows code.)
arielb1 pushed a commit to arielb1/rust that referenced this issue Mar 18, 2017
Specialize Vec::from_elem to use calloc

Fixes rust-lang#38723.  This specializes the implementation for `u8` only, but it could be extended to other zeroable types if desired.

I haven't tested this extensively, but I did verify that it gives the expected performance boost for large `vec![0; n]` allocations with both alloc_system and jemalloc, on Linux.  (I have not tested or even built the Windows code.)
arielb1 pushed a commit to arielb1/rust that referenced this issue Mar 19, 2017
Specialize Vec::from_elem to use calloc

Fixes rust-lang#38723.  This specializes the implementation for `u8` only, but it could be extended to other zeroable types if desired.

I haven't tested this extensively, but I did verify that it gives the expected performance boost for large `vec![0; n]` allocations with both alloc_system and jemalloc, on Linux.  (I have not tested or even built the Windows code.)
mbrubeck added a commit to mbrubeck/rust that referenced this issue Apr 15, 2017
frewsxcv added a commit to frewsxcv/rust that referenced this issue Apr 16, 2017
Specialize Vec::from_elem to use calloc

Fixes rust-lang#38723.  This specializes the implementation for `u8` only, but it could be extended to other zeroable types if desired.

I haven't tested this extensively, but I did verify that it gives the expected performance boost for large `vec![0; n]` allocations with both alloc_system and jemalloc, on Linux.  (I have not tested or even built the Windows code.)
bors added a commit that referenced this issue Apr 16, 2017
Specialize Vec::from_elem to use calloc

Fixes #38723.  This specializes the implementation for `u8` only, but it could be extended to other zeroable types if desired.

I haven't tested this extensively, but I did verify that it gives the expected performance boost for large `vec![0; n]` allocations with both alloc_system and jemalloc, on Linux.  (I have not tested or even built the Windows code.)
@Stargateur
Copy link
Contributor

A solution for array where all is zero ? see the stackoverflow question: Efficiently allocating a vector of structs in Rust. Also, maybe add tuple. const fn should probably be a good feature here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-enhancement Category: An issue proposing an enhancement or a PR with one. I-slow Issue: Problems and improvements with respect to performance of generated code. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

5 participants