Skip to content

Commit

Permalink
Optimize 'construct at end' loops in vector
Browse files Browse the repository at this point in the history
Summary:
This change adds local 'end' and 'pos' variables for the main loop inmstead of using the ConstructTransaction variables directly.

We observed that not all vector initialization and resize operations got properly vectorized, i.e., (partially) unrolled into XMM stores for floats.

For example, `vector<int32_t> v(n, 1)` gets vectorized, but `vector<float> v(n, 1)`. It looks like the compiler assumes the state is leaked / aliased in the latter case (unclear how/why for float, but not for int32), and because of this fails to see vectorization optimization?

See https://gcc.godbolt.org/z/UWhiie

By using a local `__new_end_` (fixed), and local `__pos` (copied into __tx.__pos_ per iteration), we offer the compiler a clean loop for unrolling.

A demonstration can be seen in the isolated logic in https://gcc.godbolt.org/z/KoCNWv

The com

Reviewers: EricWF, #libc!

Subscribers: libcxx-commits

Tags: #libc

Differential Revision: https://reviews.llvm.org/D82111
  • Loading branch information
martijnvels committed Jun 18, 2020
1 parent 3e59dfc commit 555106a
Showing 1 changed file with 9 additions and 6 deletions.
15 changes: 9 additions & 6 deletions libcxx/include/vector
Expand Up @@ -1043,8 +1043,9 @@ void
vector<_Tp, _Allocator>::__construct_at_end(size_type __n)
{
_ConstructTransaction __tx(*this, __n);
for (; __tx.__pos_ != __tx.__new_end_; ++__tx.__pos_) {
__alloc_traits::construct(this->__alloc(), _VSTD::__to_address(__tx.__pos_));
const_pointer __new_end = __tx.__new_end_;
for (pointer __pos = __tx.__pos_; __pos != __new_end; ++__pos, __tx.__pos_ = __pos) {
__alloc_traits::construct(this->__alloc(), _VSTD::__to_address(__pos));
}
}

Expand All @@ -1060,8 +1061,9 @@ void
vector<_Tp, _Allocator>::__construct_at_end(size_type __n, const_reference __x)
{
_ConstructTransaction __tx(*this, __n);
for (; __tx.__pos_ != __tx.__new_end_; ++__tx.__pos_) {
__alloc_traits::construct(this->__alloc(), _VSTD::__to_address(__tx.__pos_), __x);
const_pointer __new_end = __tx.__new_end_;
for (pointer __pos = __tx.__pos_; __pos != __new_end; ++__pos, __tx.__pos_ = __pos) {
__alloc_traits::construct(this->__alloc(), _VSTD::__to_address(__pos), __x);
}
}

Expand Down Expand Up @@ -1753,9 +1755,10 @@ vector<_Tp, _Allocator>::__move_range(pointer __from_s, pointer __from_e, pointe
{
pointer __i = __from_s + __n;
_ConstructTransaction __tx(*this, __from_e - __i);
for (; __i < __from_e; ++__i, ++__tx.__pos_) {
for (pointer __pos = __tx.__pos_; __i < __from_e;
++__i, ++__pos, __tx.__pos_ = __pos) {
__alloc_traits::construct(this->__alloc(),
_VSTD::__to_address(__tx.__pos_),
_VSTD::__to_address(__pos),
_VSTD::move(*__i));
}
}
Expand Down

0 comments on commit 555106a

Please sign in to comment.