[libc++] Speed up classic locale (take 2)

[ldionne]

[libc++] Speed up classic locale (#73533)

Locale objects use atomic reference counting, which may be very
expensive in parallel applications. The classic locale is used by
default by all streams and can be very contended. But it's never
destroyed, so the reference counting is also completely pointless on the
classic locale. Currently ~70% of time in the parallel stringstream
benchmarks is spent in locale ctor/dtor. And the execution radically
slows down with more threads.

Avoid reference counting on the classic locale. With this change
parallel benchmarks start to scale with threads.

This is a re-application of f8afc53 (aka PR #72112) which was
reverted in 4e0c48b because it broke the sanitizer builds due
to an initialization order fiasco. This issue has now been fixed by
ensuring that the locale is constinit'ed.

                              │   baseline   │    optimized                            │
                              │    sec/op    │    sec/op      vs base                  │
Istream_numbers/0/threads:1      4.672µ ± 0%   4.419µ ± 0%     -5.42% (p=0.000 n=30+39)
Istream_numbers/0/threads:72   539.817µ ± 0%   9.842µ ± 1%    -98.18% (p=0.000 n=30+40)
Istream_numbers/1/threads:1      4.890µ ± 0%   4.750µ ± 0%     -2.85% (p=0.000 n=30+40)
Istream_numbers/1/threads:72     66.44µ ± 1%   10.14µ ± 1%    -84.74% (p=0.000 n=30+40)
Istream_numbers/2/threads:1      4.888µ ± 0%   4.746µ ± 0%     -2.92% (p=0.000 n=30+40)
Istream_numbers/2/threads:72     494.8µ ± 0%   410.2µ ± 1%    -17.11% (p=0.000 n=30+40)
Istream_numbers/3/threads:1      4.697µ ± 0%   4.695µ ± 5%          ~ (p=0.391 n=30+37)
Istream_numbers/3/threads:72     421.5µ ± 7%   421.9µ ± 9%          ~ (p=0.665 n=30)
Ostream_number/0/threads:1       183.0n ± 0%   141.0n ± 2%    -22.95% (p=0.000 n=30)
Ostream_number/0/threads:72    24196.5n ± 1%   343.5n ± 3%    -98.58% (p=0.000 n=30)
Ostream_number/1/threads:1       250.0n ± 0%   196.0n ± 2%    -21.60% (p=0.000 n=30)
Ostream_number/1/threads:72    16260.5n ± 0%   407.0n ± 2%    -97.50% (p=0.000 n=30)
Ostream_number/2/threads:1       254.0n ± 0%   196.0n ± 1%    -22.83% (p=0.000 n=30)
Ostream_number/2/threads:72      28.49µ ± 1%   18.89µ ± 5%    -33.72% (p=0.000 n=30)
Ostream_number/3/threads:1       185.0n ± 0%   185.0n ± 0%      0.00% (p=0.017 n=30)
Ostream_number/3/threads:72      19.38µ ± 4%   19.33µ ± 5%          ~ (p=0.425 n=30)

[llvmbot]

@llvm/pr-subscribers-libcxx

Author: Louis Dionne (ldionne)

Changes

Technically, it is possible for the ios initializers to run before the initializer of classic_locale_imp_, which would cause .emplace to be called on a not-yet-initialized __no_destroy variable.

This patch fixes that by making the variable trivially constinit, since we're not doing anything in its constructor anyway.

---

Full diff: https://github.com/llvm/llvm-project/pull/73533.diff

2 Files Affected:

• (modified) libcxx/include/__utility/no_destroy.h (+7-6)
• (modified) libcxx/src/locale.cpp (+1-1)

diff --git a/libcxx/include/__utility/no_destroy.h b/libcxx/include/__utility/no_destroy.h
index c8581cf9606b571..fc936d2d287316a 100644
--- a/libcxx/include/__utility/no_destroy.h
+++ b/libcxx/include/__utility/no_destroy.h
@@ -28,22 +28,23 @@ struct __uninitialized_tag {};
 // initialization using __emplace.
 template <class _Tp>
 struct __no_destroy {
-  _LIBCPP_HIDE_FROM_ABI explicit __no_destroy(__uninitialized_tag) {}
-  _LIBCPP_HIDE_FROM_ABI ~__no_destroy() {
+  _LIBCPP_CONSTEXPR _LIBCPP_HIDE_FROM_ABI explicit __no_destroy(__uninitialized_tag) {}
+  _LIBCPP_CONSTEXPR _LIBCPP_HIDE_FROM_ABI ~__no_destroy() {
     // nothing
   }
 
   template <class... _Args>
-  _LIBCPP_HIDE_FROM_ABI explicit __no_destroy(_Args&&... __args) : __obj_(std::forward<_Args>(__args)...) {}
+  _LIBCPP_CONSTEXPR _LIBCPP_HIDE_FROM_ABI explicit __no_destroy(_Args&&... __args)
+      : __obj_(std::forward<_Args>(__args)...) {}
 
   template <class... _Args>
-  _LIBCPP_HIDE_FROM_ABI _Tp& __emplace(_Args&&... __args) {
+  _LIBCPP_CONSTEXPR _LIBCPP_HIDE_FROM_ABI _Tp& __emplace(_Args&&... __args) {
     new (&__obj_) _Tp(std::forward<_Args>(__args)...);
     return __obj_;
   }
 
-  _LIBCPP_HIDE_FROM_ABI _Tp& __get() { return __obj_; }
-  _LIBCPP_HIDE_FROM_ABI _Tp const& __get() const { return __obj_; }
+  _LIBCPP_CONSTEXPR _LIBCPP_HIDE_FROM_ABI _Tp& __get() { return __obj_; }
+  _LIBCPP_CONSTEXPR _LIBCPP_HIDE_FROM_ABI _Tp const& __get() const { return __obj_; }
 
 private:
   union {
diff --git a/libcxx/src/locale.cpp b/libcxx/src/locale.cpp
index 2cb75f81b36da33..d6ba490a00a1109 100644
--- a/libcxx/src/locale.cpp
+++ b/libcxx/src/locale.cpp
@@ -546,7 +546,7 @@ locale::__imp::use_facet(long id) const
 // expensive in parallel applications. The classic locale is used by default
 // in all streams. Note: if a new global locale is installed, then we lose
 // the benefit of no reference counting.
-__no_destroy<locale::__imp> locale::__imp::classic_locale_imp_(__uninitialized_tag{}); // initialized below in classic()
+constinit __no_destroy<locale::__imp> locale::__imp::classic_locale_imp_(__uninitialized_tag{}); // initialized below in classic()
 
 const locale& locale::classic() {
   static const __no_destroy<locale> classic_locale(__private_tag{}, [] {

[github-actions]

✅ With the latest revision this PR passed the C/C++ code formatter.

[mordante]

Thanks LGTM!

[ldionne]

Re-purposing this PR to re-apply #72112 which was reverted.