Skip to content

Implement P1885R12: <text_encoding> header #141312

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 61 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
4ecef4f
Implement P1885R12
smallp-o-p May 24, 2025
46758a7
Clang-Format
smallp-o-p May 24, 2025
38c97df
Fix most causes of failing CI
smallp-o-p May 24, 2025
14fad44
Clang-Format again
smallp-o-p May 24, 2025
e247aee
Guard iterator operator* and operator- behind _LIBCPP_ASSERTs
smallp-o-p May 24, 2025
10ac111
Remove useless [[__likely__]]
smallp-o-p May 24, 2025
67efeb8
Replace _NOEXCEPT with noexcept
smallp-o-p May 24, 2025
d2e6307
Remove __CHAR_BIT__ == 8 checks
smallp-o-p May 24, 2025
c64eeeb
Remove __locale_dir include in header, since it would only be require…
smallp-o-p May 24, 2025
cfc02e4
Remove unnecessary _LIBCPP_EXPORTED_FROM_ABI
smallp-o-p May 24, 2025
d5b8d92
Copy id enum specification from standard
smallp-o-p May 24, 2025
e7aaa6b
Remove #ifdef __clang__ since __clang_literal_encoding__ already indi…
smallp-o-p May 24, 2025
95dc46c
Rename __encoding_data members
smallp-o-p May 24, 2025
1660c46
Add precondition asserts for text_encoding constructors
smallp-o-p May 24, 2025
0727a20
Change nullptr sentinel __encoding_data rep value to 9999
smallp-o-p May 24, 2025
b530108
Remove localization guard
smallp-o-p May 24, 2025
3a1ab1d
Move __id_rep outside __encoding_data for convenience
smallp-o-p May 25, 2025
bba8c68
Add name size field to __encoding_data
smallp-o-p May 25, 2025
4a6b5a1
Build string_views explicitly when passing in __encoding_data names
smallp-o-p May 25, 2025
fe54c99
Assert against numbers that are in the range but not part of the id enum
smallp-o-p May 25, 2025
6bd85a0
Remove invalid data pairs from test set
smallp-o-p May 25, 2025
e3da338
Stop aliases() from going before the first alias name
smallp-o-p May 25, 2025
f2aad2f
Remove out-of-bounds tests since they're not allowed and add a ranges…
smallp-o-p May 25, 2025
5fb3748
Make __can_dereference constexpr
smallp-o-p May 25, 2025
89351ba
Don't open ranges namespace
smallp-o-p May 26, 2025
e8abce4
Add nl_langinfo_l to locale base API
smallp-o-p May 27, 2025
e374ebe
Update source file to use internal nl_langinfo_l
smallp-o-p May 27, 2025
75bab0c
Bump CXX_STANDARD to 26 for now
smallp-o-p May 27, 2025
b4db1e7
Remove std version guard in text_encoding.cpp
smallp-o-p May 27, 2025
e37706d
Update __find_encoding_data_by_id comment
smallp-o-p May 27, 2025
25dce4f
Update tests
smallp-o-p May 27, 2025
bb466f4
Annotate environment() function to _LIBCPP_EXPORTED_FROM_ABI
smallp-o-p May 28, 2025
6899ea0
Add text_encoding.cpp to LIBCXX_SOURCES
smallp-o-p May 28, 2025
7b51eca
Format tests
smallp-o-p May 28, 2025
c0d72e0
Undo CXX_STANDARD change
smallp-o-p May 28, 2025
7baa140
Format again
smallp-o-p May 28, 2025
76302e3
Add __encoding_data operator==(string_view) and use std::find to sear…
smallp-o-p May 28, 2025
1456bcb
Remove localization guard from cppm
smallp-o-p May 28, 2025
9c6c8a1
Move locale::encoding() to header
smallp-o-p May 28, 2025
ea13241
Move an include in test_text_encoding.h
smallp-o-p May 28, 2025
29204ee
Break __make_locale into it's own function and call that from environ…
smallp-o-p May 28, 2025
aa8ccb1
Pass __get_locale_encoding to locale::environment() to get the correc…
smallp-o-p May 28, 2025
ed5d697
Fix typos
smallp-o-p May 28, 2025
4c96457
Gate environment() behind _LIBCPP_HAS_LOCALIZATION
smallp-o-p May 29, 2025
c897efb
Clang-Format
smallp-o-p May 29, 2025
4428c2d
Fix style things
smallp-o-p May 29, 2025
eb68ac4
Implement locale::encoding() without __get_locale_encoding() due to m…
smallp-o-p May 29, 2025
9d7e79a
Reorder __encoding_data members to optimize padding, reorder data table
smallp-o-p May 29, 2025
9ae9a2a
Make sure to walk backwards to get primary name when searching by name
smallp-o-p May 29, 2025
bd145a4
No need to walk backwards when building aliases_view since we should …
smallp-o-p May 29, 2025
dfc737f
Update transitive includes
smallp-o-p May 29, 2025
f05e695
Put in a missing ranges include
smallp-o-p May 29, 2025
1117fa3
Properly fill out text_encoding.inc
smallp-o-p May 29, 2025
822acd5
Update environment.pass.cpp to check for the environment's default lo…
smallp-o-p May 29, 2025
5443587
Format environment.pass.cpp
smallp-o-p May 29, 2025
0c50c78
Update header information since text_encoding is no longer guarded by…
smallp-o-p May 29, 2025
699bb00
Add langinfo.h to more of the locale api
smallp-o-p May 29, 2025
9655c38
Rename THOUSANDS_SEP macro due to conflict inside langinfo.h which de…
smallp-o-p May 29, 2025
e1d21b5
Remove stray ;
smallp-o-p May 29, 2025
8e16b48
Add text_encoding test helper
smallp-o-p May 29, 2025
d628940
Regenerate test for feature test macros
smallp-o-p May 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion libcxx/docs/FeatureTestMacroTable.rst
Original file line number Diff line number Diff line change
Expand Up @@ -500,7 +500,7 @@ Status
---------------------------------------------------------- -----------------
``__cpp_lib_submdspan`` *unimplemented*
---------------------------------------------------------- -----------------
``__cpp_lib_text_encoding`` *unimplemented*
``__cpp_lib_text_encoding`` ``202306L``
---------------------------------------------------------- -----------------
``__cpp_lib_to_chars`` *unimplemented*
---------------------------------------------------------- -----------------
Expand Down
4 changes: 2 additions & 2 deletions libcxx/docs/Status/Cxx2cPapers.csv
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"`P2013R5 <https://wg21.link/P2013R5>`__","Freestanding Language: Optional ``::operator new``","2023-06 (Varna)","","",""
"`P2363R5 <https://wg21.link/P2363R5>`__","Extending associative containers with the remaining heterogeneous overloads","2023-06 (Varna)","","",""
"`P1901R2 <https://wg21.link/P1901R2>`__","Enabling the Use of ``weak_ptr`` as Keys in Unordered Associative Containers","2023-06 (Varna)","","",""
"`P1885R12 <https://wg21.link/P1885R12>`__","Naming Text Encodings to Demystify Them","2023-06 (Varna)","","",""
"`P1885R12 <https://wg21.link/P1885R12>`__","Naming Text Encodings to Demystify Them","2023-06 (Varna)","|Complete|","21",""
"`P0792R14 <https://wg21.link/P0792R14>`__","``function_ref``: a type-erased callable reference","2023-06 (Varna)","","",""
"`P2874R2 <https://wg21.link/P2874R2>`__","P2874R2: Mandating Annex D Require No More","2023-06 (Varna)","|Complete|","12",""
"`P2757R3 <https://wg21.link/P2757R3>`__","Type-checking format args","2023-06 (Varna)","","",""
Expand Down Expand Up @@ -79,7 +79,7 @@
"`P3136R1 <https://wg21.link/P3136R1>`__","Retiring niebloids","2024-11 (Wrocław)","|Complete|","14",""
"`P3138R5 <https://wg21.link/P3138R5>`__","``views::cache_latest``","2024-11 (Wrocław)","","",""
"`P3379R0 <https://wg21.link/P3379R0>`__","Constrain ``std::expected`` equality operators","2024-11 (Wrocław)","|Complete|","21",""
"`P2862R1 <https://wg21.link/P2862R1>`__","``text_encoding::name()`` should never return null values","2024-11 (Wrocław)","","",""
"`P2862R1 <https://wg21.link/P2862R1>`__","``text_encoding::name()`` should never return null values","2024-11 (Wrocław)","|Complete|","21",""
"`P2897R7 <https://wg21.link/P2897R7>`__","``aligned_accessor``: An ``mdspan`` accessor expressing pointer over-alignment","2024-11 (Wrocław)","|Complete|","21",""
"`P3355R1 <https://wg21.link/P3355R1>`__","Fix ``submdspan`` for C++26","2024-11 (Wrocław)","","",""
"`P3222R0 <https://wg21.link/P3222R0>`__","Fix C++26 by adding transposed special cases for P2642 layouts","2024-11 (Wrocław)","","",""
Expand Down
3 changes: 3 additions & 0 deletions libcxx/include/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -751,6 +751,8 @@ set(files
__system_error/error_condition.h
__system_error/system_error.h
__system_error/throw_system_error.h
__text_encoding/get_locale_encoding.h
__text_encoding/text_encoding.h
__thread/formatter.h
__thread/id.h
__thread/jthread.h
Expand Down Expand Up @@ -1062,6 +1064,7 @@ set(files
strstream
syncstream
system_error
text_encoding
tgmath.h
thread
tuple
Expand Down
28 changes: 28 additions & 0 deletions libcxx/include/__locale
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@
# include <cstddef>
# include <cstring>

# if _LIBCPP_STD_VER >= 26
# include <__text_encoding/text_encoding.h>
# endif

# if _LIBCPP_HAS_WIDE_CHARACTERS
# include <cwchar>
# else
Expand Down Expand Up @@ -99,6 +103,30 @@ public:

// locale operations:
string name() const;

# if _LIBCPP_STD_VER >= 26
_LIBCPP_HIDE_FROM_ABI text_encoding encoding() const {
std::string __name = this->name();
if (__name.size() == 1) {
if (__name[0] == 'C')
return std::text_encoding(text_encoding::id::ASCII);
if (__name[0] == '*')
return std::text_encoding();
}

auto __encoding_divider = __name.find('.'), __modifier_at = __name.find('@');
if (__encoding_divider == std::string::npos) {
return std::text_encoding();
}

if (__modifier_at == std::string::npos) {
return std::text_encoding(__name.c_str() + __encoding_divider + 1);
}

return std::text_encoding({__name.c_str() + __encoding_divider + 1, __modifier_at - 1 - __encoding_divider});
}
# endif // _LIBCPP_STD_VER >= 26

bool operator==(const locale&) const;
# if _LIBCPP_STD_VER <= 17
_LIBCPP_HIDE_FROM_ABI bool operator!=(const locale& __y) const { return !(*this == __y); }
Expand Down
9 changes: 9 additions & 0 deletions libcxx/include/__locale_dir/locale_base_api.h
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,9 @@
# if _LIBCPP_HAS_WIDE_CHARACTERS
# include <wctype.h>
# endif
# if __has_include(<langinfo.h>)
# include <langinfo.h>
# endif
_LIBCPP_BEGIN_NAMESPACE_STD
namespace __locale {
//
Expand Down Expand Up @@ -179,6 +182,12 @@ inline _LIBCPP_HIDE_FROM_ABI char* __setlocale(int __category, char const* __loc
inline _LIBCPP_HIDE_FROM_ABI void __freelocale(__locale_t __loc) { freelocale(__loc); }

inline _LIBCPP_HIDE_FROM_ABI __lconv_t* __localeconv(__locale_t& __loc) { return __libcpp_localeconv_l(__loc); }

# if __has_include(<langinfo.h>)
inline _LIBCPP_HIDE_FROM_ABI char* __nl_langinfo_l(int __category_mask, __locale_t __l) {
return ::nl_langinfo_l(__category_mask, __l);
}
# endif
# endif // _LIBCPP_BUILDING_LIBRARY

//
Expand Down
10 changes: 10 additions & 0 deletions libcxx/include/__locale_dir/locale_base_api/ibm.h
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@
#include <stdarg.h>
#include <stdio.h>

#if __has_include(<langinfo.h>)
# include <langinfo.h>
#endif

#include "cstdlib"

#if defined(__MVS__)
Expand Down Expand Up @@ -105,4 +109,10 @@ _LIBCPP_ATTRIBUTE_FORMAT(__printf__, 2, 0) int vasprintf(char** strp, const char
return str_size;
}

#if __has_include(<langinfo.h>)
inline _LIBCPP_HIDE_FROM_ABI char* __nl_langinfo_l(int __category_mask, __locale_t __l) {
return ::nl_langinfo_l(__category_mask, __l);
}
#endif

#endif // _LIBCPP___LOCALE_DIR_LOCALE_BASE_API_IBM_H
11 changes: 10 additions & 1 deletion libcxx/include/__locale_dir/support/bsd_like.h
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,9 @@
# include <wchar.h>
# include <wctype.h>
#endif

#if __has_include(<langinfo.h>)
# include <langinfo.h>
#endif
#include <xlocale.h>

#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
Expand Down Expand Up @@ -60,6 +62,13 @@ inline _LIBCPP_HIDE_FROM_ABI char* __setlocale(int __category, char const* __loc
}

inline _LIBCPP_HIDE_FROM_ABI __lconv_t* __localeconv(__locale_t& __loc) { return ::localeconv_l(__loc); }

# if __has_include(<langinfo.h>)
inline _LIBCPP_HIDE_FROM_ABI char* __nl_langinfo_l(int __category_mask, __locale_t __l) {
return ::nl_langinfo_l(__category_mask, __l);
}
# endif

#endif // _LIBCPP_BUILDING_LIBRARY

//
Expand Down
10 changes: 10 additions & 0 deletions libcxx/include/__locale_dir/support/fuchsia.h
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@
#include <cstdlib>
#include <cwchar>

#if __has_include(<langinfo.h>)
# include <langinfo.h>
#endif

#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
# pragma GCC system_header
#endif
Expand Down Expand Up @@ -69,6 +73,12 @@ inline _LIBCPP_HIDE_FROM_ABI __lconv_t* __localeconv(__locale_t& __loc) {
return std::localeconv();
}

# if __has_include(<langinfo.h>)
inline _LIBCPP_HIDE_FROM_ABI char* __nl_langinfo_l(int __category_mask, __locale_t __l) {
return ::nl_langinfo_l(__category_mask, __l);
}
# endif

//
// Other functions
//
Expand Down
10 changes: 10 additions & 0 deletions libcxx/include/__locale_dir/support/linux.h
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@
# include <cwchar>
# include <wctype.h>
#endif
#if __has_include(<langinfo.h>)
# include <langinfo.h>
#endif

#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
# pragma GCC system_header
Expand Down Expand Up @@ -77,6 +80,13 @@ inline _LIBCPP_HIDE_FROM_ABI __lconv_t* __localeconv(__locale_t& __loc) {
__locale_guard __current(__loc);
return std::localeconv();
}

# if __has_include(<langinfo.h>)
inline _LIBCPP_HIDE_FROM_ABI char* __nl_langinfo_l(int __category_mask, __locale_t __l) {
return ::nl_langinfo_l(__category_mask, __l);
}
# endif

#endif // _LIBCPP_BUILDING_LIBRARY

//
Expand Down
29 changes: 29 additions & 0 deletions libcxx/include/__text_encoding/get_locale_encoding.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
//===----------------------------------------------------------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//

#ifndef _LIBCPP___GET_LOCALE_ENCODING_H
#define _LIBCPP___GET_LOCALE_ENCODING_H

#include <__config>

#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
# pragma GCC system_header
#endif

#if _LIBCPP_HAS_LOCALIZATION

# include <string_view>

_LIBCPP_BEGIN_NAMESPACE_STD
string_view _LIBCPP_EXPORTED_FROM_ABI __get_locale_encoding(const char* __name);

_LIBCPP_END_NAMESPACE_STD

#endif // _LIBCPP_HAS_LOCALIZATION

#endif // _LIBCPP___GET_LOCALE_ENCODING_H
Loading
Loading