WG14 N2653: char8_t: A type for UTF-8 characters and strings

[tahonermann]

Proposals:

• http://wg21.link/p0482
• http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2231.htm

Reference implementation:

• https://github.com/tahonermann/gcc/tree/char8_t

[WPMGPRoSToTeMa]

What about adding std::to_chars/std::from_chars support for char8_t? (only for Basic Latin of course)

[tahonermann]

What about adding std::to_chars/std::from_chars support for char8_t? (only for Basic Latin of course)

I'd like to consider that a separate issue from this one (which I'm going to close now since we adopted char8_t for C++20). Regardless, I don't see the connection to std::to_chars and std::from_chars; those interfaces exist for conversion of (floating point) values to string representation. We do need interfaces for transcoding, but don't have any proposals in front of us yet.,

[tahonermann]

Actually, I'm not going to close this issue as it tracks updating both C and C++. We're halfway there...

[tahonermann]

Minutes from the WG14 meeting in Brno, April 23-27th, 2018 in which the char8_t proposal for C (N2231) was discussed are available at:

• http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2239.pdf

[DBJDBJ]

Sorry Tom that was 2018, not 2008 ... WG14 cogitations are long but not that long :)

[tahonermann]

@DBJDBJ, thanks, comment fixed.

[MarcusJohnson91]

So will char8_t be available in C, and do you know what macro would define it's existence (since it isn't in uchar.h like char16_t and char32_t)?

obviously __cpp_char8_t won't be C's macro, right?

[tahonermann]

Hi @MarcusJohnson91. I'm working on completing an implementation of N2231 (char8_t for C) for gcc and glibc now. Once done, I'll submit a revision of N2231 to the C committee to consider adopting and will follow up with them.

obviously __cpp_char8_t won't be C's macro, right?

Correct. I've been planning to use __STDC_CHAR8_T for the feature test macro name for C. This name matches the conventions used for other C feature test macros.

[tahonermann]

N2653 (char8_t: A type for UTF-8 characters and strings (Revision 1)) has now been submitted to WG14 as a revision of N2231.

Reference implementations for gcc and libstdc++ are available here and for glibc here.

Patches have been submitted to:

• gcc:
  • [PATCH 0/3]: C N2653 char8_t implementation
  • [PATCH 1/3]: C N2653 char8_t: Language support
  • [PATCH 2/3]: C N2653 char8_t: New tests​
  • [PATCH 3/3]: C N2653 char8_t: Documentation updates
• libstdc++:
  • [PATCH] C++ P0482R6 char8_t: declare std::c8rtomb and std::mbrtoc8 if provided by the C library
• glibc:
  • [PATCH 0/3]: C++20 P0482R6 and C2X N2653: support for char8_t, mbrtoc8(), and c8rtomb()
  • [PATCH 1/3]: C++20 P0482R6 and C2X N2653: Fix for bug 25744, mbrtowc with Big5-HKSCS
  • [PATCH 2/3]: C++20 P0482R6 and C2X N2653: Implement mbrtoc8, c8rtomb, char8_t
  • [PATCH 3/3]: C++20 P0482R6 and C2X N2653: Tests for mbrtoc8, c8rtomb, char8_t

[MarcusJohnson91]

Hey @tahonermann Has char8_t been adopted by WG14?

Reading draft N2731, and I'm not seeing it :(

[tahonermann]

@MarcusJohnson91, no, not yet. I expect it to be discussed at the next WG14 meeting.

[MarcusJohnson91]

Hey @tahonermann, my proposal for new length modifiers needs to be finished like ASAP and it sounds like we'll present at the same meeting.

I'm creating the length modifiers U16 and U32 for char16_t and char32_t characters and strings.

if you're planning on adding something like that for char8_t, I would use U8 to fit in, basically c/s/lc/ls are for locale specific conversions, and the U16/U32/U8? variants are for Unicode.

Just a heads up.

I can't wait for your paper to get in!

[tahonermann]

Hi, @MarcusJohnson91, I have yet to review your proposal. I’m curious how it deals with encoding issues. It may be worth discussing it in an SG16 telecon as we would presumably want to align behavior with a hypothetical future std::format() overload or formatter that supports the UTF char variants.

As for your proposal supporting char8_t, that is something we can reconcile if both proposals are accepted.

[MarcusJohnson91]

I've submitted a new revision of my proposal November 26th, but it hasn't been published yet.

the original proposal is N2761; the main difference is I changed it from l16/l32 to U16/U32, and I clarified that precision and width specifiers operate on Codepoints, not Codeunits for security reasons.

I believe Robert suggested I add support for char8_t, and in my draft I did specify U8 for char8_t types, but when I looked in the paper and saw that char8_t wasn't in yet, I dropped it from the draft.

[tahonermann]

N2653 was accepted for C2x during the WG14 meeting held in late January and early February of 2022. An updated working paper has not yet been published but should be forthcoming. Closing.