[Clang] Universal character names and Assertion failure in Clang 16

[hahn-absint]

With Clang 16 (since https://reviews.llvm.org/D130750), the following code causes an assertion failure, if Clang is build with assertions ("works" if assertions are disabled). It used to work with Clang 15 (with and without assertions), so this seems to be a regression.

void test() {
  int \u0024;
}

Tested on Linux with clang -E <file.c>

[llvmbot]

@llvm/issue-subscribers-clang-frontend

[shafik]

CC @cor3ntin

Confirmed on trunk: https://godbolt.org/z/1oencr6cn

Assertion:

clang++: /root/llvm-project/clang/lib/Lex/Lexer.cpp:1519:
bool isAllowedInitiallyIDChar(uint32_t, const clang::LangOptions&, bool&): 
Assertion `C > 0x7F && "isAllowedInitiallyIDChar called with an ASCII codepoint"' failed.

Backtrace:

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: /opt/compiler-explorer/clang-assertions-trunk/bin/clang++ -gdwarf-4 -g -o /app/output.s -S --gcc-toolchain=/opt/compiler-explorer/gcc-snapshot -fcolor-diagnostics -fno-crash-diagnostics -std=c++20 -E <source>
 #0 0x0000559d8b5dd26f llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x3c7926f)
 #1 0x0000559d8b5dafac llvm::sys::CleanupOnSignal(unsigned long) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x3c76fac)
 #2 0x0000559d8b528118 CrashRecoverySignalHandler(int) CrashRecoveryContext.cpp:0:0
 #3 0x00007f1863932420 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x14420)
 #4 0x00007f18633ff00b raise (/lib/x86_64-linux-gnu/libc.so.6+0x4300b)
 #5 0x00007f18633de859 abort (/lib/x86_64-linux-gnu/libc.so.6+0x22859)
 #6 0x00007f18633de729 (/lib/x86_64-linux-gnu/libc.so.6+0x22729)
 #7 0x00007f18633effd6 (/lib/x86_64-linux-gnu/libc.so.6+0x33fd6)
 #8 0x0000559d8f767987 clang::Lexer::LexUnicodeIdentifierStart(clang::Token&, unsigned int, char const*) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x7e03987)
 #9 0x0000559d8f7695dd clang::Lexer::LexTokenInternal(clang::Token&, bool) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x7e055dd)
#10 0x0000559d8f76ae63 clang::Lexer::Lex(clang::Token&) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x7e06e63)
#11 0x0000559d8f7f083f clang::Preprocessor::Lex(clang::Token&) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x7e8c83f)
#12 0x0000559d8c0a1e07 clang::DoPrintPreprocessedInput(clang::Preprocessor&, llvm::raw_ostream*, clang::PreprocessorOutputOptions const&) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x473de07)
#13 0x0000559d8c072f3d clang::PrintPreprocessedAction::ExecuteAction() (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x470ef3d)
#14 0x0000559d8c0671a9 clang::FrontendAction::Execute() (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x47031a9)
#15 0x0000559d8bfeb766 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x4687766)
#16 0x0000559d8c14b117 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x47e7117)
#17 0x0000559d88b45296 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x11e1296)
#18 0x0000559d88b410ba ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) driver.cpp:0:0
#19 0x0000559d8be5261d void llvm::function_ref<void ()>::callback_fn<clang::driver::CC1Command::Execute(llvm::ArrayRef<std::optional<llvm::StringRef>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, bool*) const::'lambda'()>(long) Job.cpp:0:0
#20 0x0000559d8b528600 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x3bc4600)
#21 0x0000559d8be52edf clang::driver::CC1Command::Execute(llvm::ArrayRef<std::optional<llvm::StringRef>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, bool*) const (.part.0) Job.cpp:0:0
#22 0x0000559d8be1a91c clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&, bool) const (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x44b691c)
#23 0x0000559d8be1b3bd clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*>>&, bool) const (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x44b73bd)
#24 0x0000559d8be2328d clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*>>&) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x44bf28d)
#25 0x0000559d88b43740 clang_main(int, char**, llvm::ToolContext const&) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x11df740)
#26 0x0000559d88a4e605 main (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x10ea605)
#27 0x00007f18633e0083 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24083)
#28 0x0000559d88b3bdce _start (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x11d7dce)
clang++: error: clang frontend command failed with exit code 134 (use -v to see invocation)
Compiler returned: 134

[cor3ntin]

Yep, tryReadUCN accept some basic code points

A universal character name shall not specify a character whose
// short identifier is less than 00A0 other than 0024 ($), 0040 (@), or
// 0060 (`)

This gets fed to LexUnicodeIdentifierStart which assumes a codepoint outside of that range.
It should be pretty easy to fix, although I'm not sure if i';l have the time soon.

we also seem to accept https://godbolt.org/z/Kn7jTjee4 in clang 15 which is wrong because ` is not a valid identifier.
and \u0024 should probably produce an extension warning

[cor3ntin]

Fix https://reviews.llvm.org/D153621