Fix issue #22535 - [REG 2.112] Unicode symbols inside q

[rikkimax]

No description provided.

[dlang-bot]

Thanks for your pull request and interest in making D better, @rikkimax! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.
Please verify that your PR follows this checklist:

• My PR is fully covered with tests (you can see the coverage diff by visiting the details link of the codecov check)
• My PR is as minimal as possible (smaller, focused PRs are easier to review than big ones)
• I have provided a detailed rationale explaining my changes
• New or modified functions have Ddoc comments (with Params: and Returns:)

Please see CONTRIBUTING.md for more information.

---

If you have addressed all reviews or aren't sure how to proceed, don't hesitate to ping us with a simple comment.

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

⚠️⚠️⚠️ Warnings ⚠️⚠️⚠️

• In preparation for migrating from Bugzilla to GitHub Issues, the issue reference syntax has changed. Please add the word "Bugzilla" to issue references. For example, Fix Bugzilla Issue 12345 or Fix Bugzilla 12345.(Reminder: the edit needs to be done in the Git commit message, not the GitHub pull request.)

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub run digger -- build "stable + dmd#22539"

[thewilsonator]

Is there a test case for this?

[ibuclaw]

IIUC

assert(q"EOS
是的0
EOS" == "是的0");

Or one can cast to int[] and assert each index has the correct numerical value.

[rikkimax]

Guess I haven't improved stable urgh.

[rikkimax]

Swapping the string buffers from identifier getting its own, to a secondary one for q strings

[rikkimax]

Okay I'm getting rather sick of this.

 ... dshell/cpp_header_gen.d
 !!! dshell/cpp_header_gen.d is disabled!
 ... dshell/defaults.d
 ... dshell/depsprot.d
 ... dshell/dll.d
 !!! dshell/dll.d is disabled!
 ... dshell/dll_cxx.d
 !!! dshell/dll_cxx.d is disabled!
 ... dshell/dwarf.d
 !!! dshell/dwarf.d is disabled!
 ... dshell/issue20444_SOURCE_DATE_EPOCH.d
 ... dshell/issue22804.d
 ... dshell/issue22816.d
 ... dshell/issue22817.d
 ... dshell/issue24111.d
 ... dshell/linker_flag_with_spaces.d
 ... dshell/sameenv.d
 ... dshell/test6952.d
 !!! dshell/test6952.d is disabled!
 ... dshell/test9377.d
 ... dshell/test_shared.d
 !!! dshell/test_shared.d is disabled!

From what I can tell, the dshell tests should've completed, and had a good 10m+ left, but the phobos ones don't start.

@rainers any ideas?

I've backported a bunch of PR's at this point and tried to hunt down more.

[thewilsonator]

silly question: why are these __gshared in the first place?

[rikkimax]

Who knows, it'll be faster than TLS.

[Herringway]

I'd be surprised if there was a meaningful difference in performance.

[thewilsonator]

I mean why are they not a regular member of the lexer? We only have one instance of the lexer at one time anyway, right?

[denizzzka]

@thewilsonator thread-local variables may be slightly slower than __gshared due to additional pointer dereference (ARM and RISC-V isn't affected by this, but x86 - maybe)
I guess it was important at one time

[ibuclaw]

The compiler is single threaded, no need for tls.

[denizzzka]

@ibuclaw seems, it will still give a performance penalty, only if the key like --fthread-model=initial-exec is not used for the compiler build

(I think these are all minor details, and for code readability and to avoid errors it's worth simply removing __gshared from here)

[Herringway]

The compiler is single threaded, no need for tls.

What about dmd-as-a-library?

[rikkimax]

There are lot of places where globals are used in the frontend.

An audit would need to take place, and a discussion with Walter.

I won't be changing this in this PR, this isn't the right PR for it.

[ibuclaw]

The compiler is single threaded, no need for tls.

What about dmd-as-a-library?

GDC and LDC use dmd as a library.

Consider that mingw and darwin are emutls targets as well.

[rikkimax]

Okay dshell tests are passing, so its something to do with either the run.d, phobos~stable, ldc, or visuald.

Will try bumping ldc and visuald first.

[rikkimax]

Ah huh!

Azure didn't tell me what failed the CI: runnable\lexer.d

Joy.

[rikkimax]

FOUND IT

FOUND IT

FOUND IT

A VARIANT OF LIBC isalpha CANNOT ACCEPT UNICODE >= 0x80

SWEAR WORDS GO HERE

[rikkimax]

The behavior of isalpha and _isalpha_l is undefined if c isn't EOF or in the range 0 through 0xFF, inclusive. When a debug CRT library is used and c isn't one of these values, the functions raise an assertion.

[rikkimax]

And its green, finally.