fs.path: Fix Windows path component comparison being ASCII-only

[squeek502]

We can use RtlEqualUnicodeString via eqlIgnoreCaseWTF16 to get Unicode-aware case insensitive path component comparison (but need to convert the components to UTF-16 first)

See #16100 (comment) and #16100 (comment) for an updated description of how this is implemented.

Addresses some TODOs I noticed while looking at the path.zig code. For some more context about RtlEqualUnicodeString, see #10595.

[squeek502]

Is std.fs.path.relativeWindows meant to be callable from non-Windows platforms? Currently that seems to be the case since it's marked pub and the test case runs both relativeWindows and relativePosix on all platforms.

If we want to keep that being true, then this particular fix can't be applied since it means relativeWindows will depend on ntdll.

[andrewrk]

Yes it is meant to be. Relying on RtlEqualUnicodeString seems problematic in general because:

• If it depends on global OS state, then it should not be used in most cases that std.fs.path.relativeWindows is used.
• If it does not depend on global OS state, then there is no reason for it to be in a separate DLL since that pure functional logic could be implemented in zig std lib and provide better insight as to what is possible, be callable at compile-time, be callable from other host operating systems, and in general avoid a DLL call for simple string manipulation.

I understand that in practice we may need to use it regardless, and I would value your recommendation in particular.

Are you sure that eqlIgnoreCaseWTF16 cannot be implemented without a call to RtlEqualUnicodeString?

[squeek502]

Are you sure that eqlIgnoreCaseWTF16 cannot be implemented without a call to RtlEqualUnicodeString?

It's definitely possible to implement on its own, but AFAIK that would involve pulling in Unicode data into the standard library which is something I've seen you mention a few times that you want to avoid. How much Unicode data it would need I'm unsure about (I think RtlEqualUnicodeString only does direct UTF-16 code unit comparison, so it might not do any normalization which may make the amount of Unicode data needed smaller than would be needed for full Unicode-aware case insensitive comparison; AFAIK the relevant Unicode data used by RtlEqualUnicodeString is within PEB.UnicodeCaseTableData but I'm unsure how large that data is in memory)

From 8bf7cff:

It is not planned to support full Unicode case-insensitivity on Windows, and in fact relying on non-ASCII case-insensitive environment variables is fundamentally problematic.

From #5772 (comment):

This looks like it adds a dependency on Unicode data, which needs to go through the approval process before having a PR accepted for it. So far zig std lib does not have a dependency on any of the non-static components of unicode data (e.g. stuff that can change in future unicode updates). There have been several proposals rejected to add a dependency of the language and standard library on this.

See #10796 (comment) for where this came up last time and it was decided that RtlEqualUnicodeString was acceptable (but in that instance it's used by an API that's Windows-only).

[andrewrk]

If it's only comparing UTF-16 codepoints, that's super easy to implement and does not require any unicode data, right? I think I'm missing something.

[squeek502]

It needs a table of lowercase <-> uppercase conversions (only one direction is strictly necessary though).

[andrewrk]

For the ASCII range only? Or do they keep their unicode data up-to-date with standards changes? Wouldn't it be a problem if the unicode consortium changed the rule of whether something is uppercase or lowercase and caused files to now alias that didn't before?

[squeek502]

For the ASCII range only?

For the range of U+0000 to U+FFFF.

I haven't investigated it too much, but I do know that codepoints that need to be encoded as a UTF-16 surrogate pair (>= U+010000) do not actually get compared case-insensitively. See ReactOS's implementation of RtlCompareUnicodeString which just compares the uppercased version u16-to-u16 (rather than codepoint-to-codepoint):

https://github.com/reactos/reactos/blob/3fa57b8ff7fcee47b8e2ed869aecaf4515603f3f/sdk/lib/rtl/unicode.c#L2197-L2200

My (total, with no basis) guess is that:

• The data in PEB.UnicodeCaseTableData changes rarely, probably only on major Windows version updates if at all
• The data in PEB.UnicodeCaseTableData is probably perennially out-of-date and maybe locked to an older version of the Unicode standard

If my guess is correct, then this might add some extra challenge if the goal is to match Windows behavior since that may be a moving target separate from the Unicode standard (but would need to introspect into the UnicodeCaseTableData on different Windows versions and then compare them with eachother and to the Unicode standard to know for sure)

[squeek502]

Just to make things clearer, the use of RtlEqualUnicodeString allows these two UNC paths to be seen as the same from the perspective of std.fs.path.relativeWindows:

\\кириллица\ελληνικά\português

\\КИРИЛЛИЦА\ΕΛΛΗΝΙΚΆ\PORTUGUÊS

[e4m2]

AFAICT the NLS Unicode tables are determined as follows (may be wrong, though):

1. Query the HKLM\SYSTEM\CurrentControlSet\Control\Nls\Language key for the value named Default.
2. Query the same key for the string data of Default. If it exists, the filename is the newly retrieved value, otherwise the filename is l_intl.nls.
3. Read the NLS file from %SystemRoot%\System32\{filename}. This is then stashed in the loader block and eventually plumbed through to the PEB on process creation.

For reference, l_intl.nls is 9926 bytes on Windows 11 22H2. I'm not sure how it changes across different Windows versions or in what situation the non-default (that is, Default - as in not l_intl.nls) file would be chosen.
Either way, it seems the exact Windows behavior would be pretty difficult or even impossible to replicate in a crossplatform manner. There's probably a good enough compromise solution, but in any case a few kB would still have to be spared for the Unicode data, that's just unavoidable.

[e4m2]

Also it should be noted the actual table used for filesystem operations is the special NTFS file $UpCase, which is just a copy of the system-wide NLS data at the time the volume is created. This can have some interesting consequences (see also the comments there). How should Zig handle this?

[squeek502]

Apparently the $UpCase/l_intl.nls data hasn't changed since Windows 8.1 (see the comments of the article posted previously by @e4m2; unsure about Windows 11, though), so it might be okay to use the current version of the data for now since Zig only officially supports Windows 8.1+.

I went ahead and wrote some code that generates a switch case for all lowercase -> uppercase u16 conversions using the data from PEB.UnicodeCaseTableData (should be the same as $UpCase/l_intl.nls for most setups AFAICT, I used PEB instead of reading l_intl.nls or just using RtlUpcaseUnicodeChar because I was curious about the PEB data):

Code to generate the function with the switch

const std = @import("std");

pub fn main() !void {
    var stdout = std.io.getStdOut();
    var buffered_stdout = std.io.bufferedWriter(stdout.writer());
    var stdout_writer = buffered_stdout.writer();

    var peb = std.os.windows.peb();
    const base_case_table_data = @ptrCast([*]u16, @alignCast(@alignOf(u16), peb.UnicodeCaseTableData));
    // See https://github.com/reactos/reactos/blob/3fa57b8ff7fcee47b8e2ed869aecaf4515603f3f/sdk/lib/rtl/nls.c#L222
    const uppercase_table_data = @ptrCast([*]u16, &base_case_table_data[2]);

    var i: u16 = 0;
    try stdout_writer.writeAll("pub fn upcase(c: u16) u16 {\n    return switch (c) {\n");
    while (true) : (i += 1) {
        const upper = upcase(i, uppercase_table_data);
        if (upper != i) {
            try stdout_writer.print("        0x{X} => 0x{X},\n", .{ i, upper });
        }
        if (i == std.math.maxInt(u16)) break;
    }
    try stdout_writer.writeAll("        else => c,\n    };\n}\n");

    try buffered_stdout.flush();
}

/// Based on the ReactOS implementation of RtlpUpcaseUnicodeChar
/// https://github.com/reactos/reactos/blob/3fa57b8ff7fcee47b8e2ed869aecaf4515603f3f/sdk/lib/rtl/nls.c#L714-L735
fn upcase(c: u16, upper_case_table: [*]u16) u16 {
    if (c < 'a') return c;
    if (c <= 'z') return c - ('a' - 'A');

    var offset: u16 = (c >> 8) & 0xFF;
    offset = upper_case_table[offset];

    offset += (c >> 4) & 0xF;
    offset = upper_case_table[offset];

    offset += c & 0xF;
    offset = upper_case_table[offset];

    // relies on wrapping overflow, e.g. the final offset could be 0xFFFF
    return c +% offset;
}

The generated function with the switch

pub fn upcase(c: u16) u16 {
    return switch (c) {
        0x61 => 0x41,
        0x62 => 0x42,
        0x63 => 0x43,
        0x64 => 0x44,
        0x65 => 0x45,
        0x66 => 0x46,
        0x67 => 0x47,
        0x68 => 0x48,
        0x69 => 0x49,
        0x6A => 0x4A,
        0x6B => 0x4B,
        0x6C => 0x4C,
        0x6D => 0x4D,
        0x6E => 0x4E,
        0x6F => 0x4F,
        0x70 => 0x50,
        0x71 => 0x51,
        0x72 => 0x52,
        0x73 => 0x53,
        0x74 => 0x54,
        0x75 => 0x55,
        0x76 => 0x56,
        0x77 => 0x57,
        0x78 => 0x58,
        0x79 => 0x59,
        0x7A => 0x5A,
        0xE0 => 0xC0,
        0xE1 => 0xC1,
        0xE2 => 0xC2,
        0xE3 => 0xC3,
        0xE4 => 0xC4,
        0xE5 => 0xC5,
        0xE6 => 0xC6,
        0xE7 => 0xC7,
        0xE8 => 0xC8,
        0xE9 => 0xC9,
        0xEA => 0xCA,
        0xEB => 0xCB,
        0xEC => 0xCC,
        0xED => 0xCD,
        0xEE => 0xCE,
        0xEF => 0xCF,
        0xF0 => 0xD0,
        0xF1 => 0xD1,
        0xF2 => 0xD2,
        0xF3 => 0xD3,
        0xF4 => 0xD4,
        0xF5 => 0xD5,
        0xF6 => 0xD6,
        0xF8 => 0xD8,
        0xF9 => 0xD9,
        0xFA => 0xDA,
        0xFB => 0xDB,
        0xFC => 0xDC,
        0xFD => 0xDD,
        0xFE => 0xDE,
        0xFF => 0x178,
        0x101 => 0x100,
        0x103 => 0x102,
        0x105 => 0x104,
        0x107 => 0x106,
        0x109 => 0x108,
        0x10B => 0x10A,
        0x10D => 0x10C,
        0x10F => 0x10E,
        0x111 => 0x110,
        0x113 => 0x112,
        0x115 => 0x114,
        0x117 => 0x116,
        0x119 => 0x118,
        0x11B => 0x11A,
        0x11D => 0x11C,
        0x11F => 0x11E,
        0x121 => 0x120,
        0x123 => 0x122,
        0x125 => 0x124,
        0x127 => 0x126,
        0x129 => 0x128,
        0x12B => 0x12A,
        0x12D => 0x12C,
        0x12F => 0x12E,
        0x133 => 0x132,
        0x135 => 0x134,
        0x137 => 0x136,
        0x13A => 0x139,
        0x13C => 0x13B,
        0x13E => 0x13D,
        0x140 => 0x13F,
        0x142 => 0x141,
        0x144 => 0x143,
        0x146 => 0x145,
        0x148 => 0x147,
        0x14B => 0x14A,
        0x14D => 0x14C,
        0x14F => 0x14E,
        0x151 => 0x150,
        0x153 => 0x152,
        0x155 => 0x154,
        0x157 => 0x156,
        0x159 => 0x158,
        0x15B => 0x15A,
        0x15D => 0x15C,
        0x15F => 0x15E,
        0x161 => 0x160,
        0x163 => 0x162,
        0x165 => 0x164,
        0x167 => 0x166,
        0x169 => 0x168,
        0x16B => 0x16A,
        0x16D => 0x16C,
        0x16F => 0x16E,
        0x171 => 0x170,
        0x173 => 0x172,
        0x175 => 0x174,
        0x177 => 0x176,
        0x17A => 0x179,
        0x17C => 0x17B,
        0x17E => 0x17D,
        0x180 => 0x243,
        0x183 => 0x182,
        0x185 => 0x184,
        0x188 => 0x187,
        0x18C => 0x18B,
        0x192 => 0x191,
        0x195 => 0x1F6,
        0x199 => 0x198,
        0x19A => 0x23D,
        0x19E => 0x220,
        0x1A1 => 0x1A0,
        0x1A3 => 0x1A2,
        0x1A5 => 0x1A4,
        0x1A8 => 0x1A7,
        0x1AD => 0x1AC,
        0x1B0 => 0x1AF,
        0x1B4 => 0x1B3,
        0x1B6 => 0x1B5,
        0x1B9 => 0x1B8,
        0x1BD => 0x1BC,
        0x1BF => 0x1F7,
        0x1C6 => 0x1C4,
        0x1C9 => 0x1C7,
        0x1CC => 0x1CA,
        0x1CE => 0x1CD,
        0x1D0 => 0x1CF,
        0x1D2 => 0x1D1,
        0x1D4 => 0x1D3,
        0x1D6 => 0x1D5,
        0x1D8 => 0x1D7,
        0x1DA => 0x1D9,
        0x1DC => 0x1DB,
        0x1DD => 0x18E,
        0x1DF => 0x1DE,
        0x1E1 => 0x1E0,
        0x1E3 => 0x1E2,
        0x1E5 => 0x1E4,
        0x1E7 => 0x1E6,
        0x1E9 => 0x1E8,
        0x1EB => 0x1EA,
        0x1ED => 0x1EC,
        0x1EF => 0x1EE,
        0x1F3 => 0x1F1,
        0x1F5 => 0x1F4,
        0x1F9 => 0x1F8,
        0x1FB => 0x1FA,
        0x1FD => 0x1FC,
        0x1FF => 0x1FE,
        0x201 => 0x200,
        0x203 => 0x202,
        0x205 => 0x204,
        0x207 => 0x206,
        0x209 => 0x208,
        0x20B => 0x20A,
        0x20D => 0x20C,
        0x20F => 0x20E,
        0x211 => 0x210,
        0x213 => 0x212,
        0x215 => 0x214,
        0x217 => 0x216,
        0x219 => 0x218,
        0x21B => 0x21A,
        0x21D => 0x21C,
        0x21F => 0x21E,
        0x223 => 0x222,
        0x225 => 0x224,
        0x227 => 0x226,
        0x229 => 0x228,
        0x22B => 0x22A,
        0x22D => 0x22C,
        0x22F => 0x22E,
        0x231 => 0x230,
        0x233 => 0x232,
        0x23C => 0x23B,
        0x242 => 0x241,
        0x247 => 0x246,
        0x249 => 0x248,
        0x24B => 0x24A,
        0x24D => 0x24C,
        0x24F => 0x24E,
        0x250 => 0x2C6F,
        0x251 => 0x2C6D,
        0x253 => 0x181,
        0x254 => 0x186,
        0x256 => 0x189,
        0x257 => 0x18A,
        0x259 => 0x18F,
        0x25B => 0x190,
        0x260 => 0x193,
        0x263 => 0x194,
        0x268 => 0x197,
        0x269 => 0x196,
        0x26B => 0x2C62,
        0x26F => 0x19C,
        0x271 => 0x2C6E,
        0x272 => 0x19D,
        0x275 => 0x19F,
        0x27D => 0x2C64,
        0x280 => 0x1A6,
        0x283 => 0x1A9,
        0x288 => 0x1AE,
        0x289 => 0x244,
        0x28A => 0x1B1,
        0x28B => 0x1B2,
        0x28C => 0x245,
        0x292 => 0x1B7,
        0x371 => 0x370,
        0x373 => 0x372,
        0x377 => 0x376,
        0x37B => 0x3FD,
        0x37C => 0x3FE,
        0x37D => 0x3FF,
        0x3AC => 0x386,
        0x3AD => 0x388,
        0x3AE => 0x389,
        0x3AF => 0x38A,
        0x3B1 => 0x391,
        0x3B2 => 0x392,
        0x3B3 => 0x393,
        0x3B4 => 0x394,
        0x3B5 => 0x395,
        0x3B6 => 0x396,
        0x3B7 => 0x397,
        0x3B8 => 0x398,
        0x3B9 => 0x399,
        0x3BA => 0x39A,
        0x3BB => 0x39B,
        0x3BC => 0x39C,
        0x3BD => 0x39D,
        0x3BE => 0x39E,
        0x3BF => 0x39F,
        0x3C0 => 0x3A0,
        0x3C1 => 0x3A1,
        0x3C3 => 0x3A3,
        0x3C4 => 0x3A4,
        0x3C5 => 0x3A5,
        0x3C6 => 0x3A6,
        0x3C7 => 0x3A7,
        0x3C8 => 0x3A8,
        0x3C9 => 0x3A9,
        0x3CA => 0x3AA,
        0x3CB => 0x3AB,
        0x3CC => 0x38C,
        0x3CD => 0x38E,
        0x3CE => 0x38F,
        0x3D7 => 0x3CF,
        0x3D9 => 0x3D8,
        0x3DB => 0x3DA,
        0x3DD => 0x3DC,
        0x3DF => 0x3DE,
        0x3E1 => 0x3E0,
        0x3E3 => 0x3E2,
        0x3E5 => 0x3E4,
        0x3E7 => 0x3E6,
        0x3E9 => 0x3E8,
        0x3EB => 0x3EA,
        0x3ED => 0x3EC,
        0x3EF => 0x3EE,
        0x3F2 => 0x3F9,
        0x3F8 => 0x3F7,
        0x3FB => 0x3FA,
        0x430 => 0x410,
        0x431 => 0x411,
        0x432 => 0x412,
        0x433 => 0x413,
        0x434 => 0x414,
        0x435 => 0x415,
        0x436 => 0x416,
        0x437 => 0x417,
        0x438 => 0x418,
        0x439 => 0x419,
        0x43A => 0x41A,
        0x43B => 0x41B,
        0x43C => 0x41C,
        0x43D => 0x41D,
        0x43E => 0x41E,
        0x43F => 0x41F,
        0x440 => 0x420,
        0x441 => 0x421,
        0x442 => 0x422,
        0x443 => 0x423,
        0x444 => 0x424,
        0x445 => 0x425,
        0x446 => 0x426,
        0x447 => 0x427,
        0x448 => 0x428,
        0x449 => 0x429,
        0x44A => 0x42A,
        0x44B => 0x42B,
        0x44C => 0x42C,
        0x44D => 0x42D,
        0x44E => 0x42E,
        0x44F => 0x42F,
        0x450 => 0x400,
        0x451 => 0x401,
        0x452 => 0x402,
        0x453 => 0x403,
        0x454 => 0x404,
        0x455 => 0x405,
        0x456 => 0x406,
        0x457 => 0x407,
        0x458 => 0x408,
        0x459 => 0x409,
        0x45A => 0x40A,
        0x45B => 0x40B,
        0x45C => 0x40C,
        0x45D => 0x40D,
        0x45E => 0x40E,
        0x45F => 0x40F,
        0x461 => 0x460,
        0x463 => 0x462,
        0x465 => 0x464,
        0x467 => 0x466,
        0x469 => 0x468,
        0x46B => 0x46A,
        0x46D => 0x46C,
        0x46F => 0x46E,
        0x471 => 0x470,
        0x473 => 0x472,
        0x475 => 0x474,
        0x477 => 0x476,
        0x479 => 0x478,
        0x47B => 0x47A,
        0x47D => 0x47C,
        0x47F => 0x47E,
        0x481 => 0x480,
        0x48B => 0x48A,
        0x48D => 0x48C,
        0x48F => 0x48E,
        0x491 => 0x490,
        0x493 => 0x492,
        0x495 => 0x494,
        0x497 => 0x496,
        0x499 => 0x498,
        0x49B => 0x49A,
        0x49D => 0x49C,
        0x49F => 0x49E,
        0x4A1 => 0x4A0,
        0x4A3 => 0x4A2,
        0x4A5 => 0x4A4,
        0x4A7 => 0x4A6,
        0x4A9 => 0x4A8,
        0x4AB => 0x4AA,
        0x4AD => 0x4AC,
        0x4AF => 0x4AE,
        0x4B1 => 0x4B0,
        0x4B3 => 0x4B2,
        0x4B5 => 0x4B4,
        0x4B7 => 0x4B6,
        0x4B9 => 0x4B8,
        0x4BB => 0x4BA,
        0x4BD => 0x4BC,
        0x4BF => 0x4BE,
        0x4C2 => 0x4C1,
        0x4C4 => 0x4C3,
        0x4C6 => 0x4C5,
        0x4C8 => 0x4C7,
        0x4CA => 0x4C9,
        0x4CC => 0x4CB,
        0x4CE => 0x4CD,
        0x4CF => 0x4C0,
        0x4D1 => 0x4D0,
        0x4D3 => 0x4D2,
        0x4D5 => 0x4D4,
        0x4D7 => 0x4D6,
        0x4D9 => 0x4D8,
        0x4DB => 0x4DA,
        0x4DD => 0x4DC,
        0x4DF => 0x4DE,
        0x4E1 => 0x4E0,
        0x4E3 => 0x4E2,
        0x4E5 => 0x4E4,
        0x4E7 => 0x4E6,
        0x4E9 => 0x4E8,
        0x4EB => 0x4EA,
        0x4ED => 0x4EC,
        0x4EF => 0x4EE,
        0x4F1 => 0x4F0,
        0x4F3 => 0x4F2,
        0x4F5 => 0x4F4,
        0x4F7 => 0x4F6,
        0x4F9 => 0x4F8,
        0x4FB => 0x4FA,
        0x4FD => 0x4FC,
        0x4FF => 0x4FE,
        0x501 => 0x500,
        0x503 => 0x502,
        0x505 => 0x504,
        0x507 => 0x506,
        0x509 => 0x508,
        0x50B => 0x50A,
        0x50D => 0x50C,
        0x50F => 0x50E,
        0x511 => 0x510,
        0x513 => 0x512,
        0x515 => 0x514,
        0x517 => 0x516,
        0x519 => 0x518,
        0x51B => 0x51A,
        0x51D => 0x51C,
        0x51F => 0x51E,
        0x521 => 0x520,
        0x523 => 0x522,
        0x561 => 0x531,
        0x562 => 0x532,
        0x563 => 0x533,
        0x564 => 0x534,
        0x565 => 0x535,
        0x566 => 0x536,
        0x567 => 0x537,
        0x568 => 0x538,
        0x569 => 0x539,
        0x56A => 0x53A,
        0x56B => 0x53B,
        0x56C => 0x53C,
        0x56D => 0x53D,
        0x56E => 0x53E,
        0x56F => 0x53F,
        0x570 => 0x540,
        0x571 => 0x541,
        0x572 => 0x542,
        0x573 => 0x543,
        0x574 => 0x544,
        0x575 => 0x545,
        0x576 => 0x546,
        0x577 => 0x547,
        0x578 => 0x548,
        0x579 => 0x549,
        0x57A => 0x54A,
        0x57B => 0x54B,
        0x57C => 0x54C,
        0x57D => 0x54D,
        0x57E => 0x54E,
        0x57F => 0x54F,
        0x580 => 0x550,
        0x581 => 0x551,
        0x582 => 0x552,
        0x583 => 0x553,
        0x584 => 0x554,
        0x585 => 0x555,
        0x586 => 0x556,
        0x1D79 => 0xA77D,
        0x1D7D => 0x2C63,
        0x1E01 => 0x1E00,
        0x1E03 => 0x1E02,
        0x1E05 => 0x1E04,
        0x1E07 => 0x1E06,
        0x1E09 => 0x1E08,
        0x1E0B => 0x1E0A,
        0x1E0D => 0x1E0C,
        0x1E0F => 0x1E0E,
        0x1E11 => 0x1E10,
        0x1E13 => 0x1E12,
        0x1E15 => 0x1E14,
        0x1E17 => 0x1E16,
        0x1E19 => 0x1E18,
        0x1E1B => 0x1E1A,
        0x1E1D => 0x1E1C,
        0x1E1F => 0x1E1E,
        0x1E21 => 0x1E20,
        0x1E23 => 0x1E22,
        0x1E25 => 0x1E24,
        0x1E27 => 0x1E26,
        0x1E29 => 0x1E28,
        0x1E2B => 0x1E2A,
        0x1E2D => 0x1E2C,
        0x1E2F => 0x1E2E,
        0x1E31 => 0x1E30,
        0x1E33 => 0x1E32,
        0x1E35 => 0x1E34,
        0x1E37 => 0x1E36,
        0x1E39 => 0x1E38,
        0x1E3B => 0x1E3A,
        0x1E3D => 0x1E3C,
        0x1E3F => 0x1E3E,
        0x1E41 => 0x1E40,
        0x1E43 => 0x1E42,
        0x1E45 => 0x1E44,
        0x1E47 => 0x1E46,
        0x1E49 => 0x1E48,
        0x1E4B => 0x1E4A,
        0x1E4D => 0x1E4C,
        0x1E4F => 0x1E4E,
        0x1E51 => 0x1E50,
        0x1E53 => 0x1E52,
        0x1E55 => 0x1E54,
        0x1E57 => 0x1E56,
        0x1E59 => 0x1E58,
        0x1E5B => 0x1E5A,
        0x1E5D => 0x1E5C,
        0x1E5F => 0x1E5E,
        0x1E61 => 0x1E60,
        0x1E63 => 0x1E62,
        0x1E65 => 0x1E64,
        0x1E67 => 0x1E66,
        0x1E69 => 0x1E68,
        0x1E6B => 0x1E6A,
        0x1E6D => 0x1E6C,
        0x1E6F => 0x1E6E,
        0x1E71 => 0x1E70,
        0x1E73 => 0x1E72,
        0x1E75 => 0x1E74,
        0x1E77 => 0x1E76,
        0x1E79 => 0x1E78,
        0x1E7B => 0x1E7A,
        0x1E7D => 0x1E7C,
        0x1E7F => 0x1E7E,
        0x1E81 => 0x1E80,
        0x1E83 => 0x1E82,
        0x1E85 => 0x1E84,
        0x1E87 => 0x1E86,
        0x1E89 => 0x1E88,
        0x1E8B => 0x1E8A,
        0x1E8D => 0x1E8C,
        0x1E8F => 0x1E8E,
        0x1E91 => 0x1E90,
        0x1E93 => 0x1E92,
        0x1E95 => 0x1E94,
        0x1EA1 => 0x1EA0,
        0x1EA3 => 0x1EA2,
        0x1EA5 => 0x1EA4,
        0x1EA7 => 0x1EA6,
        0x1EA9 => 0x1EA8,
        0x1EAB => 0x1EAA,
        0x1EAD => 0x1EAC,
        0x1EAF => 0x1EAE,
        0x1EB1 => 0x1EB0,
        0x1EB3 => 0x1EB2,
        0x1EB5 => 0x1EB4,
        0x1EB7 => 0x1EB6,
        0x1EB9 => 0x1EB8,
        0x1EBB => 0x1EBA,
        0x1EBD => 0x1EBC,
        0x1EBF => 0x1EBE,
        0x1EC1 => 0x1EC0,
        0x1EC3 => 0x1EC2,
        0x1EC5 => 0x1EC4,
        0x1EC7 => 0x1EC6,
        0x1EC9 => 0x1EC8,
        0x1ECB => 0x1ECA,
        0x1ECD => 0x1ECC,
        0x1ECF => 0x1ECE,
        0x1ED1 => 0x1ED0,
        0x1ED3 => 0x1ED2,
        0x1ED5 => 0x1ED4,
        0x1ED7 => 0x1ED6,
        0x1ED9 => 0x1ED8,
        0x1EDB => 0x1EDA,
        0x1EDD => 0x1EDC,
        0x1EDF => 0x1EDE,
        0x1EE1 => 0x1EE0,
        0x1EE3 => 0x1EE2,
        0x1EE5 => 0x1EE4,
        0x1EE7 => 0x1EE6,
        0x1EE9 => 0x1EE8,
        0x1EEB => 0x1EEA,
        0x1EED => 0x1EEC,
        0x1EEF => 0x1EEE,
        0x1EF1 => 0x1EF0,
        0x1EF3 => 0x1EF2,
        0x1EF5 => 0x1EF4,
        0x1EF7 => 0x1EF6,
        0x1EF9 => 0x1EF8,
        0x1EFB => 0x1EFA,
        0x1EFD => 0x1EFC,
        0x1EFF => 0x1EFE,
        0x1F00 => 0x1F08,
        0x1F01 => 0x1F09,
        0x1F02 => 0x1F0A,
        0x1F03 => 0x1F0B,
        0x1F04 => 0x1F0C,
        0x1F05 => 0x1F0D,
        0x1F06 => 0x1F0E,
        0x1F07 => 0x1F0F,
        0x1F10 => 0x1F18,
        0x1F11 => 0x1F19,
        0x1F12 => 0x1F1A,
        0x1F13 => 0x1F1B,
        0x1F14 => 0x1F1C,
        0x1F15 => 0x1F1D,
        0x1F20 => 0x1F28,
        0x1F21 => 0x1F29,
        0x1F22 => 0x1F2A,
        0x1F23 => 0x1F2B,
        0x1F24 => 0x1F2C,
        0x1F25 => 0x1F2D,
        0x1F26 => 0x1F2E,
        0x1F27 => 0x1F2F,
        0x1F30 => 0x1F38,
        0x1F31 => 0x1F39,
        0x1F32 => 0x1F3A,
        0x1F33 => 0x1F3B,
        0x1F34 => 0x1F3C,
        0x1F35 => 0x1F3D,
        0x1F36 => 0x1F3E,
        0x1F37 => 0x1F3F,
        0x1F40 => 0x1F48,
        0x1F41 => 0x1F49,
        0x1F42 => 0x1F4A,
        0x1F43 => 0x1F4B,
        0x1F44 => 0x1F4C,
        0x1F45 => 0x1F4D,
        0x1F51 => 0x1F59,
        0x1F53 => 0x1F5B,
        0x1F55 => 0x1F5D,
        0x1F57 => 0x1F5F,
        0x1F60 => 0x1F68,
        0x1F61 => 0x1F69,
        0x1F62 => 0x1F6A,
        0x1F63 => 0x1F6B,
        0x1F64 => 0x1F6C,
        0x1F65 => 0x1F6D,
        0x1F66 => 0x1F6E,
        0x1F67 => 0x1F6F,
        0x1F70 => 0x1FBA,
        0x1F71 => 0x1FBB,
        0x1F72 => 0x1FC8,
        0x1F73 => 0x1FC9,
        0x1F74 => 0x1FCA,
        0x1F75 => 0x1FCB,
        0x1F76 => 0x1FDA,
        0x1F77 => 0x1FDB,
        0x1F78 => 0x1FF8,
        0x1F79 => 0x1FF9,
        0x1F7A => 0x1FEA,
        0x1F7B => 0x1FEB,
        0x1F7C => 0x1FFA,
        0x1F7D => 0x1FFB,
        0x1F80 => 0x1F88,
        0x1F81 => 0x1F89,
        0x1F82 => 0x1F8A,
        0x1F83 => 0x1F8B,
        0x1F84 => 0x1F8C,
        0x1F85 => 0x1F8D,
        0x1F86 => 0x1F8E,
        0x1F87 => 0x1F8F,
        0x1F90 => 0x1F98,
        0x1F91 => 0x1F99,
        0x1F92 => 0x1F9A,
        0x1F93 => 0x1F9B,
        0x1F94 => 0x1F9C,
        0x1F95 => 0x1F9D,
        0x1F96 => 0x1F9E,
        0x1F97 => 0x1F9F,
        0x1FA0 => 0x1FA8,
        0x1FA1 => 0x1FA9,
        0x1FA2 => 0x1FAA,
        0x1FA3 => 0x1FAB,
        0x1FA4 => 0x1FAC,
        0x1FA5 => 0x1FAD,
        0x1FA6 => 0x1FAE,
        0x1FA7 => 0x1FAF,
        0x1FB0 => 0x1FB8,
        0x1FB1 => 0x1FB9,
        0x1FB3 => 0x1FBC,
        0x1FC3 => 0x1FCC,
        0x1FD0 => 0x1FD8,
        0x1FD1 => 0x1FD9,
        0x1FE0 => 0x1FE8,
        0x1FE1 => 0x1FE9,
        0x1FE5 => 0x1FEC,
        0x1FF3 => 0x1FFC,
        0x214E => 0x2132,
        0x2170 => 0x2160,
        0x2171 => 0x2161,
        0x2172 => 0x2162,
        0x2173 => 0x2163,
        0x2174 => 0x2164,
        0x2175 => 0x2165,
        0x2176 => 0x2166,
        0x2177 => 0x2167,
        0x2178 => 0x2168,
        0x2179 => 0x2169,
        0x217A => 0x216A,
        0x217B => 0x216B,
        0x217C => 0x216C,
        0x217D => 0x216D,
        0x217E => 0x216E,
        0x217F => 0x216F,
        0x2184 => 0x2183,
        0x24D0 => 0x24B6,
        0x24D1 => 0x24B7,
        0x24D2 => 0x24B8,
        0x24D3 => 0x24B9,
        0x24D4 => 0x24BA,
        0x24D5 => 0x24BB,
        0x24D6 => 0x24BC,
        0x24D7 => 0x24BD,
        0x24D8 => 0x24BE,
        0x24D9 => 0x24BF,
        0x24DA => 0x24C0,
        0x24DB => 0x24C1,
        0x24DC => 0x24C2,
        0x24DD => 0x24C3,
        0x24DE => 0x24C4,
        0x24DF => 0x24C5,
        0x24E0 => 0x24C6,
        0x24E1 => 0x24C7,
        0x24E2 => 0x24C8,
        0x24E3 => 0x24C9,
        0x24E4 => 0x24CA,
        0x24E5 => 0x24CB,
        0x24E6 => 0x24CC,
        0x24E7 => 0x24CD,
        0x24E8 => 0x24CE,
        0x24E9 => 0x24CF,
        0x2C30 => 0x2C00,
        0x2C31 => 0x2C01,
        0x2C32 => 0x2C02,
        0x2C33 => 0x2C03,
        0x2C34 => 0x2C04,
        0x2C35 => 0x2C05,
        0x2C36 => 0x2C06,
        0x2C37 => 0x2C07,
        0x2C38 => 0x2C08,
        0x2C39 => 0x2C09,
        0x2C3A => 0x2C0A,
        0x2C3B => 0x2C0B,
        0x2C3C => 0x2C0C,
        0x2C3D => 0x2C0D,
        0x2C3E => 0x2C0E,
        0x2C3F => 0x2C0F,
        0x2C40 => 0x2C10,
        0x2C41 => 0x2C11,
        0x2C42 => 0x2C12,
        0x2C43 => 0x2C13,
        0x2C44 => 0x2C14,
        0x2C45 => 0x2C15,
        0x2C46 => 0x2C16,
        0x2C47 => 0x2C17,
        0x2C48 => 0x2C18,
        0x2C49 => 0x2C19,
        0x2C4A => 0x2C1A,
        0x2C4B => 0x2C1B,
        0x2C4C => 0x2C1C,
        0x2C4D => 0x2C1D,
        0x2C4E => 0x2C1E,
        0x2C4F => 0x2C1F,
        0x2C50 => 0x2C20,
        0x2C51 => 0x2C21,
        0x2C52 => 0x2C22,
        0x2C53 => 0x2C23,
        0x2C54 => 0x2C24,
        0x2C55 => 0x2C25,
        0x2C56 => 0x2C26,
        0x2C57 => 0x2C27,
        0x2C58 => 0x2C28,
        0x2C59 => 0x2C29,
        0x2C5A => 0x2C2A,
        0x2C5B => 0x2C2B,
        0x2C5C => 0x2C2C,
        0x2C5D => 0x2C2D,
        0x2C5E => 0x2C2E,
        0x2C61 => 0x2C60,
        0x2C65 => 0x23A,
        0x2C66 => 0x23E,
        0x2C68 => 0x2C67,
        0x2C6A => 0x2C69,
        0x2C6C => 0x2C6B,
        0x2C73 => 0x2C72,
        0x2C76 => 0x2C75,
        0x2C81 => 0x2C80,
        0x2C83 => 0x2C82,
        0x2C85 => 0x2C84,
        0x2C87 => 0x2C86,
        0x2C89 => 0x2C88,
        0x2C8B => 0x2C8A,
        0x2C8D => 0x2C8C,
        0x2C8F => 0x2C8E,
        0x2C91 => 0x2C90,
        0x2C93 => 0x2C92,
        0x2C95 => 0x2C94,
        0x2C97 => 0x2C96,
        0x2C99 => 0x2C98,
        0x2C9B => 0x2C9A,
        0x2C9D => 0x2C9C,
        0x2C9F => 0x2C9E,
        0x2CA1 => 0x2CA0,
        0x2CA3 => 0x2CA2,
        0x2CA5 => 0x2CA4,
        0x2CA7 => 0x2CA6,
        0x2CA9 => 0x2CA8,
        0x2CAB => 0x2CAA,
        0x2CAD => 0x2CAC,
        0x2CAF => 0x2CAE,
        0x2CB1 => 0x2CB0,
        0x2CB3 => 0x2CB2,
        0x2CB5 => 0x2CB4,
        0x2CB7 => 0x2CB6,
        0x2CB9 => 0x2CB8,
        0x2CBB => 0x2CBA,
        0x2CBD => 0x2CBC,
        0x2CBF => 0x2CBE,
        0x2CC1 => 0x2CC0,
        0x2CC3 => 0x2CC2,
        0x2CC5 => 0x2CC4,
        0x2CC7 => 0x2CC6,
        0x2CC9 => 0x2CC8,
        0x2CCB => 0x2CCA,
        0x2CCD => 0x2CCC,
        0x2CCF => 0x2CCE,
        0x2CD1 => 0x2CD0,
        0x2CD3 => 0x2CD2,
        0x2CD5 => 0x2CD4,
        0x2CD7 => 0x2CD6,
        0x2CD9 => 0x2CD8,
        0x2CDB => 0x2CDA,
        0x2CDD => 0x2CDC,
        0x2CDF => 0x2CDE,
        0x2CE1 => 0x2CE0,
        0x2CE3 => 0x2CE2,
        0x2D00 => 0x10A0,
        0x2D01 => 0x10A1,
        0x2D02 => 0x10A2,
        0x2D03 => 0x10A3,
        0x2D04 => 0x10A4,
        0x2D05 => 0x10A5,
        0x2D06 => 0x10A6,
        0x2D07 => 0x10A7,
        0x2D08 => 0x10A8,
        0x2D09 => 0x10A9,
        0x2D0A => 0x10AA,
        0x2D0B => 0x10AB,
        0x2D0C => 0x10AC,
        0x2D0D => 0x10AD,
        0x2D0E => 0x10AE,
        0x2D0F => 0x10AF,
        0x2D10 => 0x10B0,
        0x2D11 => 0x10B1,
        0x2D12 => 0x10B2,
        0x2D13 => 0x10B3,
        0x2D14 => 0x10B4,
        0x2D15 => 0x10B5,
        0x2D16 => 0x10B6,
        0x2D17 => 0x10B7,
        0x2D18 => 0x10B8,
        0x2D19 => 0x10B9,
        0x2D1A => 0x10BA,
        0x2D1B => 0x10BB,
        0x2D1C => 0x10BC,
        0x2D1D => 0x10BD,
        0x2D1E => 0x10BE,
        0x2D1F => 0x10BF,
        0x2D20 => 0x10C0,
        0x2D21 => 0x10C1,
        0x2D22 => 0x10C2,
        0x2D23 => 0x10C3,
        0x2D24 => 0x10C4,
        0x2D25 => 0x10C5,
        0xA641 => 0xA640,
        0xA643 => 0xA642,
        0xA645 => 0xA644,
        0xA647 => 0xA646,
        0xA649 => 0xA648,
        0xA64B => 0xA64A,
        0xA64D => 0xA64C,
        0xA64F => 0xA64E,
        0xA651 => 0xA650,
        0xA653 => 0xA652,
        0xA655 => 0xA654,
        0xA657 => 0xA656,
        0xA659 => 0xA658,
        0xA65B => 0xA65A,
        0xA65D => 0xA65C,
        0xA65F => 0xA65E,
        0xA663 => 0xA662,
        0xA665 => 0xA664,
        0xA667 => 0xA666,
        0xA669 => 0xA668,
        0xA66B => 0xA66A,
        0xA66D => 0xA66C,
        0xA681 => 0xA680,
        0xA683 => 0xA682,
        0xA685 => 0xA684,
        0xA687 => 0xA686,
        0xA689 => 0xA688,
        0xA68B => 0xA68A,
        0xA68D => 0xA68C,
        0xA68F => 0xA68E,
        0xA691 => 0xA690,
        0xA693 => 0xA692,
        0xA695 => 0xA694,
        0xA697 => 0xA696,
        0xA723 => 0xA722,
        0xA725 => 0xA724,
        0xA727 => 0xA726,
        0xA729 => 0xA728,
        0xA72B => 0xA72A,
        0xA72D => 0xA72C,
        0xA72F => 0xA72E,
        0xA733 => 0xA732,
        0xA735 => 0xA734,
        0xA737 => 0xA736,
        0xA739 => 0xA738,
        0xA73B => 0xA73A,
        0xA73D => 0xA73C,
        0xA73F => 0xA73E,
        0xA741 => 0xA740,
        0xA743 => 0xA742,
        0xA745 => 0xA744,
        0xA747 => 0xA746,
        0xA749 => 0xA748,
        0xA74B => 0xA74A,
        0xA74D => 0xA74C,
        0xA74F => 0xA74E,
        0xA751 => 0xA750,
        0xA753 => 0xA752,
        0xA755 => 0xA754,
        0xA757 => 0xA756,
        0xA759 => 0xA758,
        0xA75B => 0xA75A,
        0xA75D => 0xA75C,
        0xA75F => 0xA75E,
        0xA761 => 0xA760,
        0xA763 => 0xA762,
        0xA765 => 0xA764,
        0xA767 => 0xA766,
        0xA769 => 0xA768,
        0xA76B => 0xA76A,
        0xA76D => 0xA76C,
        0xA76F => 0xA76E,
        0xA77A => 0xA779,
        0xA77C => 0xA77B,
        0xA77F => 0xA77E,
        0xA781 => 0xA780,
        0xA783 => 0xA782,
        0xA785 => 0xA784,
        0xA787 => 0xA786,
        0xA78C => 0xA78B,
        0xFF41 => 0xFF21,
        0xFF42 => 0xFF22,
        0xFF43 => 0xFF23,
        0xFF44 => 0xFF24,
        0xFF45 => 0xFF25,
        0xFF46 => 0xFF26,
        0xFF47 => 0xFF27,
        0xFF48 => 0xFF28,
        0xFF49 => 0xFF29,
        0xFF4A => 0xFF2A,
        0xFF4B => 0xFF2B,
        0xFF4C => 0xFF2C,
        0xFF4D => 0xFF2D,
        0xFF4E => 0xFF2E,
        0xFF4F => 0xFF2F,
        0xFF50 => 0xFF30,
        0xFF51 => 0xFF31,
        0xFF52 => 0xFF32,
        0xFF53 => 0xFF33,
        0xFF54 => 0xFF34,
        0xFF55 => 0xFF35,
        0xFF56 => 0xFF36,
        0xFF57 => 0xFF37,
        0xFF58 => 0xFF38,
        0xFF59 => 0xFF39,
        0xFF5A => 0xFF3A,
        else => c,
    };
}

Test case to verify that the generated switch matches the behavior of `RtlUpcaseUnicodeChar` for all possible `u16` values

const std = @import("std");

test {
    var c: u16 = 0;
    while (true) {
        var win32_upper = std.os.windows.ntdll.RtlUpcaseUnicodeChar(c);
        var zig_upper = upcase(c);

        std.testing.expectEqual(win32_upper, zig_upper) catch |err| {
            std.debug.print("c=0x{X}\n", .{c});
            return err;
        };

        if (c == 0xFFFF) break;
        c += 1;
    }
}

Some notes:

• There are exactly 973 lowercase -> uppercase mappings in the data when including the ASCII range, or 947 when excluding the ASCII range
• This would be enough to implement eqlIgnoreCaseWTF16 in a cross-platform way if that's the way we end up wanting to go.
  • When targeting Windows, it would still probably make more sense to call the ntdll functions instead, though; the cross-platform implementation would therefore only get used on non-Windows targets (EDIT: Thinking about this more, there may be no benefit to using the ntdll stuff for something like std.fs.path.relativeWindows since we're comparing the path component-by-component so even if the underlying filesystem uses different case conversion data, using RtlEqualUnicodeString wouldn't be able to provide us with the 'filesystem-correct' comparison anyway).
• As mentioned, unsure about how we'd go about dealing with changes to this uppercasing data in Windows if it ends up changing (or is already different on Windows 11)
• There might be a better way to encode the data than a giant switch case, maybe some sort of minimal perfect hashing or something like that
  • I believe a theoretically perfect MPH solution would be able to use an array of 947 u16s or 1,894 bytes total (since the ASCII range wouldn't need to be part of the table)

[e4m2]

On newer Windows, only the l_intl.nls file is ever read, so that's good news. On the other hand, in Windows 11 the function was modified to look like this (RE'd):

WCHAR __stdcall RtlUpcaseUnicodeCharWin11(WCHAR Source) {
    if (Source < 'a') {
        return Source;
    }
    if (Source <= 'z') {
        return Source - ' ';
    }

    if (Table && Source >= 0xC0) {
        Source += Table[(Source & 0xF) + Table[((Source >> 4) & 0xF) + Table[(Source >> 8) & 0xFF]]];
    }
    return Source;
}

The table lookup is the same, but the table itself might not exist now and an additional condition is added on the source argument. The tables are initialized in the aptly named RtlpInitUppercaseTables.

RE'd here

void __stdcall RtlpInitUppercaseTables(USHORT UseCaseMapping) {
    if (UseCaseMapping == 0) {
        NlsUppercaseTables = NULL;
        NlsLowercaseTables = NULL;
        NlsSectionPtr = NULL;
        return;
    }

    USHORT *NlsPtr = NULL;
    NTSTATUS Result = NtGetNlsSectionPtr(14, 0, NULL, (PVOID *)&NlsPtr, NULL);
    if (Result < 0) {
        NlsPtr = NULL;
    }

    if (NlsPtr) {
        _InterlockedExchange64((LONG64 volatile *)&NlsUppercaseTables, (LONG64)(NlsPtr + 2));
        _InterlockedExchange64((LONG64 volatile *)&NlsLowercaseTables, (LONG64)(NlsPtr + NlsPtr[1] + 2));
        NlsSectionPtr = NlsPtr;
    } else {
        NlsUppercaseTables = NULL;
        NlsLowercaseTables = NULL;
        NlsSectionPtr = NULL;
    }
}

The tables are initialized only if UseCaseMapping is non-zero. If they stay uninitialized, only the ASCII range will be converted in RtlUpcaseUnicodeChar. UseCaseMapping is a new field in the PEB (UnicodeCaseTableData is now null as well) passed down to RtlpInitUppercaseTables from the loader. MmCreatePeb sets UseCaseMapping if the equivalent ESERVERSILO_GLOBALS.RtlNlsState.UnicodeUpcaseTable844 (not sure what 844 stands for) table is also initialized, it's uninitialized only when the kernel fails to open the \NLS object directory (makes sense because the case mapping table is later created and read from this namespace) - I honestly have no clue when or how it would fail, but clearly the error condition somehow exists if Microsoft bothers to check against it even in user-space.

TL;DR

The upcasing function is different in Windows 11 and may somehow fallback to the ASCII range if a very specific condition arises, but it's probably nothing to worry about. Especially since, as @squeek502 said, there really is no advantage to calling the Windows function directly (even on Windows) as there may be an inherent mismatch with the filesystem anyway.

So I think a reimplementation is the way to go. I prefer the table-based version over the switch-based, but it might indeed be worth investigating whether a better solution exists.

LUT-based upcase

const table = [2544]u16{
    272,   288,   304,   320,   336,   352,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,
    256,   256,   256,   256,   256,   368,   384,   400,   256,   416,   256,   256,   432,   256,   256,   256,   256,   256,   256,   256,   448,   464,   256,   256,
    256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,
    256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,
    256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,
    256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,
    256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   480,   496,
    256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,
    256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,
    256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,
    256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   256,   512,   528,   528,   528,   528,   528,   528,   528,   528,
    528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   544,   560,   528,   528,   528,   576,   528,   528,   592,   608,
    624,   640,   656,   672,   688,   704,   720,   736,   752,   768,   784,   800,   816,   832,   848,   864,   880,   896,   912,   928,   944,   960,   976,   992,
    1008,  1024,  528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   1040,  528,   528,   1056,  528,   528,   1072,  1088,  1104,  1120,  1136,  1152,
    528,   528,   528,   1168,  1184,  1200,  1216,  1232,  1248,  1264,  1280,  1296,  1312,  1328,  1344,  1360,  1376,  1392,  1408,  528,   528,   528,   1424,  1440,
    1456,  528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   1472,  528,   528,   528,   528,   528,   528,   528,   528,
    1488,  1504,  1520,  1536,  1552,  1568,  1584,  1600,  1616,  1632,  1648,  1664,  1680,  1696,  1712,  1728,  1744,  1760,  1776,  1792,  1808,  1824,  1840,  1856,
    1872,  1888,  1904,  1920,  1936,  1952,  1968,  1984,  528,   528,   528,   528,   2000,  528,   528,   2016,  2032,  528,   528,   528,   528,   528,   528,   528,
    528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   2048,  2064,  528,   528,   528,   528,   2080,  2096,  2112,  2128,  2144,
    2160,  2176,  2192,  2208,  2224,  2240,  2256,  528,   2272,  2288,  2304,  528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   528,
    528,   528,   528,   528,   2320,  2336,  2352,  528,   2368,  2384,  528,   528,   528,   528,   528,   528,   528,   528,   2400,  2416,  2432,  2448,  2464,  2480,
    2496,  528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   528,   2512,  2528,  528,   528,   528,   528,   528,   528,   528,   528,   528,   528,
    0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     65504, 65504, 65504, 65504, 65504, 65504, 65504,
    65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 0,     0,     0,     0,     0,
    0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504,
    65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 0,     65504, 65504, 65504, 65504, 65504, 65504, 65504, 121,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     0,     0,     65535, 0,     65535, 0,     65535, 0,     0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,
    65535, 0,     0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     0,     65535, 0,     65535, 0,     65535, 0,     195,   0,     0,     65535, 0,     65535, 0,     0,     65535, 0,     0,     0,     65535, 0,     0,     0,
    0,     0,     65535, 0,     0,     97,    0,     0,     0,     65535, 163,   0,     0,     0,     130,   0,     0,     65535, 0,     65535, 0,     65535, 0,     0,
    65535, 0,     0,     0,     0,     65535, 0,     0,     65535, 0,     0,     0,     65535, 0,     65535, 0,     0,     65535, 0,     0,     0,     65535, 0,     56,
    0,     0,     0,     0,     0,     0,     65534, 0,     0,     65534, 0,     0,     65534, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,
    65535, 0,     65535, 0,     65535, 65457, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     0,     0,     65534, 0,     65535, 0,     0,     0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     0,     0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     0,     0,     0,
    0,     0,     0,     0,     65535, 0,     0,     0,     0,     0,     65535, 0,     0,     0,     0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    10783, 10780, 0,     65326, 65330, 0,     65331, 65331, 0,     65334, 0,     65333, 0,     0,     0,     0,     65331, 0,     0,     65329, 0,     0,     0,     0,
    65327, 65325, 0,     10743, 0,     0,     0,     65325, 0,     10749, 65323, 0,     0,     65322, 0,     0,     0,     0,     0,     0,     0,     10727, 0,     0,
    65318, 0,     0,     65318, 0,     0,     0,     0,     65318, 65467, 65319, 65319, 65465, 0,     0,     0,     0,     0,     65317, 0,     0,     0,     0,     0,
    0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
    0,     65535, 0,     65535, 0,     0,     0,     65535, 0,     0,     0,     130,   130,   130,   0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
    0,     0,     0,     0,     65498, 65499, 65499, 65499, 0,     65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504,
    65504, 65504, 0,     65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65472, 65473, 65473, 0,     0,     0,     0,     0,     0,     0,     0,     65528,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     0,     7,     0,     0,     0,     0,     0,     65535, 0,     0,     65535, 0,     0,     0,     0,     65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504,
    65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504,
    65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     0,     0,     0,     0,     0,     0,     0,     0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     0,     65535, 0,     65535, 0,     65535, 0,
    65535, 0,     65535, 0,     65535, 0,     65535, 65521, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     0,     0,     0,
    0,     0,     0,     0,     0,     0,     0,     0,     0,     65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488,
    65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 0,
    0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     35332, 0,     0,     0,     3814,  0,     0,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 8,     8,     8,     8,     8,     8,     8,     8,
    0,     0,     0,     0,     0,     0,     0,     0,     8,     8,     8,     8,     8,     8,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
    8,     8,     8,     8,     8,     8,     8,     8,     0,     0,     0,     0,     0,     0,     0,     0,     8,     8,     8,     8,     8,     8,     8,     8,
    0,     0,     0,     0,     0,     0,     0,     0,     8,     8,     8,     8,     8,     8,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
    0,     8,     0,     8,     0,     8,     0,     8,     0,     0,     0,     0,     0,     0,     0,     0,     8,     8,     8,     8,     8,     8,     8,     8,
    0,     0,     0,     0,     0,     0,     0,     0,     74,    74,    86,    86,    86,    86,    100,   100,   128,   128,   112,   112,   126,   126,   0,     0,
    8,     8,     8,     8,     8,     8,     8,     8,     0,     0,     0,     0,     0,     0,     0,     0,     8,     8,     8,     8,     8,     8,     8,     8,
    0,     0,     0,     0,     0,     0,     0,     0,     8,     8,     8,     8,     8,     8,     8,     8,     0,     0,     0,     0,     0,     0,     0,     0,
    8,     8,     0,     9,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     9,     0,     0,     0,     0,
    0,     0,     0,     0,     0,     0,     0,     0,     8,     8,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
    8,     8,     0,     0,     0,     7,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     9,     0,     0,     0,     0,
    0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     65508, 0,
    65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 0,     0,     0,     0,     65535, 0,     0,     0,
    0,     0,     0,     0,     0,     0,     0,     0,     65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510,
    65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 0,     0,     0,     0,     0,     0,     65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488,
    65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488,
    65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 0,     0,     65535, 0,     0,     0,     54741, 54744, 0,
    65535, 0,     65535, 0,     65535, 0,     0,     0,     0,     0,     0,     65535, 0,     0,     65535, 0,     0,     0,     0,     0,     0,     0,     0,     0,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272,
    58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272,
    58272, 58272, 58272, 58272, 58272, 58272, 0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     0,     0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     0,     0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     0,     0,     0,     0,     0,     0,     0,
    0,     0,     0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     0,     0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     0,     0,     0,     0,     0,     0,     0,     0,     0,     65535, 0,     65535, 0,     0,     65535,
    0,     65535, 0,     65535, 0,     65535, 0,     65535, 0,     0,     0,     0,     65535, 0,     0,     0,     0,     65504, 65504, 65504, 65504, 65504, 65504, 65504,
    65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 0,     0,     0,     0,     0,
};

fn upcase(c: u16) u16 {
    if (c < 'a') {
        return c;
    }
    if (c <= 'z') {
        return c - ('a' - 'A');
    }
    if (c >= 0xC0) {
        var offset: u16 = 0;

        offset += @truncate(u8, c >> 8);
        offset = table[offset];
        offset += @truncate(u4, c >> 4);
        offset = table[offset];
        offset += @truncate(u4, c);
        offset = table[offset];

        return c +% offset;
    }

    return c;
}

[squeek502]

I've updated the PR with something of a proof-of-concept of a cross-platform eqlIgnoreCaseWTF16 implementation. The details are a bit messier than I'd like, but I'm not really sure how to avoid the messy parts (the biggest one being when to use the ntdll implementation and when to use our own, since our own implementation is faster but increases the binary size if its used).

The Windows NLS format lookup table version was both faster than a switch-based version and contributed less to the final executable size, so that's the implementation that was used here. I benchmarked using the code here:

https://gist.github.com/squeek502/2d5d6a60448e856bd352511911d69401

where news-week-17aug24.txt is the data from here but with everything but the titles removed (and with one title per line) and news-week-17aug24-upper.txt is the same file but converted to uppercase using Sublime Text (so not the same as the Windows uppercasing; not all lines will compare equally). Here are the results on Windows 10 for me:

Benchmark 1: zig-out\bin\upcase-table.exe
  Time (mean ± σ):      1.384 s ±  0.006 s    [User: 1.258 s, System: 0.126 s]
  Range (min … max):    1.378 s …  1.396 s    10 runs

Benchmark 2: zig-out\bin\upcase-switch.exe
  Time (mean ± σ):      1.528 s ±  0.006 s    [User: 1.400 s, System: 0.129 s]
  Range (min … max):    1.519 s …  1.538 s    10 runs

Benchmark 3: zig-out\bin\upcase-switch2.exe
  Time (mean ± σ):      1.519 s ±  0.009 s    [User: 1.375 s, System: 0.141 s]
  Range (min … max):    1.509 s …  1.540 s    10 runs

Benchmark 4: zig-out\bin\upcase-rtl.exe
  Time (mean ± σ):      1.551 s ±  0.002 s    [User: 1.396 s, System: 0.149 s]
  Range (min … max):    1.546 s …  1.554 s    10 runs

Summary
  'zig-out\bin\upcase-table.exe' ran
    1.10 ± 0.01 times faster than 'zig-out\bin\upcase-switch2.exe'
    1.10 ± 0.01 times faster than 'zig-out\bin\upcase-switch.exe'
    1.12 ± 0.00 times faster than 'zig-out\bin\upcase-rtl.exe'

(similar results on Linux but I don't have those handy anymore, would have to re-run them)

---

When the added std.os.windows.nls.upcaseW is referenced, it pulls in 5,088 bytes that will be included in the final executable (in practice, it seems to only add 4,608 bytes but presumably that's for unrelated reasons like padding bytes in the exe format). It seems possible that this size could be compressed further if necessary.

More details from the commit message:

The cross-platform Zig implementation is a bit faster than RtlUpcaseUnicodeChar on my system (1.12 times faster in a benchmark using 100mb of text in many different languages), so it is generally preferred. However, when targeting windows and with the mode set to ReleaseSmall, RtlUpcaseUnicodeChar function is used to avoid having to include the 5,088 bytes of case conversion data in the executable.

Users can also opt to use an ntdll-dependent version of os.windows.eqlIgnoreCaseWTF16 called eqlIgnoreCaseWTF16Ntdll if they know they are targeting windows and want to avoid the uppercase data being included. eqlIgnoreCaseWTF16Ntdll is used in std.os.getenvW for this reason.

An alternate path would be to just to always use the ntdll version when targeting windows to avoid this speed vs binary size tradeoff and favor the smaller binary size generally.

[e4m2]

Judging by the problems of the current conditional impl selection, I think either the Zig or the ntdll implementation should be used fully and consistently (except maybe for eqlIgnoreCaseWTF16Ntdll), as the current half-measures inherently lead to some undesirable situations.

There is a third option, though, that may be promising: Only keep the Zig version of the upcase function, but choose the tables it ends up using at compile time:

• On other systems, obviously just pay the ~5KiB cost if needed.
• On Windows, stash the native table pointer. The Zig table stays unreferenced and doesn't take up any additional space. Ideally NtGetNlsSectionPtr(14, ... would work on every supported version (remember PEB data is null on Windows 11), which would make this easier, but I'm unable to test right now.

[squeek502]

Thanks for the review @e4m2, I think you hit on every reason I wasn't happy with the previous implementation. Updated with one that's much more straightforward:

• std.os.windows.nls.upcaseW always pulls in the uppercase table when referenced. As of now, it is never referenced when targeting Windows unless the user calls it directly.
• std.os.windows.eqlIgnoreCaseWTF16/eqlIgnoreCaseUtf8 always use ntdll calls when targeting Windows to avoid pulling in a redundant copy of the uppercase data
  • caveat: nls.upcaseW is used on Windows when @inComptime() to allow eqlIgnoreCaseWTF16/eqlIgnoreCaseUtf8 to work during comptime on Windows

The NtGetNlsSectionPtr approach may be an improvement but seems like it might introduce more complication than its worth; I'll leave that for a future improvement if there's an indication that it would be beneficial.

[andrewrk]

Outstanding work. Thank you @squeek502.