Update to Unicode 14.0

[cuviper]

The Unicode Standard announced Version 14.0 on September 14, 2021, and this pull request updates the generated tables in core accordingly.

This did require a little prep-work in unicode-table-generator. First, #81358 had modified the generated file instead of the tool, so that change is now reflected in the tool as well. Next, I found that the "Alphabetic" property in version 14 was panicking when generating a bitset, "cannot pack 264 into 8 bits". We've been using the skiplist for that anyway, so I changed this to fail gracefully. Finally, I confirmed that the tool still created the exact same tables for 13 before moving to 14.

[rust-highfive]

r? @joshtriplett

(rust-highfive has picked a reviewer for you, use r? to override)

[cuviper]

For comparison, here's the tool output with version 13 data:

Alphabetic     : 1599 bytes, 132875 codepoints in 695 ranges (65 - 201547) using skiplist
Case_Ignorable : 949 bytes, 2413 codepoints in 410 ranges (39 - 918000) using skiplist
Cased          : 359 bytes, 4286 codepoints in 141 ranges (65 - 127370) using skiplist
Cc             : 9 bytes, 65 codepoints in 2 ranges (0 - 160) using skiplist
Grapheme_Extend: 813 bytes, 1979 codepoints in 344 ranges (768 - 918000) using skiplist
Lowercase      : 867 bytes, 2344 codepoints in 652 ranges (97 - 125252) using bitset
N              : 419 bytes, 1781 codepoints in 133 ranges (48 - 130042) using skiplist
Uppercase      : 777 bytes, 1911 codepoints in 643 ranges (65 - 127370) using bitset
White_Space    : 37 bytes, 25 codepoints in 10 ranges (9 - 12289) using skiplist
Total table sizes: 5829 bytes

And here's with version 14 data:

Alphabetic     : 1649 bytes, 133396 codepoints in 722 ranges (65 - 201547) using skiplist
Case_Ignorable : 995 bytes, 2602 codepoints in 427 ranges (39 - 918000) using skiplist
Cased          : 395 bytes, 4453 codepoints in 155 ranges (65 - 127370) using skiplist
Cc             : 9 bytes, 65 codepoints in 2 ranges (0 - 160) using skiplist
Grapheme_Extend: 835 bytes, 2090 codepoints in 353 ranges (768 - 918000) using skiplist
Lowercase      : 907 bytes, 2471 codepoints in 668 ranges (97 - 125252) using bitset
N              : 421 bytes, 1791 codepoints in 134 ranges (48 - 130042) using skiplist
Uppercase      : 791 bytes, 1951 codepoints in 651 ranges (65 - 127370) using bitset
White_Space    : 37 bytes, 25 codepoints in 10 ranges (9 - 12289) using skiplist
Total table sizes: 6039 bytes

[gilescope]

Sorry! I know for next time. Thank you for all the work you're doing upgrading unicode!

[klensy]

There some unicode crates around, should they be updated too?

[cuviper]

@klensy sure, other crates will have to be updated individually. e.g. unicode-xid might be good to get rustc to recognize new characters in identifiers.

[joshtriplett]

@bors r+

[bors]

📌 Commit 459a7e3 has been approved by joshtriplett

[camelid]

First, #81358 had modified the generated file instead of the tool, so that change is now reflected in the tool as well.

Maybe a tidy check should be added that compares the output of the tool to the checked-in file to make sure they're the same?

[gilescope]

Ideally there's some kind of #![autogen_code] metadata and rust analyzer warns me I am altering autogenerated code.

…

On Sat, 9 Oct 2021 at 20:05, bors ***@***.***> wrote: Merged #89614 <#89614> into master. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#89614 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGEJCAZ5OOBOPJFBGFV4I3UGCG65ANCNFSM5FQDVQ4A> .

[cuviper]

@Mark-Simulacrum would mingw-check be a good place to check the generated code? Or maybe a rust-highfive mention?

[camelid]

Or maybe a rust-highfive mention?

That's a good idea; a rust-highfive ping should be quick to set up and sufficient since the file is not changed often.

[Mark-Simulacrum]

Highfive definitely seems like a good idea, someone can file a PR adding that rule somewhere here - https://github.com/rust-lang/highfive/blob/master/highfive/configs/rust-lang/rust.json#L130

I think running the script on each CI to check it's still in sync might be a bit much, though, seems not worth it to me. If we want something more than highfive, I would just fail CI if the file doesn't match a hardcoded hash (e.g., in tidy). Easy to update when needed and fast, as well as potentially more broadly useful (we have several such files, I think).