Add a lower bound check to `unicode-table-generator` output #122013

Swatinem · 2024-03-05T07:23:32Z

This adds a dedicated check for the lower bound
(if it is outside of ASCII range) to the output of the unicode-table-generator tool.

This generalized the ASCII-only fast-path, but only for the Grapheme_Extend property for now, as that is the only one with a lower bound outside of ASCII.

rustbot · 2024-03-05T07:23:40Z

r? @scottmcm

rustbot has assigned @scottmcm.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Swatinem · 2024-03-05T07:25:02Z

As doing this only for the ASCII lower bound only applied to a single table, how about covering the full range? As in: (lower..upper).contains(c as u32) or something like that?

workingjubilee · 2024-03-05T07:53:46Z

That doesn't sound like it would be faster, as it would be both a lower bound check and an upper bound check before moving on to the bitset search? Checking the lower bound seems good, but for the upper bound, I would assume letting the binary search play out would be faster.

tgross35 · 2024-03-05T08:29:11Z

src/tools/unicode-table-generator/src/raw_emitter.rs

@@ -101,7 +102,10 @@ impl RawEmitter {
        )
        .unwrap();
        writeln!(&mut self.file, "pub const fn lookup(c: char) -> bool {{").unwrap();
-        writeln!(&mut self.file, "    super::bitset_search(",).unwrap();
+        if first_code_point > 0x7f {
+            writeln!(&mut self.file, "    (c as u32) >= {first_code_point} &&").unwrap();


Suggested change

writeln!(&mut self.file, " (c as u32) >= {first_code_point} &&").unwrap();

writeln!(&mut self.file, " (c as u32) >= {first_code_point:#04x} &&").unwrap();

Just to keep the hex consistent. Could also do u32::from(c) rather than the as cast since it has that impl, but makes no difference.

All the other numbers in this file are printed as decimals, but I did this change. This might also solve the confusion in the other review comment.

src/tools/unicode-table-generator/src/skiplist.rs

tgross35 · 2024-03-05T08:34:08Z

Honestly I think you may as well close #121138 before it merges in favor of this, otherwise you'll just have to undo it 😄

scottmcm · 2024-03-05T08:37:25Z

library/core/src/unicode/unicode_data.rs

@@ -316,6 +316,7 @@ pub mod grapheme_extend {
        128, 240, 0,
    ];
    pub fn lookup(c: char) -> bool {
+        (c as u32) >= 768 &&


unsure: I see in SHORT_OFFSET_RUNS that the first one is 768, which looks awfully similar to this 768. Should this be doing c as u32 - 768 and making each of the statics shorter?

I haven’t really spent much time figuring out how the skip list search actually works, and what the meaning of those entries is.
Looking at all the other tables, the first entry in SHORT_OFFSET_RUNS does not match the lower bound, so it might just be coincidence.

Maybe its possible to do a checked_sub and take advantage of this, but I would have to dig deeper into the code.

Fair, can separate this improvement from any tweak to the tables.

Swatinem · 2024-03-05T19:29:54Z

I ended up reverting #121138 in this PR as well, since this impl is more general than that.
Although I’m not entirely sure, since the lookup function does not have an #[inline] annotations, so I’m not sure if it will have the same perf impact.

cuviper · 2024-03-08T00:07:23Z

@bors try @rust-timer queue

Add a lower bound check to `unicode-table-generator` output This adds a dedicated check for the lower bound (if it is outside of ASCII range) to the output of the `unicode-table-generator` tool. This generalized the ASCII-only fast-path, but only for the `Grapheme_Extend` property for now, as that is the only one with a lower bound outside of ASCII.

bors · 2024-03-08T00:08:34Z

⌛ Trying commit 6d7daa0 with merge 554f230...

bors · 2024-03-08T01:36:21Z

☀️ Try build successful - checks-actions
Build commit: 554f230 (554f2305f7f70bf404662f432d49f94ad0c72ec6)

rust-timer · 2024-03-08T03:38:48Z

Finished benchmarking commit (554f230): comparison URL.

Overall result: ✅ improvements - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.3%	[-0.3%, -0.3%]	1
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	5.8%	[5.1%, 6.6%]	2
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-5.4%	[-5.4%, -5.4%]	1
All ❌✅ (primary)	5.8%	[5.1%, 6.6%]	2

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 647.485s -> 647.691s (0.03%)
Artifact size: 172.63 MiB -> 172.63 MiB (0.00%)

Swatinem · 2024-03-08T09:24:40Z

The fmt-debug-derive Runtime benchmark reports a regression of 12.18%, so it seems like the #[inline] annotation is indeed significant 🤔

scottmcm · 2024-03-27T17:55:33Z

Let's re-run perf to ensure that things are fine with the #[inline] still there.

@bors try @rust-timer queue

bors · 2024-03-27T17:56:44Z

⌛ Trying commit 6d7daa0 with merge fb17f9e...

Add a lower bound check to `unicode-table-generator` output This adds a dedicated check for the lower bound (if it is outside of ASCII range) to the output of the `unicode-table-generator` tool. This generalized the ASCII-only fast-path, but only for the `Grapheme_Extend` property for now, as that is the only one with a lower bound outside of ASCII.

bors · 2024-03-27T19:29:11Z

☀️ Try build successful - checks-actions
Build commit: fb17f9e (fb17f9e4e044d2120c9a5a58f07bebaa24a1a93b)

scottmcm · 2024-04-18T18:06:22Z

@bors try @rust-timer queue

bors · 2024-04-18T18:07:35Z

⌛ Trying commit 580c6a1 with merge 2746faf...

Add a lower bound check to `unicode-table-generator` output This adds a dedicated check for the lower bound (if it is outside of ASCII range) to the output of the `unicode-table-generator` tool. This generalized the ASCII-only fast-path, but only for the `Grapheme_Extend` property for now, as that is the only one with a lower bound outside of ASCII.

bors · 2024-04-18T19:40:30Z

☀️ Try build successful - checks-actions
Build commit: 2746faf (2746faf07f3279b425969d2ad1069c1f955af851)

rust-timer · 2024-04-18T21:12:16Z

Finished benchmarking commit (2746faf): comparison URL.

Overall result: ❌ regressions - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.9%	[3.9%, 3.9%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	8.7%	[8.7%, 8.7%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-1.0%	[-1.0%, -1.0%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	3.9%	[-1.0%, 8.7%]	2

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.6%	[3.6%, 3.6%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 676.731s -> 676.644s (-0.01%)
Artifact size: 316.11 MiB -> 315.35 MiB (-0.24%)

scottmcm

Perf looks happy with the inline -- sorry it took weeks to actually get a run -- so I think this is nearly good to go. Just had a couple minor requests.

library/core/src/unicode/unicode_data.rs

This adds a dedicated check for the lower bound (if it is outside of ASCII range) to the output of the `unicode-table-generator` tool. This generalized the ASCII-only fast-path, but only for the `Grapheme_Extend` property for now, as that is the only one with a lower bound outside of ASCII.

Swatinem · 2024-04-20T08:17:50Z

Rebased and applied the suggestions.

Swatinem · 2024-04-20T08:20:40Z

@rustbot ready

scottmcm · 2024-04-20T14:57:14Z

Thanks!
@bors r+ rollup=iffy (should now be perf-neutral so I don't think it needs never)

This also has me curious what would happen if we added a lower-bound check outside the probably-not-inlined part for everything, but that's definitely not a this-PR kind of thing 🙃

bors · 2024-04-20T14:57:17Z

📌 Commit 488598c has been approved by scottmcm

It is now in the queue for this repository.

bors · 2024-04-20T20:33:28Z

⌛ Testing commit 488598c with merge dbce3b4...

bors · 2024-04-20T22:34:48Z

☀️ Test successful - checks-actions
Approved by: scottmcm
Pushing dbce3b4 to master...

rust-timer · 2024-04-21T00:27:07Z

Finished benchmarking commit (dbce3b4): comparison URL.

Overall result: ❌ regressions - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	1.3%	[1.3%, 1.3%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	4.4%	[0.8%, 8.0%]	2
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	4.4%	[0.8%, 8.0%]	2

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.8%	[-3.0%, -2.6%]	2
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 671.454s -> 673.418s (0.29%)
Artifact size: 315.20 MiB -> 315.27 MiB (0.02%)

rustbot assigned scottmcm Mar 5, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Mar 5, 2024

Swatinem mentioned this pull request Mar 5, 2024

Add ASCII fast-path for char::is_grapheme_extended #121138

Merged

tgross35 reviewed Mar 5, 2024

View reviewed changes

scottmcm reviewed Mar 5, 2024

View reviewed changes

Swatinem force-pushed the unicode-gen-fastpath branch 2 times, most recently from ad7b782 to 6d7daa0 Compare March 5, 2024 19:25

cuviper mentioned this pull request Mar 6, 2024

Add a fast-path to Debug ASCII &str #121150

Open

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Mar 8, 2024

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Mar 8, 2024

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Mar 27, 2024

scottmcm closed this Apr 18, 2024

scottmcm reopened this Apr 18, 2024

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 18, 2024

scottmcm requested changes Apr 18, 2024

View reviewed changes

library/core/src/unicode/unicode_data.rs Outdated Show resolved Hide resolved

library/core/src/unicode/unicode_data.rs Outdated Show resolved Hide resolved

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 18, 2024

Swatinem force-pushed the unicode-gen-fastpath branch from 580c6a1 to 488598c Compare April 20, 2024 08:17

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 20, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 20, 2024

bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 20, 2024

bors merged commit dbce3b4 into rust-lang:master Apr 20, 2024
13 checks passed

rustbot added this to the 1.79.0 milestone Apr 20, 2024

bors mentioned this pull request Apr 20, 2024

Add APIs for dealing with titlecase #122668

Open

	writeln!(&mut self.file, " (c as u32) >= {first_code_point} &&").unwrap();
	writeln!(&mut self.file, " (c as u32) >= {first_code_point:#04x} &&").unwrap();

Add a lower bound check to unicode-table-generator output #122013

Add a lower bound check to unicode-table-generator output #122013

Conversation

Swatinem commented Mar 5, 2024

rustbot commented Mar 5, 2024

Swatinem commented Mar 5, 2024

workingjubilee commented Mar 5, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tgross35 commented Mar 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Swatinem commented Mar 5, 2024

cuviper commented Mar 8, 2024

This comment has been minimized.

bors commented Mar 8, 2024

bors commented Mar 8, 2024

This comment has been minimized.

rust-timer commented Mar 8, 2024

Overall result: ✅ improvements - no action needed

Swatinem commented Mar 8, 2024

scottmcm commented Mar 27, 2024

This comment has been minimized.

bors commented Mar 27, 2024

bors commented Mar 27, 2024

scottmcm commented Apr 18, 2024

This comment has been minimized.

bors commented Apr 18, 2024

bors commented Apr 18, 2024

This comment has been minimized.

rust-timer commented Apr 18, 2024

Overall result: ❌ regressions - no action needed

scottmcm left a comment

Choose a reason for hiding this comment

Swatinem commented Apr 20, 2024

Swatinem commented Apr 20, 2024

scottmcm commented Apr 20, 2024

bors commented Apr 20, 2024

bors commented Apr 20, 2024

bors commented Apr 20, 2024

rust-timer commented Apr 21, 2024

Overall result: ❌ regressions - no action needed

Add a lower bound check to `unicode-table-generator` output #122013

Add a lower bound check to `unicode-table-generator` output #122013

workingjubilee commented Mar 5, 2024 •

edited