Further improve `space_between` #120227

nnethercote · 2024-01-22T09:19:07Z

space_between is used by print_tts to decide when spaces should be put between tokens. This PR improves it in two ways:

avoid unnecessary spaces before semicolons, and
don't omit some necessary spaces before/after some punctuation symbols.

r? @petrochenkov

This gives better output for code produced by proc macros.

There are a number of cases where we erroneously omit the space between two tokens, all involving an exception to a more general case. The affected tokens are `$`, `!`, `.`, `,`, and `let` followed by a parenthesis. This fixes a lot of FIXME comments.

nnethercote · 2024-01-22T09:22:33Z

#117433 was an attempt to make space_between much cleverer, which didn't work out. This PR is a greatly cut down version of that, which gets some nice improvements with very little extra complexity.

It will need a crater run before being merged, but based on the crater run from #117433 I think it should be ok, because I haven't include the cases that caused crater failures in #117433, which were mostly about spacing around : and :: tokens.

petrochenkov · 2024-01-22T15:30:21Z

@bors try

…tween, r=<try> Further improve `space_between` `space_between` is used by `print_tts` to decide when spaces should be put between tokens. This PR improves it in two ways: - avoid unnecessary spaces before semicolons, and - don't omit some necessary spaces before/after some punctuation symbols. r? `@petrochenkov`

bors · 2024-01-22T15:31:31Z

⌛ Trying commit 1fbabee with merge ad2e388...

bors · 2024-01-22T16:57:46Z

☀️ Try build successful - checks-actions
Build commit: ad2e388 (ad2e3884fa8975e4827c1b0b0af203fc59856ef2)

petrochenkov · 2024-01-22T21:51:45Z

@craterbot check

craterbot · 2024-01-22T21:51:57Z

👌 Experiment pr-120227 created and queued.
🤖 Automatically detected try build ad2e388
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

craterbot · 2024-01-23T15:55:36Z

🚧 Experiment pr-120227 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

craterbot · 2024-01-25T11:40:28Z

🎉 Experiment pr-120227 is completed!
📊 6 regressed and 2 fixed (409984 total)
📰 Open the full report.

⚠️ If you notice any spurious failure please add them to the blacklist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

petrochenkov · 2024-01-25T23:23:20Z

Only 2 regressions out of 6 are spurious.
@rustbot author

nnethercote · 2024-01-25T23:29:00Z

odbc-futures-0.2.2

This line is the problem, it does substring matching with "extern crate crate ;", and there is no longer a space before the semicolon.

The crate has only 438 downloads ever and hasn't been updated in 5 years.

KittyCAD.kcl-lsp.844488870375d9e36254825ab04a2cfe5d5fd107

Due to kcl-lib/derive-docs, as described here. In short, this:

Result < Box < ExtrudeGroup >, KclError > {}

becomes this:

Result < Box < ExtrudeGroup > , KclError > {}

which breaks some string matching in a proc macro.

kcl-lib and derive-docs have had both had updates in the past two months.

carghai.contain.be6f4b896c7594bac76ae43a2027ed7dd602a433

Due to const-random use, seen before here, so I'm surprised it's showing up here.

[INFO] [stdout] error: proc macro panicked
[INFO] [stdout]   --> src/encryption/base.rs:11:32
[INFO] [stdout]    |
[INFO] [stdout] 11 | const RANDOM_BYTES: [u8; 16] = const_random!([u8 ; 16]);

gen-nested-iter-yield-0.1.3

Substring matching done here in the nested_iter_yield proc macro. I'm not sure the exact problem, probably . or , changes.

Crate has 3,376 total downloads ever, and was last updated almost 2 years ago.

Malikazz.ProjectEuler.81ee5516c17a5425ef7b0b2dbf8ef010031deefc, tperdue321.rust_structs.c5244df34b978d5d014720c1fa42514cc89a7c37, Mallig.advent-of-code-2020.27889bb9ca6c8080172bef594543c49e52fc524c (fixed)

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: unable to start unit "docker-937d68c71184d5335f9a4e5480e9d18bbcc3b5922aec6d9c73612f74bbdad7d2.scope

Seems unrelated to this PR, looks like temporary infra problems.

lcdr.lu_packets.144e56d2c46043da7924ffa34705f763982676fc (fixed)

error[E0275]: overflow evaluating the requirement `BitWriter<endio_bit::endian::BigEndian, _>: std::io::Write`
  |
  = help: consider increasing the recursion limit by adding a `#![recursion_limit = "256"]` attribute to your crate (`lu_packets`)
  = note: required for `BitWriter<endio_bit::endian::BigEndian, _>` to implement `EWrite<LittleEndian>`
  = note: required for `&'a [_]` to implement `for<'a> Serialize<LittleEndian, BitWriter<endio_bit::endian::BigEndian, _>>`
  = note: 125 redundant requirements hidden
  = note: required for `&Option<[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[...]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]>` to implement `Serialize<LittleEndian, BitWriter<endio_bit::endian::BigEndian, _>>`
  = note: the full type name has been written to '/opt/rustwide/target/debug/deps/lu_packets-8f97e4ad4ec14e69.long-type-5894850763885052302.txt'

Old failure was due to hitting the recursion limit. Unclear why it worked now, but seems unrelated to this PR.

nnethercote · 2024-01-26T03:17:24Z

carghai.contain.be6f4b896c7594bac76ae43a2027ed7dd602a433

Due to const-random use, seen before here, so I'm surprised it's showing up here.

This would make sense if crater is comparing this PR against release Rust, rather than against nightly Rust.

I filed tkaitchuck/constrandom#33 to fix const-random, making the proc macro iterate over the token trees rather than using to_string and then matching substrings. This seemed a reasonable way to do it, but I've never done it before, so I'd be happy to hear if there is a better way. (The use of syn seems like overkill for that proc macro, because it's very simple.)

nnethercote · 2024-01-29T00:32:36Z

@petrochenkov: with tkaitchuck/constrandom#33 filed, I think the only other breakage that matters is the kcl-lib/derive-docs case, where both crates are written by the same people.

I suggest we merge this PR and then I'll file an issue on derive-docs. And then perhaps add something to the release docs, as per #119875.

@rustbot reviewer

petrochenkov · 2024-01-29T15:44:11Z

Okay, seems reasonable.
@bors r+

bors · 2024-01-29T15:44:13Z

📌 Commit 1fbabee has been approved by petrochenkov

It is now in the queue for this repository.

bors · 2024-01-30T09:23:53Z

⌛ Testing commit 1fbabee with merge 323c4b4...

bors · 2024-01-30T13:38:53Z

💥 Test timed out

rust-log-analyzer · 2024-01-30T13:50:39Z

A job failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

nnethercote · 2024-01-30T20:20:12Z

@bors retry

bors · 2024-01-31T02:01:46Z

⌛ Testing commit 1fbabee with merge 80deabd...

bors · 2024-01-31T04:01:09Z

☀️ Test successful - checks-actions
Approved by: petrochenkov
Pushing 80deabd to master...

rust-timer · 2024-01-31T05:16:33Z

Finished benchmarking commit (80deabd): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Next Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please open an issue or create a new PR that fixes the regressions, add a comment linking to the newly created issue or PR, and then add the perf-regression-triaged label to this PR.

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	3.8%	[3.8%, 3.8%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	3.8%	[3.8%, 3.8%]	1

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.3%	[0.6%, 3.8%]	46
Regressions ❌ (secondary)	3.0%	[0.7%, 5.2%]	77
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	2.3%	[0.6%, 3.8%]	46

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	3.7%	[3.7%, 3.7%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	3.7%	[3.7%, 3.7%]	1

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 660.046s -> 661.683s (0.25%)
Artifact size: 308.08 MiB -> 308.09 MiB (0.01%)

nnethercote · 2024-01-31T05:52:31Z

I filed an issue on derive-docs: KittyCAD/modeling-app#1339

therealprof · 2024-01-31T13:32:17Z

@nnethercote https://perf.rust-lang.org/compare.html?start=cb4d9a1902b3ea17e93872dafb76d24aa6295c47&end=80deabd0987201e1b8d060400f50e03309a0105e&stat=max-rss doesn't look great to me... Is the noticable increase in RSS expected?

Kobzol · 2024-01-31T13:41:14Z

(Discussion about that ongoing here).

Mark-Simulacrum · 2024-01-31T13:42:22Z

Tagging relnotes for future compat notice. See #120227 (comment) for crater.

trevyn · 2024-02-07T20:39:08Z

This is hitting my turbosql crate, I think because of matching on ; as above: https://github.com/trevyn/turbosql/actions/runs/7735113959/job/21090247508

nnethercote · 2024-02-08T03:13:32Z

This is hitting my turbosql crate, I think because of matching on ; as above: https://github.com/trevyn/turbosql/actions/runs/7735113959/job/21090247508

Looking at the code, I would guess these regexes are to blame.

For the first regex, changing this:

^Option < \[u8 ; \d+\] >$

to this:

^Option < \[u8\s*; \d+\] >$

should fix the problem caused by this particular PR. Changing it to this:

^Option\s*<\s*\[\s*u8\s*;\s*\d+\s*\]\s*>$

would be more robust against possible future pretty-printing changes. But the awfulness of that regex shows the limits of substring matching for this purpose. The fully robust way to do this kind of thing is at the token level, like I did in tkaitchuck/constrandom#33.

rust-lang/rust#120227 (comment)

Pkgsrc changes: * Adapt checksums and patches. Upstream chnages: Version 1.77.0 (2024-03-21) ========================== - [Reveal opaque types within the defining body for exhaustiveness checking.] (rust-lang/rust#116821) - [Stabilize C-string literals.] (rust-lang/rust#117472) - [Stabilize THIR unsafeck.] (rust-lang/rust#117673) - [Add lint `static_mut_refs` to warn on references to mutable statics.] (rust-lang/rust#117556) - [Support async recursive calls (as long as they have indirection).] (rust-lang/rust#117703) - [Undeprecate lint `unstable_features` and make use of it in the compiler.] (rust-lang/rust#118639) - [Make inductive cycles in coherence ambiguous always.] (rust-lang/rust#118649) - [Get rid of type-driven traversal in const-eval interning] (rust-lang/rust#119044), only as a [future compatiblity lint] (rust-lang/rust#122204) for now. - [Deny braced macro invocations in let-else.] (rust-lang/rust#119062) Compiler -------- - [Include lint `soft_unstable` in future breakage reports.] (rust-lang/rust#116274) - [Make `i128` and `u128` 16-byte aligned on x86-based targets.] (rust-lang/rust#116672) - [Use `--verbose` in diagnostic output.] (rust-lang/rust#119129) - [Improve spacing between printed tokens.] (rust-lang/rust#120227) - [Merge the `unused_tuple_struct_fields` lint into `dead_code`.] (rust-lang/rust#118297) - [Error on incorrect implied bounds in well-formedness check] (rust-lang/rust#118553), with a temporary exception for Bevy. - [Fix coverage instrumentation/reports for non-ASCII source code.] (rust-lang/rust#119033) - [Fix `fn`/`const` items implied bounds and well-formedness check.] (rust-lang/rust#120019) - [Promote `riscv32{im|imafc}-unknown-none-elf` targets to tier 2.] (rust-lang/rust#118704) - Add several new tier 3 targets: - [`aarch64-unknown-illumos`] (rust-lang/rust#112936) - [`hexagon-unknown-none-elf`] (rust-lang/rust#117601) - [`riscv32imafc-esp-espidf`] (rust-lang/rust#119738) - [`riscv32im-risc0-zkvm-elf`] (rust-lang/rust#117958) Refer to Rust's [platform support page][platform-support-doc] for more information on Rust's tiered platform support. Libraries --------- - [Implement `From<&[T; N]>` for `Cow<[T]>`.] (rust-lang/rust#113489) - [Remove special-case handling of `vec.split_off (0)`.](rust-lang/rust#119917) Stabilized APIs --------------- - [`array::each_ref`] (https://doc.rust-lang.org/stable/std/primitive.array.html#method.each_ref) - [`array::each_mut`] (https://doc.rust-lang.org/stable/std/primitive.array.html#method.each_mut) - [`core::net`] (https://doc.rust-lang.org/stable/core/net/index.html) - [`f32::round_ties_even`] (https://doc.rust-lang.org/stable/std/primitive.f32.html#method.round_ties_even) - [`f64::round_ties_even`] (https://doc.rust-lang.org/stable/std/primitive.f64.html#method.round_ties_even) - [`mem::offset_of!`] (https://doc.rust-lang.org/stable/std/mem/macro.offset_of.html) - [`slice::first_chunk`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.first_chunk) - [`slice::first_chunk_mut`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.first_chunk_mut) - [`slice::split_first_chunk`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.split_first_chunk) - [`slice::split_first_chunk_mut`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.split_first_chunk_mut) - [`slice::last_chunk`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.last_chunk) - [`slice::last_chunk_mut`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.last_chunk_mut) - [`slice::split_last_chunk`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.split_last_chunk) - [`slice::split_last_chunk_mut`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.split_last_chunk_mut) - [`slice::chunk_by`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.chunk_by) - [`slice::chunk_by_mut`] (https://doc.rust-lang.org/stable/std/primitive.slice.html#method.chunk_by_mut) - [`Bound::map`] (https://doc.rust-lang.org/stable/std/ops/enum.Bound.html#method.map) - [`File::create_new`] (https://doc.rust-lang.org/stable/std/fs/struct.File.html#method.create_new) - [`Mutex::clear_poison`] (https://doc.rust-lang.org/stable/std/sync/struct.Mutex.html#method.clear_poison) - [`RwLock::clear_poison`] (https://doc.rust-lang.org/stable/std/sync/struct.RwLock.html#method.clear_poison) Cargo ----- - [Extend the build directive syntax with `cargo::`.] (rust-lang/cargo#12201) - [Stabilize metadata `id` format as `PackageIDSpec`.] (rust-lang/cargo#12914) - [Pull out as `cargo-util-schemas` as a crate.] (rust-lang/cargo#13178) - [Strip all debuginfo when debuginfo is not requested.] (rust-lang/cargo#13257) - [Inherit jobserver from env for all kinds of runners.] (rust-lang/cargo#12776) - [Deprecate rustc plugin support in cargo.] (rust-lang/cargo#13248) Rustdoc ----- - [Allows links in markdown headings.] (rust-lang/rust#117662) - [Search for tuples and unit by type with `()`.] (rust-lang/rust#118194) - [Clean up the source sidebar's hide button.] (rust-lang/rust#119066) - [Prevent JS injection from `localStorage`.] (rust-lang/rust#120250) Misc ---- - [Recommend version-sorting for all sorting in style guide.] (rust-lang/rust#115046) Internal Changes ---------------- These changes do not affect any public interfaces of Rust, but they represent significant improvements to the performance or internals of rustc and related tools. - [Add more weirdness to `weird-exprs.rs`.] (rust-lang/rust#119028)

Don't insert spaces before most semicolons in print_tts.

41e4a3e

This gives better output for code produced by proc macros.

rustbot assigned petrochenkov Jan 22, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jan 22, 2024

Fix some cases in space_between.

1fbabee

There are a number of cases where we erroneously omit the space between two tokens, all involving an exception to a more general case. The affected tokens are `$`, `!`, `.`, `,`, and `let` followed by a parenthesis. This fixes a lot of FIXME comments.

nnethercote force-pushed the further-improve-space_between branch from 60d86ae to 1fbabee Compare January 22, 2024 09:19

petrochenkov added S-waiting-on-crater Status: Waiting on a crater run to be completed. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 22, 2024

craterbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-crater Status: Waiting on a crater run to be completed. labels Jan 25, 2024

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 25, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 29, 2024

bors removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jan 29, 2024

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jan 30, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 30, 2024

bors added the merged-by-bors This PR was explicitly merged by bors. label Jan 31, 2024

bors merged commit 80deabd into rust-lang:master Jan 31, 2024
12 checks passed

rustbot added this to the 1.77.0 milestone Jan 31, 2024

nnethercote deleted the further-improve-space_between branch January 31, 2024 05:12

rustbot added the perf-regression Performance regression. label Jan 31, 2024

nnethercote mentioned this pull request Jan 31, 2024

regression: macro expansion/resolution change of some kind #119875

Closed

Mark-Simulacrum added the relnotes Marks issues that should be documented in the release notes of the next release. label Jan 31, 2024

smoelius mentioned this pull request Feb 1, 2024

Update inner_attributes_example test trailofbits/cast_checks#32

Merged

Mark-Simulacrum removed the perf-regression Performance regression. label Feb 5, 2024

trevyn added a commit to trevyn/turbosql that referenced this pull request Feb 8, 2024

update for rust 1.77 pretty-printing changes

e68beb9

rust-lang/rust#120227 (comment)

compiler-errors mentioned this pull request Feb 26, 2024

regression: no rules expected the token . #121611

Open

biryukovmaxim mentioned this pull request Mar 24, 2024

fix deterministic.rs 'seal changed -was: 0xa7a4 now 0x544d' kaspanet/rusty-kaspa#438

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Further improve `space_between` #120227

Further improve `space_between` #120227

nnethercote commented Jan 22, 2024

nnethercote commented Jan 22, 2024 •

edited

petrochenkov commented Jan 22, 2024

bors commented Jan 22, 2024

bors commented Jan 22, 2024

petrochenkov commented Jan 22, 2024

craterbot commented Jan 22, 2024

craterbot commented Jan 23, 2024

craterbot commented Jan 25, 2024

petrochenkov commented Jan 25, 2024 •

edited

nnethercote commented Jan 25, 2024

nnethercote commented Jan 26, 2024

carghai.contain.be6f4b896c7594bac76ae43a2027ed7dd602a433

nnethercote commented Jan 29, 2024

petrochenkov commented Jan 29, 2024

bors commented Jan 29, 2024

bors commented Jan 30, 2024

bors commented Jan 30, 2024

rust-log-analyzer commented Jan 30, 2024

nnethercote commented Jan 30, 2024

bors commented Jan 31, 2024

bors commented Jan 31, 2024

rust-timer commented Jan 31, 2024

nnethercote commented Jan 31, 2024

therealprof commented Jan 31, 2024

Kobzol commented Jan 31, 2024

Mark-Simulacrum commented Jan 31, 2024

trevyn commented Feb 7, 2024

nnethercote commented Feb 8, 2024

Further improve space_between #120227

Further improve space_between #120227

Conversation

nnethercote commented Jan 22, 2024

nnethercote commented Jan 22, 2024 • edited

petrochenkov commented Jan 22, 2024

bors commented Jan 22, 2024

bors commented Jan 22, 2024

petrochenkov commented Jan 22, 2024

craterbot commented Jan 22, 2024

craterbot commented Jan 23, 2024

craterbot commented Jan 25, 2024

petrochenkov commented Jan 25, 2024 • edited

nnethercote commented Jan 25, 2024

odbc-futures-0.2.2

KittyCAD.kcl-lsp.844488870375d9e36254825ab04a2cfe5d5fd107

carghai.contain.be6f4b896c7594bac76ae43a2027ed7dd602a433

gen-nested-iter-yield-0.1.3

Malikazz.ProjectEuler.81ee5516c17a5425ef7b0b2dbf8ef010031deefc, tperdue321.rust_structs.c5244df34b978d5d014720c1fa42514cc89a7c37, Mallig.advent-of-code-2020.27889bb9ca6c8080172bef594543c49e52fc524c (fixed)

lcdr.lu_packets.144e56d2c46043da7924ffa34705f763982676fc (fixed)

nnethercote commented Jan 26, 2024

carghai.contain.be6f4b896c7594bac76ae43a2027ed7dd602a433

nnethercote commented Jan 29, 2024

petrochenkov commented Jan 29, 2024

bors commented Jan 29, 2024

bors commented Jan 30, 2024

bors commented Jan 30, 2024

rust-log-analyzer commented Jan 30, 2024

nnethercote commented Jan 30, 2024

bors commented Jan 31, 2024

bors commented Jan 31, 2024

rust-timer commented Jan 31, 2024

Overall result: ❌ regressions - ACTION NEEDED

Instruction count

Max RSS (memory usage)

Cycles

Binary size

nnethercote commented Jan 31, 2024

therealprof commented Jan 31, 2024

Kobzol commented Jan 31, 2024

Mark-Simulacrum commented Jan 31, 2024

trevyn commented Feb 7, 2024

nnethercote commented Feb 8, 2024

Further improve `space_between` #120227

Further improve `space_between` #120227

nnethercote commented Jan 22, 2024 •

edited

petrochenkov commented Jan 25, 2024 •

edited