rustc_ast: Stop using "string typing" for doc comment tokens #74627

petrochenkov · 2020-07-22T12:17:05Z

Explicitly store their kind and style retrieved during lexing in the token::DocComment.

Also don't "beautify" doc comments before converting them to #[doc] attributes when passing them to macros (both declarative and procedural).
The trimming of empty lines, lines containing only *s, etc is purely a rustdoc's job as a part of its presentation of doc strings to users, rustc must not do this and must pass tokens as precisely as possible internally.

petrochenkov · 2020-07-22T12:18:10Z

This needs a crater run, proc macros sometimes rely on very strange things.
@bors try

bors · 2020-07-22T12:18:22Z

⌛ Trying commit e5c176725dd72cadd04f66f74cb85ad5dd811eb5 with merge 1f8642fb70240fe4e9791c141bd2f3f6bcbf1eb1...

petrochenkov · 2020-07-22T15:00:41Z

@bors try

bors · 2020-07-22T15:00:52Z

⌛ Trying commit 6a57098ae079d8dd9adac6c969a8092d63cf6dfc with merge 22dd2dd1dc07f63ae9394352d412f020d2985684...

bors · 2020-07-22T17:06:38Z

☀️ Try build successful - checks-actions, checks-azure
Build commit: 22dd2dd1dc07f63ae9394352d412f020d2985684 (22dd2dd1dc07f63ae9394352d412f020d2985684)

petrochenkov · 2020-07-22T17:23:37Z

@craterbot check

craterbot · 2020-07-22T17:23:41Z

👌 Experiment pr-74627 created and queued.
🤖 Automatically detected try build 22dd2dd1dc07f63ae9394352d412f020d2985684
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

craterbot · 2020-07-28T13:44:54Z

🚧 Experiment pr-74627 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

craterbot · 2020-07-28T14:21:58Z

🚨 Experiment pr-74627 has encountered an error: command Command { std: "/workspace/cargo-home/bin/rustup-toolchain-install-master" "e22b61bff0bdd08be7665607cb7be3748c8a35d2" "-c" "cargo", kill_on_drop: false } failed
🛠️ If the error is fixed use the retry command.

🆘 Can someone from the infra team check in on this? @rust-lang/infra
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

Mark-Simulacrum · 2020-07-28T17:23:19Z

@craterbot retry seems to have been spurious

craterbot · 2020-07-28T17:23:22Z

🚨 Error: failed to parse the command

🆘 If you have any trouble with Crater please ping @rust-lang/infra!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

Mark-Simulacrum · 2020-07-28T17:38:03Z

@craterbot retry

craterbot · 2020-07-28T17:38:07Z

🛠️ Experiment pr-74627 queued again.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

craterbot · 2020-08-01T06:17:34Z

🚧 Experiment pr-74627 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

craterbot · 2020-08-03T17:44:36Z

🎉 Experiment pr-74627 is completed!
📊 4 regressed and 5 fixed (114124 total)
📰 Open the full report.

⚠️ If you notice any spurious failure please add them to the blacklist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

petrochenkov · 2020-08-03T21:26:12Z

No regressions related to this PR.
r? @Aaron1011, I guess

petrochenkov · 2020-08-04T21:30:30Z

Updated with the requested comments.

Aaron1011 · 2020-08-05T09:14:04Z

@petrochenkov: Could you add a test verifying that proc-macros receive the original doc comment value (e.g. without any whitespace trimming)?

r=me once that's done.

Explicitly store their kind and style retrieved during lexing in the token

…cros Beautify all doc strings in rustdoc instead, including those in `#[doc]` attributes

petrochenkov · 2020-08-06T20:02:16Z

@bors r=Aaron1011

bors · 2020-08-06T20:02:17Z

📌 Commit a7eabec has been approved by Aaron1011

bors · 2020-08-07T13:29:30Z

⌛ Testing commit a7eabec with merge 64f99b4...

bors · 2020-08-07T15:27:59Z

☀️ Test successful - checks-actions, checks-azure
Approved by: Aaron1011
Pushing 64f99b4 to master...

tesuji · 2020-08-12T05:19:53Z

src/librustc_ast/util/comments.rs

+/// and returns `None` if it's a regular comment.
+pub fn line_doc_comment_style(line_comment: &str) -> Option<AttrStyle> {
+    let line_comment = line_comment.as_bytes();
+    assert!(line_comment.starts_with(b"//"));


Is the choice is to panic instead of return None?

We are not using this function on arbitrary strings, only on actual comments, so we can make it stricter.

In that case I think unwrapping at call site seems to be better.

petrochenkov added the S-waiting-on-crater Status: Waiting on a crater run to be completed. label Jul 22, 2020

petrochenkov mentioned this pull request Jul 22, 2020

Refactor doc comment parsing #74209

Closed

This comment has been minimized.

Sign in to view

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-crater Status: Waiting on a crater run to be completed. labels Jul 22, 2020

This comment has been minimized.

Sign in to view

petrochenkov force-pushed the docbeauty2 branch from e5c1767 to 6a57098 Compare July 22, 2020 14:59

craterbot added S-waiting-on-crater Status: Waiting on a crater run to be completed. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 22, 2020

craterbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-crater Status: Waiting on a crater run to be completed. labels Aug 3, 2020

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 4, 2020

petrochenkov force-pushed the docbeauty2 branch from 6a57098 to 291e9d4 Compare August 4, 2020 21:29

petrochenkov added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 4, 2020

petrochenkov force-pushed the docbeauty2 branch from 291e9d4 to e93a29d Compare August 4, 2020 21:32

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 5, 2020

petrochenkov added 5 commits August 6, 2020 22:13

rustc_ast: Stop using "string typing" for doc comment tokens

46f48d3

Explicitly store their kind and style retrieved during lexing in the token

rustc_ast/comments: Modernize some enum reexports

000c070

rustc_expand: Don not beautify doc comments before passing them to ma…

712de2b

…cros Beautify all doc strings in rustdoc instead, including those in `#[doc]` attributes

Fix clippy

6b25c50

Add some comments for magic numbers + Add tests

a7eabec

petrochenkov force-pushed the docbeauty2 branch from e93a29d to a7eabec Compare August 6, 2020 20:01

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 6, 2020

bors added the merged-by-bors This PR was explicitly merged by bors. label Aug 7, 2020

bors merged commit 64f99b4 into rust-lang:master Aug 7, 2020

This was referenced Aug 7, 2020

Remove librustc_ast session globals #74932

Merged

rust_ast::ast => rustc_ast #75120

Merged

tesuji reviewed Aug 12, 2020

View reviewed changes

Aaron1011 mentioned this pull request Jan 2, 2021

regression processing c-style doc comments in proc macros between 1.46 and 1.47 #80545

Open

ehuss mentioned this pull request Apr 7, 2021

breaking change on stable: handling of block doc comments #83982

Closed

cuviper added this to the 1.47.0 milestone May 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustc_ast: Stop using "string typing" for doc comment tokens #74627

rustc_ast: Stop using "string typing" for doc comment tokens #74627

petrochenkov commented Jul 22, 2020 •

edited

Loading

petrochenkov commented Jul 22, 2020

bors commented Jul 22, 2020

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

petrochenkov commented Jul 22, 2020

bors commented Jul 22, 2020

bors commented Jul 22, 2020

petrochenkov commented Jul 22, 2020

craterbot commented Jul 22, 2020

craterbot commented Jul 28, 2020

craterbot commented Jul 28, 2020

Mark-Simulacrum commented Jul 28, 2020

craterbot commented Jul 28, 2020

Mark-Simulacrum commented Jul 28, 2020

craterbot commented Jul 28, 2020

craterbot commented Aug 1, 2020

craterbot commented Aug 3, 2020

petrochenkov commented Aug 3, 2020

petrochenkov commented Aug 4, 2020

Aaron1011 commented Aug 5, 2020

petrochenkov commented Aug 6, 2020

bors commented Aug 6, 2020

bors commented Aug 7, 2020

bors commented Aug 7, 2020

tesuji Aug 12, 2020

petrochenkov Aug 12, 2020

tesuji Aug 12, 2020

rustc_ast: Stop using "string typing" for doc comment tokens #74627

rustc_ast: Stop using "string typing" for doc comment tokens #74627

Conversation

petrochenkov commented Jul 22, 2020 • edited Loading

petrochenkov commented Jul 22, 2020

bors commented Jul 22, 2020

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

petrochenkov commented Jul 22, 2020

bors commented Jul 22, 2020

bors commented Jul 22, 2020

petrochenkov commented Jul 22, 2020

craterbot commented Jul 22, 2020

craterbot commented Jul 28, 2020

craterbot commented Jul 28, 2020

Mark-Simulacrum commented Jul 28, 2020

craterbot commented Jul 28, 2020

Mark-Simulacrum commented Jul 28, 2020

craterbot commented Jul 28, 2020

craterbot commented Aug 1, 2020

craterbot commented Aug 3, 2020

petrochenkov commented Aug 3, 2020

petrochenkov commented Aug 4, 2020

Aaron1011 commented Aug 5, 2020

petrochenkov commented Aug 6, 2020

bors commented Aug 6, 2020

bors commented Aug 7, 2020

bors commented Aug 7, 2020

tesuji Aug 12, 2020

Choose a reason for hiding this comment

petrochenkov Aug 12, 2020

Choose a reason for hiding this comment

tesuji Aug 12, 2020

Choose a reason for hiding this comment

petrochenkov commented Jul 22, 2020 •

edited

Loading