Skip to content

Clean rustc/parse/src/lexer to improve maintainability #144239

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jul 23, 2025
Merged

Clean rustc/parse/src/lexer to improve maintainability #144239

merged 1 commit into from
Jul 23, 2025

Conversation

xizheyin
Copy link
Member

This PR refactors the lexer code to improve maintainability and eliminate code duplication.
In the first commit, I improve the error handling:

  • rename make_unclosed_delims_error to more appropriate make_mismatched_closing_delims_errors
  • changes return type from Option to Vec<Diag> to avoid lengthy vec processing at lex_token_trees
  • use splice instead of extend to make the logic clearer, since errs sounds more generic and better suited as a return value

In the second commit, I replace the magic number 5 with UNCLOSED_DELIMITER_SHOW_LIMIT constant.

In the third commit, I moves eof_err function below parsing logic for better code flow.

In the forth one, I extract calculate_spacing function to eliminate duplicate spacing logic between bump and bump_minimal functions.

r? compiler

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jul 21, 2025
@xizheyin xizheyin changed the title Clean lexer Clean rustc/parse/src/lexer to improve maintainability Jul 21, 2025
fee1-dead
fee1-dead requested changes Jul 21, 2025
View reviewed changes
@rustbot rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 21, 2025
@xizheyin
Copy link
Member Author

@rustbot ready

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 22, 2025
@fee1-dead
Copy link
Member

Please squash the commits

@xizheyin
Copy link
Member Author

ready. :)

@fee1-dead
Copy link
Member

@bors r+ rollup

@bors
Copy link
Collaborator

bors commented Jul 22, 2025

📌 Commit 7a50fe7 has been approved by fee1-dead

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 22, 2025
@fee1-dead
Copy link
Member

@bors r-

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jul 22, 2025
fee1-dead
fee1-dead requested changes Jul 22, 2025
View reviewed changes
@xizheyin
Clean code for rustc_parse/src/lexer
2832517 
1. Rename `make_unclosed_delims_error` and return `Vec<Diag>`
2. change magic number `unclosed_delimiter_show_limit` to const
3. move `eof_err` below parsing logic
4. Add `calculate_spacing` for `bump` and `bump_minimal`

Signed-off-by: xizheyin <xizheyin@smail.nju.edu.cn>
@xizheyin
Copy link
Member Author

@rustbot ready

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 22, 2025
@fee1-dead
Copy link
Member

@bors r+ rollup

@bors
Copy link
Collaborator

bors commented Jul 23, 2025

📌 Commit 2832517 has been approved by fee1-dead

It is now in the queue for this repository.

@bors bors removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 23, 2025
@bors bors added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Jul 23, 2025
jhpratt added a commit to jhpratt/rust that referenced this pull request Jul 23, 2025
@jhpratt
Rollup merge of rust-lang#144239 - xizheyin:clean-lexer, r=fee1-dead
48caa60 
Clean `rustc/parse/src/lexer` to improve maintainability

This PR refactors the lexer code to improve maintainability and eliminate code duplication.
In the first commit, I improve the error handling:
- rename `make_unclosed_delims_error` to more appropriate `make_mismatched_closing_delims_errors`
- changes return type from Option<Diag> to `Vec<Diag>` to avoid lengthy vec processing at `lex_token_trees`
- use `splice` instead of `extend` to make the logic clearer, since `errs` sounds more generic and better suited as a return value

In the second commit, I replace the magic number 5 with UNCLOSED_DELIMITER_SHOW_LIMIT constant.

In the third commit, I moves `eof_err` function below parsing logic for better code flow.

In the forth one, I extract `calculate_spacing` function to eliminate duplicate spacing logic between `bump` and `bump_minimal` functions.

r? compiler
@jhpratt jhpratt mentioned this pull request Jul 23, 2025
Closed
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Jul 23, 2025
@matthiaskrgr
Rollup merge of rust-lang#144239 - xizheyin:clean-lexer, r=fee1-dead
7599e0c 
Clean `rustc/parse/src/lexer` to improve maintainability

This PR refactors the lexer code to improve maintainability and eliminate code duplication.
In the first commit, I improve the error handling:
- rename `make_unclosed_delims_error` to more appropriate `make_mismatched_closing_delims_errors`
- changes return type from Option<Diag> to `Vec<Diag>` to avoid lengthy vec processing at `lex_token_trees`
- use `splice` instead of `extend` to make the logic clearer, since `errs` sounds more generic and better suited as a return value

In the second commit, I replace the magic number 5 with UNCLOSED_DELIMITER_SHOW_LIMIT constant.

In the third commit, I moves `eof_err` function below parsing logic for better code flow.

In the forth one, I extract `calculate_spacing` function to eliminate duplicate spacing logic between `bump` and `bump_minimal` functions.

r? compiler
@matthiaskrgr matthiaskrgr mentioned this pull request Jul 23, 2025
Closed
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Jul 23, 2025
@matthiaskrgr
Rollup merge of rust-lang#144239 - xizheyin:clean-lexer, r=fee1-dead
a4e3745 
Clean `rustc/parse/src/lexer` to improve maintainability

This PR refactors the lexer code to improve maintainability and eliminate code duplication.
In the first commit, I improve the error handling:
- rename `make_unclosed_delims_error` to more appropriate `make_mismatched_closing_delims_errors`
- changes return type from Option<Diag> to `Vec<Diag>` to avoid lengthy vec processing at `lex_token_trees`
- use `splice` instead of `extend` to make the logic clearer, since `errs` sounds more generic and better suited as a return value

In the second commit, I replace the magic number 5 with UNCLOSED_DELIMITER_SHOW_LIMIT constant.

In the third commit, I moves `eof_err` function below parsing logic for better code flow.

In the forth one, I extract `calculate_spacing` function to eliminate duplicate spacing logic between `bump` and `bump_minimal` functions.

r? compiler
@matthiaskrgr matthiaskrgr mentioned this pull request Jul 23, 2025
Closed
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Jul 23, 2025
@matthiaskrgr
Rollup merge of rust-lang#144239 - xizheyin:clean-lexer, r=fee1-dead
d49ba15 
Clean `rustc/parse/src/lexer` to improve maintainability

This PR refactors the lexer code to improve maintainability and eliminate code duplication.
In the first commit, I improve the error handling:
- rename `make_unclosed_delims_error` to more appropriate `make_mismatched_closing_delims_errors`
- changes return type from Option<Diag> to `Vec<Diag>` to avoid lengthy vec processing at `lex_token_trees`
- use `splice` instead of `extend` to make the logic clearer, since `errs` sounds more generic and better suited as a return value

In the second commit, I replace the magic number 5 with UNCLOSED_DELIMITER_SHOW_LIMIT constant.

In the third commit, I moves `eof_err` function below parsing logic for better code flow.

In the forth one, I extract `calculate_spacing` function to eliminate duplicate spacing logic between `bump` and `bump_minimal` functions.

r? compiler
@matthiaskrgr matthiaskrgr mentioned this pull request Jul 23, 2025
Closed
bors added a commit that referenced this pull request Jul 23, 2025
@bors
Auto merge of #144349 - matthiaskrgr:rollup-0eegcyt, r=matthiaskrgr
6adc93b 
Rollup of 10 pull requests

Successful merges:

 - #144173 (Remove tidy checks for `tests/ui/issues/`)
 - #144234 (Fix broken TLS destructors on 32-bit win7)
 - #144239 (Clean `rustc/parse/src/lexer` to improve maintainability)
 - #144247 (coretests/num: use ldexp instead of hard-coding a power of 2)
 - #144256 (Don't ICE on non-TypeId metadata within TypeId)
 - #144290 (update tests/ui/SUMMARY.md)
 - #144292 (mbe: Use concrete type for `get_unused_rule`)
 - #144298 (coverage: Enlarge empty spans during MIR instrumentation, not codegen)
 - #144311 (Add powerpc64le-unknown-linux-musl to CI rustc targets)
 - #144315 (bootstrap: add package.json and package-lock.json to dist tarball)

r? `@ghost`
`@rustbot` modify labels: rollup
GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this pull request Jul 23, 2025
@GuillaumeGomez
Rollup merge of rust-lang#144239 - xizheyin:clean-lexer, r=fee1-dead
a3e9041 
Clean `rustc/parse/src/lexer` to improve maintainability

This PR refactors the lexer code to improve maintainability and eliminate code duplication.
In the first commit, I improve the error handling:
- rename `make_unclosed_delims_error` to more appropriate `make_mismatched_closing_delims_errors`
- changes return type from Option<Diag> to `Vec<Diag>` to avoid lengthy vec processing at `lex_token_trees`
- use `splice` instead of `extend` to make the logic clearer, since `errs` sounds more generic and better suited as a return value

In the second commit, I replace the magic number 5 with UNCLOSED_DELIMITER_SHOW_LIMIT constant.

In the third commit, I moves `eof_err` function below parsing logic for better code flow.

In the forth one, I extract `calculate_spacing` function to eliminate duplicate spacing logic between `bump` and `bump_minimal` functions.

r? compiler
@GuillaumeGomez GuillaumeGomez mentioned this pull request Jul 23, 2025
Closed
bors added a commit that referenced this pull request Jul 23, 2025
@bors
Auto merge of #144349 - matthiaskrgr:rollup-0eegcyt, r=matthiaskrgr
69e29e8 
Rollup of 10 pull requests

Successful merges:

 - #144173 (Remove tidy checks for `tests/ui/issues/`)
 - #144234 (Fix broken TLS destructors on 32-bit win7)
 - #144239 (Clean `rustc/parse/src/lexer` to improve maintainability)
 - #144247 (coretests/num: use ldexp instead of hard-coding a power of 2)
 - #144256 (Don't ICE on non-TypeId metadata within TypeId)
 - #144290 (update tests/ui/SUMMARY.md)
 - #144292 (mbe: Use concrete type for `get_unused_rule`)
 - #144298 (coverage: Enlarge empty spans during MIR instrumentation, not codegen)
 - #144311 (Add powerpc64le-unknown-linux-musl to CI rustc targets)
 - #144315 (bootstrap: add package.json and package-lock.json to dist tarball)

r? `@ghost`
`@rustbot` modify labels: rollup

try-job: x86_64-gnu-aux
@matthiaskrgr matthiaskrgr mentioned this pull request Jul 23, 2025
Merged
bors added a commit that referenced this pull request Jul 23, 2025
@bors
Auto merge of #144360 - matthiaskrgr:rollup-b6ej0mm, r=matthiaskrgr
29a5872 
Rollup of 9 pull requests

Successful merges:

 - #144173 (Remove tidy checks for `tests/ui/issues/`)
 - #144234 (Fix broken TLS destructors on 32-bit win7)
 - #144239 (Clean `rustc/parse/src/lexer` to improve maintainability)
 - #144256 (Don't ICE on non-TypeId metadata within TypeId)
 - #144290 (update tests/ui/SUMMARY.md)
 - #144292 (mbe: Use concrete type for `get_unused_rule`)
 - #144298 (coverage: Enlarge empty spans during MIR instrumentation, not codegen)
 - #144311 (Add powerpc64le-unknown-linux-musl to CI rustc targets)
 - #144315 (bootstrap: add package.json and package-lock.json to dist tarball)

r? `@ghost`
`@rustbot` modify labels: rollup
@bors bors merged commit 8034261 into rust-lang:master Jul 23, 2025
11 checks passed
@rustbot rustbot added this to the 1.90.0 milestone Jul 23, 2025
rust-timer added a commit that referenced this pull request Jul 23, 2025
@rust-timer
Unrolled build for #144239
3170f83 
Rollup merge of #144239 - xizheyin:clean-lexer, r=fee1-dead

Clean `rustc/parse/src/lexer` to improve maintainability

This PR refactors the lexer code to improve maintainability and eliminate code duplication.
In the first commit, I improve the error handling:
- rename `make_unclosed_delims_error` to more appropriate `make_mismatched_closing_delims_errors`
- changes return type from Option<Diag> to `Vec<Diag>` to avoid lengthy vec processing at `lex_token_trees`
- use `splice` instead of `extend` to make the logic clearer, since `errs` sounds more generic and better suited as a return value

In the second commit, I replace the magic number 5 with UNCLOSED_DELIMITER_SHOW_LIMIT constant.

In the third commit, I moves `eof_err` function below parsing logic for better code flow.

In the forth one, I extract `calculate_spacing` function to eliminate duplicate spacing logic between `bump` and `bump_minimal` functions.

r? compiler
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fee1-dead fee1-dead fee1-dead requested changes

Assignees

@fee1-dead fee1-dead

Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
1.90.0
Development

Successfully merging this pull request may close these issues.
4 participants
@xizheyin @fee1-dead @bors @rustbot