Use a DSL for grammar definition, and add next_token #564

AntonyBlakey · 2023-08-07T09:53:40Z

No description provided.

changeset-bot · 2023-08-07T09:53:44Z

🦋 Changeset detected

Latest commit: da722fe

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@nomicfoundation/slang	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

crates/codegen/grammar/src/grammar.rs

.vscode/settings.json

crates/codegen/grammar/src/lib.rs

crates/codegen/schema/src/types/production.rs

crates/solidity/outputs/npm/crate/build.rs

crates/solidity/outputs/cargo/crate/Cargo.toml

crates/solidity/outputs/npm/build/src/main.rs

crates/solidity/inputs/language/src/lib.rs

crates/solidity/outputs/cargo/tests/src/doc_examples/visitor_api.rs

crates/solidity/outputs/cargo/crate/src/generated/support/optional_helper.rs

crates/solidity/outputs/cargo/crate/src/lib.rs

crates/solidity/outputs/npm/package/src/generated/index.js

crates/codegen/parser_runtime/src/support/parser_result.rs

Xanewok

Looks great, however it's hard for me to do an in-depth review, since I am not familiar with the codebase from before this PR and the scope of the changes is very big, as it basically overhauls the parsing.

Left some notes, happy to merge it once we bring the changes from #561 back 🙏

crates/solidity/inputs/language/src/lib.rs

crates/solidity/outputs/cargo/tests/src/doc_examples/visitor_api.rs

crates/infra/utils/src/codegen/read_write.rs

crates/codegen/parser_runtime/src/templates/language.tera

crates/codegen/parser_runtime/src/support/stream.rs

crates/solidity/inputs/language/src/dsl.rs

crates/solidity/outputs/cargo/crate/src/generated/language.rs

This reverts commit 5b02c94.

AntonyBlakey · 2023-08-23T06:06:29Z

Not a blocker, but I think we should keep it up to date with the generated sources, to eliminate confusion.

I did.

They were disabled in #564 .. This enables them and updates the snapshots.

Based on #564 EDIT: As asked, this was split into supporting PRs (#579, #580, merged now), doesn't use the Error CST node anymore and only contains terminated-by recovery. To support error side-channel in a backtracking scenario, `Vec<ParseError>` was added to the `Stream` which serves more as a parse context now. To backtrack, we first record the position of the stream and how many errors are there; then, we reset the errors and the position to the first recorded `Marker` struct (basically copying what `chumsky` does). This wasn't changed everywhere as `Stream::set_position` in the lexer will never emit errors (only parsers do), nor will the optional trivia parsers. I'm happy to polish the lexer interface a bit to accommodate for this change later, if that's okay. First, I wanted to make sure the approach is fine and accepted, until we proceed with the DelimitedBy error recovery.

@Xanewok

This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @nomicfoundation/slang@0.10.0 ### Minor Changes - [#595](#595) [`1a258c4`](1a258c4) Thanks [@Xanewok](https://github.com/Xanewok)! - Attempt error recovery when parsing incomplete lists - [#564](#564) [`e326a06`](e326a06) Thanks [@AntonyBlakey](https://github.com/AntonyBlakey)! - Parsing operators with missing operands should no longer panic - [#564](#564) [`e326a06`](e326a06) Thanks [@AntonyBlakey](https://github.com/AntonyBlakey)! - Inline parse rules are no longer exposed to the API. - [#564](#564) [`e326a06`](e326a06) Thanks [@AntonyBlakey](https://github.com/AntonyBlakey)! - Scanners are no longer available as methods - use next_token instead - [#564](#564) [`e326a06`](e326a06) Thanks [@AntonyBlakey](https://github.com/AntonyBlakey)! - Scanners are now grouped into contexts to deal with contextual scanning ### Patch Changes - [#601](#601) [`cbd2a79`](cbd2a79) Thanks [@Xanewok](https://github.com/Xanewok)! - Attempt parser error recovery between bracket delimiters - [#599](#599) [`4bbad48`](4bbad48) Thanks [@Xanewok](https://github.com/Xanewok)! - Use correct versions for the `revert` and `global` keywords - [#561](#561) [`cb6a138`](cb6a138) Thanks [@Xanewok](https://github.com/Xanewok)! - Add preliminary documentation for the `solidity_language` Rust package Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

AntonyBlakey requested a review from a team as a code owner August 7, 2023 09:53

AntonyBlakey self-assigned this Aug 7, 2023

AntonyBlakey force-pushed the AntonyBlakey/to_rust_types branch from f0d1896 to 63bbea5 Compare August 7, 2023 10:01

AntonyBlakey marked this pull request as draft August 7, 2023 10:02

AntonyBlakey force-pushed the AntonyBlakey/to_rust_types branch 5 times, most recently from ca341c1 to 6671e9d Compare August 10, 2023 10:17

AntonyBlakey marked this pull request as ready for review August 10, 2023 10:18

AntonyBlakey requested review from OmarTawfik and removed request for a team August 10, 2023 10:18

AntonyBlakey force-pushed the AntonyBlakey/to_rust_types branch from 6671e9d to cbdc4c2 Compare August 10, 2023 10:48

AntonyBlakey enabled auto-merge August 10, 2023 12:16

Xanewok reviewed Aug 10, 2023

View reviewed changes

crates/codegen/grammar/src/grammar.rs Outdated Show resolved Hide resolved