pest_generator 2.5.6 does not compile

[J-Loudet]

Describe the bug

We depend on json5 in our project which, in turn, depends on pest.
A fresh cargo build, i.e. after a cargo clean, fails because of a non-exhaustive pattern on pest_meta::optimizer::OptimizedExpr::NodeTag(_, _).

To Reproduce

1. cd /tmp && git clone https://github.com/eclipse-zenoh/zenoh-flow.git --depth 1
2. cd zenoh-flow && cargo build

Expected behavior

We would expect pest_generator to compile.

Additional context

The json5 crate includes pest as following:

pest = "2.0"

The full error is as following:

error[E0004]: non-exhaustive patterns: `pest_meta::optimizer::OptimizedExpr::NodeTag(_, _)` not covered
   --> /Users/julien/.cargo/registry/src/github.com-1ecc6299db9ec823/pest_generator-2.5.6/src/generator.rs:378:11
    |
378 |     match expr {
    |           ^^^^ pattern `pest_meta::optimizer::OptimizedExpr::NodeTag(_, _)` not covered
    |
note: `pest_meta::optimizer::OptimizedExpr` defined here
   --> /Users/julien/.cargo/registry/src/github.com-1ecc6299db9ec823/pest_meta-2.6.0/src/optimizer/mod.rs:142:5
    |
114 | pub enum OptimizedExpr {
    | ----------------------
...
142 |     NodeTag(Box<OptimizedExpr>, String),
    |     ^^^^^^^ not covered
    = note: the matched value is of type `pest_meta::optimizer::OptimizedExpr`
help: ensure that all possible cases are being handled by adding a match arm with a wildcard pattern or an explicit pattern as shown
    |
519 ~         }
520 +         pest_meta::optimizer::OptimizedExpr::NodeTag(_, _) => todo!()
    |

error[E0004]: non-exhaustive patterns: `pest_meta::optimizer::OptimizedExpr::NodeTag(_, _)` not covered
   --> /Users/julien/.cargo/registry/src/github.com-1ecc6299db9ec823/pest_generator-2.5.6/src/generator.rs:524:11
    |
524 |     match expr {
    |           ^^^^ pattern `pest_meta::optimizer::OptimizedExpr::NodeTag(_, _)` not covered
    |
note: `pest_meta::optimizer::OptimizedExpr` defined here
   --> /Users/julien/.cargo/registry/src/github.com-1ecc6299db9ec823/pest_meta-2.6.0/src/optimizer/mod.rs:142:5
    |
114 | pub enum OptimizedExpr {
    | ----------------------
...
142 |     NodeTag(Box<OptimizedExpr>, String),
    |     ^^^^^^^ not covered
    = note: the matched value is of type `pest_meta::optimizer::OptimizedExpr`
help: ensure that all possible cases are being handled by adding a match arm with a wildcard pattern or an explicit pattern as shown
    |
651 ~         }
652 +         pest_meta::optimizer::OptimizedExpr::NodeTag(_, _) => todo!()
    |

[J-Loudet]

Maybe the issue stems from Cargo and the way it resolves dependencies: somehow pest was bumped to 2.6.0 but pest_generator refuses (?) to move past 2.5.6. We have no idea why though.

[tomtau]

@J-Loudet I tried to force pest_generator = { version = "=2.6.0" } in one of the crates, and got this error:

error: failed to select a version for `quote`.
    ... required by package `syn v2.0.0`
    ... which satisfies dependency `syn = "^2.0"` of package `pest_generator v2.6.0`
    ... which satisfies dependency `pest_generator = "^2.6.0"` of package `zenoh-flow v0.5.0-dev (/private/tmp/zenoh-flow/zenoh-flow)`
    ... which satisfies path dependency `zenoh-flow` (locked to 0.5.0-dev) of package `cargo-zenoh-flow v0.5.0-dev (/private/tmp/zenoh-flow/cargo-zenoh-flow)`
versions that meet the requirements `^1.0.25` are: 1.0.26

all possible versions conflict with previously selected packages.

  previously selected package `quote v1.0.23`
    ... which satisfies dependency `quote = "^1.0.18"` of package `darling_core v0.14.4`
    ... which satisfies dependency `darling_core = "=0.14.4"` of package `darling v0.14.4`
    ... which satisfies dependency `darling = "^0.14"` of package `zenoh-flow-derive v0.5.0-dev (/private/tmp/zenoh-flow/zenoh-flow-derive)`
    ... which satisfies path dependency `zenoh-flow-derive` (locked to 0.5.0-dev) of package `zenoh-flow v0.5.0-dev (/private/tmp/zenoh-flow/zenoh-flow)`
    ... which satisfies path dependency `zenoh-flow` (locked to 0.5.0-dev) of package `cargo-zenoh-flow v0.5.0-dev (/private/tmp/zenoh-flow/cargo-zenoh-flow)`

failed to select a version for `quote` which could resolve this conflict

So, that's a bit obscure, but I guess that's the culprit why Cargo mixes up 2.5.6 and 2.6.0... Anyway, @J-Loudet you can try forcing 2.5.6 for the moment: pest = { version = "=2.5.6" } that seems to compile fine, but perhaps tweaking the other dependencies can help Cargo to move all pest dependencies to 2.6.0

[J-Loudet]

@tomtau thanks for the pointers, this seems indeed the way to go! 👍

I am closing this issue as it is not directly related to pest.

[liukun4515]

@J-Loudet will it be resolved?

I also meet this issue.

[liukun4515]

From this, the pest_generator will use the latest version of pest_meta. But the pest_meta has break changes api for thepest_generator.

[J-Loudet]

Hi @liukun4515, we ended up doing what @tomtau suggested: we tweaked our internal dependencies to "help" Cargo resolve compatible versions of pest, pest_meta and pest_generator.

From our understanding, the root problem is that Cargo assumes that packages are following SemVer which does not seem to be what pest did: version 2.6 introduces non backward compatible changes with 2.5.6.
Hence, because Cargo assumes packages are following SemVer, it assumes it should not a problem to have 2.6 and 2.5.6 mixed and it happily does so.

---

As a side note, I am not part of pest so there is nothing I can personally do.

[tomtau]

Hi @liukun4515, we put a warning in the release notes: https://github.com/pest-parser/pest/releases/tag/v2.6.0
There was a similar semver-breaking change in 2.5 and noone reported any issue, so we assumed it may be ok, but if this causes a lot of trouble to people (and tweaking internal dependencies does not work), we can try to discuss and see what to do... anyway, ultimately, we'll want to at least have one more semver violation to bump MSRV and put non_exhaustive annotations on those internal structures, so that there are no more semver violations from non-breaking grammar changes.

[turboMaCk]

Does anyone one know about a way to pinpoint transitive dependency (on pest) if I want to use 3rd party library which fails to compile due to this issue? I was trying it use [path.crates-io] for this but it seems to make no difference. I guess this issue must have broken a lot of downstream projects.

[turboMaCk]

Would it be possible to re-release version without braking change as a latest patch for 2.5 so that cargo resolves to the version without the breaking change witch which downstream dependencies could compile? It seems to me that pinning the transitive dependencies is next to impossible with cargo.

[tomtau]

@turboMaCk can you try adding pest = { version = "=2.5.6" } (or potentially tweaking other crates, such as pest_meta and pest_generator, that may be mismatched to 2.6.0) to your dependencies to pin the version? Releasing a new version may need a bit of work, because reverting the new grammar feature isn't ideal... we can perhaps find a middle ground with some feature-guarding.
One more alternative may be to yank 2.6.0 and publish it as 2.6.0-something (I think Cargo won't be resolving to versions with dashes / "pre-releases", so it'll stay at the latest one without dashes). But yanking is a bit disruptive as well.

[turboMaCk]

Thanks @tomtau this is exactly what I was looking for but did not know how to do.

[jlyonsmith]

@tomtau Forcing the pest version to 2.5.6 fixes the issue for json5, but it would be really great if future breaking versions of pest followed SemVer conventions and bumped the major version. Pest is the second most popular parsing solution for Rust, so breaking changes affect the entire Rust ecosystem.

[tomtau]

@jlyonsmith yes, I plan to find a middle-ground between allowing new features and not breaking semver in these Cargo version resolution mixup situations (2.5 also had semver breaking changes, but they didn't affect interdependencies as much as 2.6)

[tomtau]

@jlyonsmith @turboMaCk @J-Loudet @liukun4515 I'm exploring this solution: #871 but it seems there's no way to test it without publishing the crate.
So I guess after publishing the v2.7.0 version with that patch, I'll try if the original issue @J-Loudet posted can be resolved without forcing v2.5.6. If it works, I'll yank v2.6.x crates.

[tomtau]

@jlyonsmith @turboMaCk @J-Loudet @liukun4515 2.7.0 was published and it seems to work ok when mixed with the pest_generator 2.5.6 under default features, so I yanked 2.6.X crates. You can try if it works for you now when you remove pest = { version = "=2.5.6" } and perhaps cargo update?

[turboMaCk]

Thanks @tomtau I can confirm this works as expected in my project.

[jlyonsmith]

Sorry @tomtau I got distracted. I can confirm that 2.7.0 works for my project also. Thanks for the fix.