Allow tests to be updated automatically

[ecstatic-morse]

Uses the expect-test crate (thanks for the recommendation @lnicola!) to implement snapshot testing. expect-test is a bit more flexible than insta, so it seems like a good fit. This gives nicer diffs as well, since we don't strip whitespace prior to the comparison.

Setting UPDATE_EXPECT=1 before running cargo test will update all tests automatically.

Blockers

• Omit empty lists from Solution's Display impl.
  • chalk currently compares prefixes to make things less noisy, but that seems error prone anyways.
  • Harder than it seems at first, since you need to unintern to check whether a list is empty.
• Handle recursive/SLG comparisons better.
  • Many tests use yields, which compares the results of the two solvers against a known output. Unfortunately, the test macro will duplicate the expect invocation, so if neither output matches the expected one, expect-test will overwrite it twice, almost assuredly corrupting the test file.
  • There's many ways to fix this. I've hacked around it here by adding a method to Expect (which is not yet upstream). Most robust is to collect the various engine results first and only compare them against the output if they agree. This is implemented now.
• "]]" inside an expect breaks updating rust-analyzer/expect-test#20

Nice to haves

• Don't use raw strings with a hash unconditionally when blessing (Use the minimal number of escapes when updating tests rust-analyzer/expect-test#28).
• Shorter alias for expect![[? It's currently hard-coded into the expect-test crate (Allow users to rebind expect! rust-analyzer/expect-test#26).
• Remove newline escapes from existing tests.

[ecstatic-morse]

@jackh726 This is ready to review now (commit-by-commit), but no rush. Enjoy the holidays. UPDATE_EXPECT with the current version of expect-test will fail on a few tests (see rust-analyzer/expect-test#20), so I did the last commit with my fork.

[jackh726]

So, what's the tangible benefit here? Blessable tests? It doesn't seem like it really removes any testing infra.

There are definitely some niceties that can be split out regardless of whether we land this - notably the test output formatting changes.

Any way it's possible to have the test macro generate the expect!?

[ecstatic-morse]

So, what's the tangible benefit here? Blessable tests?

Blessable tests. I'd encourage you to ask around before pooh-poohing them. rust-analyzer and polonius.next both use them. Besides making writing new tests much easier, they also make changing Debug output feasible, since whoever does that work doesn't have to go back and update all the tests by hand.

Any way it's possible to have the test macro generate the expect!?

I don't think so. expect calls file/line internally to record the location of the expect macro invocation. That's how snapshot testing frameworks know which part of the source code to update when an assertion fails. You could vendor expect-test and modify so it that it can bless entire test!s, but that's not worth it IMO. Better to just improve expect-test to support invocations like x!["Expected thing"] as I mentioned in my PR summary.

[matklad]

cc @nikomatsakis, that's relevant to the "what snapshot testing we should use for salsa" question

[matklad]

Besides making writing new tests much easier, they also make changing Debug output feasible, since whoever does that work doesn't have to go back and update all the tests by hand.

Yeah, I found expect-style tests to be a force multiplier. Another big benefit is that you can refactor the testing framework itself. Ie, if the shape of the thing you test changes, you don't have to do O(N) work for updating all the tests. And this in turn encourages evolving testing strategies. To give a concrete example, suppose that some day you start caring not only about the answer, but also about the amount of "steps" it took chalk to arrive at the answer. With blessing, you can trivial say "ok, let's just add the amount of steps to each test".

Looking at this PR, it seems that you already have a dedicate testing macro:

    test! {
        program {
            #[lang(copy)]
            trait Copy { }
            struct Foo { }
            impl Copy for Foo { }
        }
        goal {
            forall<const N> {
                [Foo; N]: Copy
            }
        } yields {
            "Unique"
        }
    }

I don't have a full context here, but I'd probably do a different thing that this PR does. Instead, I'd do either of the two:

• instead of depending on expect-test, just add blessing functionality to the test! macro itself. That is, copy-paste the "runtime" parts of expect-test to chalk, and replace "look for expect![[" logic with "look for yields {}". expect-test is a simple thing, I wouldn't worry about copy-pasting creating some extra maintenance burden for chalk .
• changed the API to not put calls to expect! into the macro:

program! {
    #[lang(copy)]
    trait Copy { }
    struct Foo { }
    impl Copy for Foo { }
}.check(
  goal! {
    forall<const N> {
        [Foo; N]: Copy
    }
  },
  expect![["Unique"]]
);

That is, the approach of this PR of putting expect! calls directly into the test! feels a bit like path-dependent combination of two ideas.

What we do in rust-analyzer is that we use raw-strings for both input and output of the tests:

check(r#"
  fn some_rust_code() {
  }
"#,
  expect![["some debug output"]]
);

[ecstatic-morse]

Yeah, I found expect-style tests to be a force multiplier.

To be concrete, I'm interested in simplifying lifetime constraints arising from subtyping relationships on higher-ranked types for use in Polonius, as well as (in the longer-term) extending Rust's type system to support constraints on those types (e.g. for<'a, 'b> where 'a: 'b). I'm not sure exactly what this will require yet 😥, but it will likely include the kind of "broad-but-shallow" changes that are made much easier with expect-style tests.

instead of depending on expect-test, just add blessing functionality to the test! macro itself. That is, copy-paste the "runtime" parts of expect-test to chalk, and replace "look for expect![[" logic with "look for yields {}". expect-test is a simple thing, I wouldn't worry about copy-pasting creating some extra maintenance burden for chalk.

All this to avoid a macro invocation? In addition to vendoring expect-test, "look for yields {}" would require a small parser for the test! DSL. yields[$e:expr] is currently valid, and matching delimited text with regular expressions in publicly available code makes me a bit embarrassed, frankly.

changed the API to not put calls to expect! into the macro:

I probably would have implemented test as a builder like matklad shows, though obviously we have more information about the final interface than whoever did the original version. Once again, though, this is a fairly major change compared to what I've done here, and this PR is "one step closer" to matklad's version in the sense that expect appears explicitly in both.

[jackh726]

I'm convinced enough by the arguments here. But, I'd at least like to run this by @nikomatsakis first (though, I doubt he'll have strong feelings one way or the other).

[ecstatic-morse]

assert_same strips whitespace, uses prefix matching, and has a different diff output than Expect::assert_eq, so the error that occurs when two solvers don't match looks quite different from the one that occurs when the expected output is wrong. Expect::assert_eq does not work without something to bless, though it could be made to with rust-analyzer/expect-test#22. If this is merged, I expect that uses of assert_same will slowly be removed in favor of Expect, so I don't think we need to resolve this right now. Others may disagree, however.

[jackh726]

@nikomatsakis and I talked about this and agreed that this is probably a good idea :)

I just need to devote some time to do a final quick review before I r+ this (so please ping me if I forget and this sits)

[nikomatsakis]

Quick comment: can we use one of the existing libraries for this? (I didn't even read the PR, sorry...)

e.g., insta or https://github.com/rust-analyzer/expect-test

[lnicola]

It's using expect-test.

[nikomatsakis]

well then I'll just go away :)

[jackh726]

@bors r+

[bors]

📌 Commit 1627b6e has been approved by jackh726

[bors]

⌛ Testing commit 1627b6e with merge de7fcd0...

[bors]

☀️ Test successful - checks-actions
Approved by: jackh726
Pushing de7fcd0 to master...