Include file/line/column of failed assertion in output?

[owickstrom]

Hi! This is going to sound like a very Emacs-specific problem, but please bear with me. 🙂

I'm using Hedgehog a lot through Emacs' compilation-mode (or the haskell-compilation buffer, to be specific). I was trying to add a pattern to haskell-compilation-error-regexp-alist, which would let me navigate between failed tests using the regular "previous/next error" functions in Emacs.

The problem is that Hedgehog's output is awesome for human consumption, but not so much for Emacs consumption as it doesn't support multi-line error matchers. Even if it did, I couldn't come up with a reliable way of parsing file, line, and column information.

Would it make sense to add, somewhere in the output of a failing test, a line like:

Test failed at: my/path/to/FailedTest.hs:54:4

Or, in case the test threw some exception:

Test errored at: my/path/to/FailedTest.hs:54:4

That would be much more tractable to parse in Emacs (and I guess in other editors, recalling my fun days with Vim's errorformat).

Outstanding questions that I can come up with:

• Where in the existing output, if anywhere, could this be placed?
• Do we have column information? If not, just having the line number is still good, so that you can navigate to the failing assertion's line.
• Should this be hidden by a flag/environment variable? Maybe it clutters the existing nice output.

I'd be happy to put together a PR after we've discussed options.

Thanks!

[jacobstanley]

I believe if you search for Span in https://github.com/hedgehogqa/haskell-hedgehog/blob/master/hedgehog/src/Hedgehog/Internal/Report.hs you should find the info you're looking for, it does indeed have the column as we need it to render the ^^^^^ arrows

Haha with a complex enough multiline regex I think we could do what you want already (looking for the arrows) but perhaps we could make that a little easier.

I could imagine we might be able to put it in the header line e.g:

        ┏━━ my/path/to/FailedTest.hs:54:4 ━━━

Not sure if that's better or not than a separate printout.

[owickstrom]

That would absolutely work! As long as it's fine that 54:4 in this case is not necessarily the first actual line number in the code listing, but the line where the assertion failed/threw?

[jacobstanley]

yeah both sets of information should be available, so would be up to you to pick the right one, I'm happy for it to be where the error occurred

[jacobstanley]

there is also the possibility that there are other code blocks / functions are printed, like if you annotate in one function and the error occurs in another.

for those auxiliary code blocks I would expect they wouldn't have the line:col suffix

[owickstrom]

Yeah, that sort of suggests that the line/column printout should be separate. I'm a bit weary about conflating the pretty code listings with this information.

[jacobstanley]

Yeah so I trust you'll preserve the aesthetics as best you can! I'm happy for you to experiment and come up with something you think looks nice.

[owickstrom]

Something like this could work, after the failed after 4 tests and 1 shrink line (example from https://teh.id.au/posts/2017/07/15/state-machine-testing/):

*Test.Example.Turnstile> check prop_turnstile
  ✗ <interactive> failed after 4 tests and 1 shrink,
    at test/Test/Example/Turnstile.hs:153:7.

        ┏━━ test/Test/Example/Turnstile.hs ━━━
    130 ┃ s_push_unlocked :: (Monad n, MonadIO m, MonadTest m) => Turnstile -> Command n m ModelState
          <snip>
    150 ┃         -- Postcondition: we gain admission, the turnstile gate is locked.
    151 ┃       , Ensure $ \before after Push b -> do
    152 ┃           before === TUnlocked
    153 ┃           assert b
        ┃           ^^^^^^^^

        ┏━━ test/Test/Example/Turnstile.hs ━━━
    159 ┃ prop_turnstile :: Property
    160 ┃ prop_turnstile =
    161 ┃   property $ do
    162 ┃     turnstile <- liftIO newTurnstile
    163 ┃     actions <- forAll $
    164 ┃       Gen.actions (Range.linear 1 100) initialState [
    165 ┃           s_coin turnstile
    166 ┃         , s_push_locked turnstile
    167 ┃         , s_push_unlocked turnstile
    168 ┃         ]
        ┃         │ [ Var 0 :<- Push ]
    169 ┃     executeSequential initialState actions

    This failure can be reproduced by running:
    > recheck (Size 3) (Seed 8794484450743003252 4983793967879311195) <property>

with the at ... always being on a separate line.

[jacobstanley]

Yeah that's nice, I would try some variations too and see how they look:

*Test.Example.Turnstile> check prop_turnstile
  ✗ <interactive> failed after 4 tests and 1 shrink,
    at test/Test/Example/Turnstile.hs:153:7

*Test.Example.Turnstile> check prop_turnstile
  ✗ <interactive> failed after 4 tests and 1 shrink.
    test/Test/Example/Turnstile.hs:153:7

*Test.Example.Turnstile> check prop_turnstile
  ✗ <interactive> failed at test/Test/Example/Turnstile.hs:153:7 after 4 tests and 1 shrink.

*Test.Example.Turnstile> check prop_turnstile
  ✗ <interactive> failed at test/Test/Example/Turnstile.hs:153:7
    after 4 tests and 1 shrink.

[owickstrom]

I think I like number 4 the most.