doctest's spec tests failing with GHC 9

[simonmichael]

There are some test failures when building with GHC 9. Eg:

# stack8.10.yaml
resolver: lts-18.0
packages:
- .

# stack.yaml
resolver: nightly-2021-07-16
packages:
- .

With ghc 8:

$ stack test doctest:test:spec --stack-yaml stack8.10.yaml
...
Finished in 82.0428 seconds
170 examples, 0 failures

doctest> Test suite spec passed

With ghc 9:

$ stack test doctest:test:spec
...
Failures:

  test/MainSpec.hs:94:7: 
  1) Main.doctest template-haskell
       template-haskell ["Foo.hs"]
       expected: Examples: 2  Tried: 2  Errors: 0  Failures: 0
        but got: Examples: 2  Tried: 2  Errors: 0  Failures: 1

  To rerun use: --match "/Main/doctest/template-haskell/"

  test/MainSpec.hs:148:7: 
  2) Main, doctest as a runner for QuickCheck properties, runs $setup before each property
       property-setup ["Foo.hs"]
       expected: Examples: 3  Tried: 3  Errors: 0  Failures: 0
        but got: Examples: 3  Tried: 3  Errors: 0  Failures: 1

  To rerun use: --match "/Main/doctest as a runner for QuickCheck properties/runs $setup before each property/"

  test/MainSpec.hs:177:7: 
  3) Main, doctest (regression tests), template-haskell-bugfix
       template-haskell-bugfix ["Main.hs"]
       expected: Examples: 2  Tried: 2  Errors: 0  Failures: 0
        but got: Examples: 2  Tried: 2  Errors: 0  Failures: 1

  To rerun use: --match "/Main/doctest (regression tests)/template-haskell-bugfix/"

  test/RunSpec.hs:108:5: 
  4) Run.doctest can deal with potentially problematic GHC options
       uncaught exception: ExitCode
       ExitFailure 1

  To rerun use: --match "/Run/doctest/can deal with potentially problematic GHC options/"

  test/RunSpec.hs:128:11: 
  5) Run.doctestWithOptions, on parse error, prints a useful error message
       expected: "\nFoo.hs:6:1: error:\n    parse error (possibly incorrect indentation or mismatched brackets)\n"
        but got: "\n\ESC[;1mFoo.hs:6:1: \ESC[;1m\ESC[31merror:\ESC[0m\ESC[0m\ESC[;1m\ESC[0m\ESC[0m\ESC[;1m\n    parse error (possibly incorrect indentation or mismatched brackets)\ESC[0m\ESC[0m\n"

  To rerun use: --match "/Run/doctestWithOptions/on parse error/prints a useful error message/"

Randomized with seed 2096325755

Finished in 90.4501 seconds
170 examples, 5 failures

[andreasabel]

I can confirm the first 4 failures.
The 5th seems to be a problem with ANSI coloring (added by stack?).

[sol]

I think the issue is that enabling -XTemplateHaskell clears the scope in some situations with recent versions of GHC. I'm looking into it.

[sol]

I think we can work around this in doctest by always enabling TemplateHaskell. We already do it if there are any prop> statements + doctest is already restricted to platforms that support TemplateHaskell (via ghci). So I hope this is safe to do.

I think it's still a bug. Specifically, it not only clears the scope but also unloads any modules that were specified on the command line. So if somebody puts :set -XTemplateHaskell into e.g. a setup chunk, he will still run into issues.

I filled an upstream bug here: https://gitlab.haskell.org/ghc/ghc/-/issues/20670

[sol]

I think #312 will work.