Access the executable build by cabal build (with a custom Setup.hs) from cabal test

[andreasabel]

Question: What is the canonical way to write a test-suite that calls the executable as process, e.g. via system?

While writing up the question, I discovered an answer that I want to share here, since I could not discover it so far by googling (despite many tries). This answer could possibly find its way into the documentation, e.g. near https://cabal.readthedocs.io/en/3.4/cabal-package.html#example-package-using-exitcode-stdio-1-0-interface.

Here is a minimal test-suite code Test.hs that tries to call the executable hello-world defined in the same cabal project:

import System.Exit     ( exitWith  )
import System.Process  ( system    )

main :: IO ()
main = exitWith =<< system ("hello-world")

We can run hello-world via cabal run -- hello-world.
However, without special provision, cabal test fails:

Build profile: -w ghc-9.0.1 -O1
In order, the following will be built (use -v for more details):
 - hello-1.0.0.3 (test:hello-test) 
...
Running 1 test suites...
Test suite hello-test: RUNNING...
/bin/sh: hello-world: command not found

Test suite hello-test: FAIL

The solution is to add the executable to the test-suite stanza, via field build-tool-depends:

name:                 hello-world
version:              0.0.1
cabal-version:        >= 1.8
build-type:           Simple

executable hello-world
  hs-source-dirs:     .
  main-is:            Main.hs
  build-depends:      base >= 4.2 && < 5

test-suite hello-test
  type:               exitcode-stdio-1.0
  hs-source-dirs:     .
  main-is:            Test.hs
  build-depends:      base, process
  build-tool-depends: hello-world:hello-world

For completeness, here is Main.hs:

main = putStrLn "Hello, World!"

Now cabal test works!

Build profile: -w ghc-9.0.1 -O1
In order, the following will be built (use -v for more details):
 - hello-world-0.0.1 (exe:hello-world) (first run)
 - hello-world-0.0.1 (test:hello-test) (first run)
...
Building executable 'hello-world' for hello-world-0.0.1..
...
Building test suite 'hello-test' for hello-world-0.0.1..
...
Running 1 test suites...
Test suite hello-test: RUNNING...
Test suite hello-test: PASS

Context: agda/agda#5302 (comment)

Please tag this issue with cabal-install: cmd/test and type: user question.

[fgaz]

There are three ways of doing this

1. if possible, do not call the executable via system, and instead expose the main function (or similar) in a library, so it can be called directly from the Haskell test code
2. like you wrote, declare the dependency with build-tool-depends. this is documented here
3. have a --with-$program option in the test suite that can be used to set $program's path by external means (script, make target, manually...) when running the tests
   • for example cabal-testsuite works this way: there is a --with-cabal option that is usually set by running cabal run cabal-testsuite:cabal-tests -- --with-cabal $(cabal list-bin cabal). if --with-cabal isn't set, the cli tests are skipped. this does mean that an user won't be able to straightforwardly run all tests, but that usually isn't a problem since those are mostly useful for developers (especially the kind of extensive testing done in cabal-testsuite)

The canonical way of testing an executable via system is 2, at least until we get something like #5411. This was discussed in #5418, clarified in #5561, and further brought up in #7505.

---

I'm tagging with Cabal: cmd/test since this should be independent from the tool

[fgaz]

I was wrong about build-tool-depends: after some digging I found that it is the current canonical solution, even though (as #5411 points out) it isn't semantically perfect. I heavily edited my previous comment to reflect this.

PRs to further improve the docs (eg. link to https://cabal.readthedocs.io/en/3.4/cabal-package.html#pkg-field-build-tool-depends from https://cabal.readthedocs.io/en/3.4/cabal-package.html#example-package-using-exitcode-stdio-1-0-interface) are welcome!

[andreasabel]

Unfortunately, I discovered that this only works with build-type: Simple.
If I switch to build-type: Custom with a generic stanza

custom-setup
  setup-depends:  base, Cabal

and a standard Setup.hs

import Distribution.Simple
main = defaultMain

then cabal test does not work anymore:

$ cabal test
Build profile: -w ghc-9.0.1 -O1
In order, the following will be built (use -v for more details):
 - hello-world-0.0.1 (ephemeral targets)
Preprocessing executable 'hello-world' for hello-world-0.0.1..
Building executable 'hello-world' for hello-world-0.0.1..
Preprocessing test suite 'hello-test' for hello-world-0.0.1..
Building test suite 'hello-test' for hello-world-0.0.1..
Running 1 test suites...
Test suite hello-test: RUNNING...
/bin/sh: hello-world: command not found

Test suite hello-test: FAIL
...
cabal: Tests failed for hello-world-0.0.1.

Likewise, which cabal run hello-world (executable) still works, cabal run hello-test (test-suite) does not find hello-world.

@fgaz Is this a bug or is there a trick to get it to work?

[andreasabel]

In communication with @fgaz, we can demonstrate that this is problem is present in Setup.hs (even with build-type: Simple):

$ runhaskell Setup.hs configure --enable-tests
Configuring hello-world-0.0.1...

$ runhaskell Setup.hs build
Preprocessing executable 'hello-world' for hello-world-0.0.1..
Building executable 'hello-world' for hello-world-0.0.1..
Preprocessing test suite 'hello-test' for hello-world-0.0.1..
Building test suite 'hello-test' for hello-world-0.0.1..
[1 of 1] Compiling Main             ( Test.hs, dist/build/hello-test/hello-test-tmp/Main.o )
Linking dist/build/hello-test/hello-test ...

$ runhaskell Setup.hs test
Running 1 test suites...
Test suite hello-test: RUNNING...
/bin/sh: hello-world: command not found

Test suite hello-test: FAIL
Test suite logged to: dist/test/hello-world-0.0.1-hello-test.log
0 of 1 test suites (0 of 1 test cases) passed.

UPDATE: According to an experiment, build-tool-depends in the test-suite is completely ignored by both runhaskell Setup.hs build and runhaskell Setup.hs test. Meaning the program mentioned there is neither built nor put in the path.

[phadej]

This looks like an instance of run-tool-depends #5411

FWIW, I don't think Setup.hs interface is fixable. IMO the caller has (somehow) specify the environment (what if run-time needed executable doesn't come from the same package?). It would be conceptually simpler to emphasise building only individual components of package (instead of everything). With e.g. test or benchmark dependency cycles that would be a necessity anyway.

[fgaz]

I found that this specific problem only occurs between components in a single monolithic package. Separating the test and the exe in two packages works independently from the build-type of either. Which kinda explains why build-type: Simple works in the single-package+cabal-install case: the granularity there is the component (component-based builds are only enabled on Simple)

[fgaz]

Could it be this? why are intra-package dependencies not collected?

cabal/Cabal/src/Distribution/Simple/Configure.hs

Lines 553 to 563 in 7d4ce47

	-- First, we collect any tool dep that we know is external. This is,
	-- in practice:
	--
	-- 1. `build-tools` entries on the whitelist
	--
	-- 2. `build-tool-depends` that aren't from the current package.
	let externBuildToolDeps =
	[ LegacyExeDependency (unUnqualComponentName eName) versionRange
	| buildTool@(ExeDependency _ eName versionRange)
	<- getAllToolDependencies pkg_descr bi
	, not $ isInternal pkg_descr buildTool ]

[fgaz]

here is when it was originally done, but the commit message does not explain why and it's from ten years ago. @dcoutts?

70196d5#diff-5df11a3bbf4660041725cd6c42025edefbf6e601774e4214e333a08a947048d7L432-R444

[phadej]

Might be related to happy needing happy to build itself. (That was the case back then).

EDIT: IMHO such special cases don't need to be supported. That's complexity which everyone has to pay, and happy bootstrapping was improved so only happy maintainers & developers have to pay for its complexity.

[jneira]

myabe it should be closed (or the title changed) as you can access the executable inthegeneral case

[andreasabel]

I changed the title to be more precise.
AFAIK, the issue is not fixed. @fgaz located the source of the problem, but there isn't a PR yet addressing it.
A reproducer is in #7577 (comment).