`cabal test` sees the executable, but `cabal run` does not. #8391

kindaro · 2022-08-17T15:17:53Z

There is a package called fourmolu that is going to be my example here. The relevant issue is fourmolu/fourmolu#231. The appropriate commit is 12af899611cea08c5d9a1f547d432edfbce38be4 — the current main.

See for yourself:

Remove all instances of fourmolu on the path. You should have this:
```
% where fourmolu
fourmolu not found
```

See that the tests pass:

% cabal test
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
 - fourmolu-0.8.1.0 (test:fixity-tests) (ephemeral targets)
 - fourmolu-0.8.1.0 (test:region-tests) (ephemeral targets)
 - fourmolu-0.8.1.0 (test:tests) (ephemeral targets)
Preprocessing test suite 'fixity-tests' for fourmolu-0.8.1.0..
Building test suite 'fixity-tests' for fourmolu-0.8.1.0..
Preprocessing test suite 'region-tests' for fourmolu-0.8.1.0..
Building test suite 'region-tests' for fourmolu-0.8.1.0..
Preprocessing test suite 'tests' for fourmolu-0.8.1.0..
Building test suite 'tests' for fourmolu-0.8.1.0..
Running 1 test suites...
Running 1 test suites...
Test suite region-tests: RUNNING...
Test suite fixity-tests: RUNNING...
Test suite region-tests: PASS
Test suite logged to:
/home/kindaro/projects/haskell/fourmolu/dist-newstyle/build/x86_64-linux/ghc-8.10.7/fourmolu-0.8.1.0/t/region-tests/test/fourmolu-0.8.1.0-region-tests.log
1 of 1 test suites (1 of 1 test cases) passed.
Test suite fixity-tests: PASS
Test suite logged to:
/home/kindaro/projects/haskell/fourmolu/dist-newstyle/build/x86_64-linux/ghc-8.10.7/fourmolu-0.8.1.0/t/fixity-tests/test/fourmolu-0.8.1.0-fixity-tests.log
1 of 1 test suites (1 of 1 test cases) passed.
Running 1 test suites...
Test suite tests: RUNNING...
Test suite tests: PASS
Test suite logged to:
/home/kindaro/projects/haskell/fourmolu/dist-newstyle/build/x86_64-linux/ghc-8.10.7/fourmolu-0.8.1.0/t/tests/test/fourmolu-0.8.1.0-tests.log
1 of 1 test suites (1 of 1 test cases) passed.

See that the tests fail if run with cabal run:

% grep 'test-suite' fourmolu.cabal | cut -d ' ' -f 2 | while read name; do cabal run test:"$name"; done
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
 - fourmolu-0.8.1.0 (test:tests) (additional components to build)
Preprocessing test suite 'tests' for fourmolu-0.8.1.0..
Building test suite 'tests' for fourmolu-0.8.1.0..
…
Finished in 4.1124 seconds
985 examples, 1 failure, 2 pending
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
 - fourmolu-0.8.1.0 (test:region-tests) (additional components to build)
Preprocessing test suite 'region-tests' for fourmolu-0.8.1.0..
Building test suite 'region-tests' for fourmolu-0.8.1.0..
…
Finished in 0.0011 seconds
8 examples, 1 failure, 7 pending
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
 - fourmolu-0.8.1.0 (test:fixity-tests) (additional components to build)
Preprocessing test suite 'fixity-tests' for fourmolu-0.8.1.0..
Building test suite 'fixity-tests' for fourmolu-0.8.1.0..
…
Finished in 0.0015 seconds
6 examples, 1 failure, 5 pending

See that the tests pass if run with cabal run after cabal install:

% cabal install
…
Symlinking 'fourmolu' to '/home/kindaro/.cabal/bin/fourmolu'
% grep 'test-suite' fourmolu.cabal | cut -d ' ' -f 2 | while read name; do cabal run test:"$name"; done
Resolving dependencies...
Up to date
…
Finished in 6.4715 seconds
985 examples, 0 failures
Up to date
…
Finished in 0.3642 seconds
8 examples, 0 failures
Up to date
…
Finished in 0.7731 seconds
6 examples, 0 failures

Expected:

Either the tests fail, or the tests pass. It does not matter which way I run them. In particular, Cabal makes the same executables available in the same way: either the fourmolu executable is made available to both cabal run and cabal test or to neither.

Actual:

The executable fourmolu is made available to the test programs when they are run with cabal test but not with cabal run.

System information:

% uname --all
Linux kindrom-silverheart 5.18.13-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 22 Jul 2022 13:05:04 +0000 x86_64 GNU/Linux
% cabal --version
cabal-install version 3.8.1.0
compiled using version 3.8.1.0 of the Cabal library 
% ghc --version
The Glorious Glasgow Haskell Compilation System, version 8.10.7

The text was updated successfully, but these errors were encountered:

Mikolaj · 2022-08-17T15:34:58Z

So is the underlying bug that build-tool-depends: fourmolu:fourmolu is not honoured by cabal run test:foo? What's the semantics of build-tool-depends? That the tool should only be made available for building the code, or also for running the artefact?

Mikolaj · 2022-08-17T15:52:50Z

Or, from another angle, is cabal run foo supposed to be equivalent to cabal build foo; path_to_foo/foo or not? How about cabal test foo?

georgefst · 2022-08-17T16:34:33Z

Or, from another angle, is cabal run foo supposed to be equivalent to cabal build foo; path_to_foo/foo or not?

This is an excellent point. To me, it would be surprising if it wasn't.

What's the semantics of build-tool-depends?

I'd always assumed from the name that it's just for build tools. But the user guide states that it also makes executables available at run time. And does not make any qualifications about cabal run vs. cabal test, or test-suite stanzas vs. executable.

It's too late to change the behaviour of adding to PATH for cabal test without very good reason, since people rely on the behaviour. Besides, some new test-tool-depends or similar mechanism would need to be introduced to cover the old use cases.

Perhaps the best fix here is to:

Fix the user's guide to make it clear that build-tool-depends makes executables available on PATH for cabal test (and cabal bench? maybe even cabal repl?), but not cabal run.
Improve the cabal test CLI, so that all the reasons for currently preferring to run cabal run test:foo no longer apply. --test-options are split wrong #8090 is one. Coloured output is another. There may be more.

gbaz · 2022-08-17T16:41:56Z

Why not just make it available in the PATH in cabal run as well? That seems most in keeping with the spirit of the field.

georgefst · 2022-08-17T16:45:09Z

Why not just make it available in the PATH in cabal run as well? That seems most in keeping with the spirit of the field.

I'm not totally against it. But as @Mikolaj alluded to, it seems desirable to have cabal run target be equivalent to cabal build target && $(cabal list-bin target). Unless there's some reason why this is already not strictly the case?

Mikolaj · 2022-08-17T16:51:19Z

IIRC cabal run can also be used for running interpreted Haskell scripts (as opposed to binaries), so perhaps it sets up the environments specially for them. If so, we are not complicating the semantics so much by unifying how the environments are set up for exes and scripts. Not sure how test-tool-depends works with scripts, though, nor how it should work.

georgefst · 2022-08-17T17:55:20Z

IIRC cabal run can also be used for running interpreted Haskell scripts (as opposed to binaries),

I thought those were still compiled (even after #7851)? I've always wanted them to (at least have the option to) go through the interpreter, but I don't remember seeing anything about it in release notes.

Mikolaj · 2022-08-17T18:04:35Z

I thought those were still compiled

You may be right. There were recent great contributions preventing them from being compiled anew each time they are executed, etc., etc., so I thought they may have been compiled to bytecode or interpreted beforehand. I'm not sure about any of that, but what I'm interested in in this context is whether they are executed in a somehow modified environment or not. There's also cabal exec. Basically, I'm looking for a convincing story we may write in the manual, as opposed to "everything works with its own special quirks, which are too numerous to mention".

ulysses4ever · 2022-08-18T02:18:16Z

My understanding is that scripts are still compiled to native, but the build artifacts are cached now (just like with "normal" packages). GHC recently added a --run option that may look like a good alternative to use for cabal run (e.g. cabal run --byte-code), but I didn't see a feature request for it. So far, Ben only proposed (#7393) it for usage in custom setups (which no one wants to touch with a six-feet pole).

Mikolaj · 2022-08-18T09:14:55Z

My understanding is that scripts are still compiled to native, but the build artifacts are cached now (just like with "normal" packages). GHC recently added a --run option that may look like a good alternative to use for cabal run (e.g. cabal run --byte-code), but I didn't see a feature request for it. So far, Ben only proposed (#7393) it for usage in custom setups (which no one wants to touch with a six-feet pole).

Thank you. That would imply that cabal run never changes the env in which the exe is run in any way. OTOH, cabal exec seems to do something close to what people want to use cabal run for:

https://cabal.readthedocs.io/en/latest/cabal-commands.html?highlight=exec#cabal-exec

Am I reading this right? Brings exes into scope? Would cabal exec test:foo (or cabal exec run test:foo? how is it called exactly?) work properly? If so, we could leave cabal run be and not mix it with cabal exec.

fgaz · 2022-08-19T12:45:51Z

cabal run foo supposed to be equivalent to cabal build foo; path_to_foo/foo or not

it isn't equivalent. A package with data files will not run with cabal build foo; path_to_foo/foo because Paths_pkg.hs points to the cabal store, so cabal run sets an environment variable that overrides that path (and so does cabal exec, you can try running cabal exec env).

If we want to support #6919 then the variable will have to stay, otherwise we could compile directly with local paths

fgaz · 2022-08-19T12:51:15Z

Why not just make it available in the PATH in cabal run as well? That seems most in keeping with the spirit of the field.

makes sense to me. after all cabal run test is just a convenience to avoid passing --test-arguments (see #4861)

Mikolaj · 2022-08-19T14:18:28Z

Why not just make it available in the PATH in cabal run as well? That seems most in keeping with the spirit of the field.

makes sense to me. after all cabal run test is just a convenience to avoid passing --test-arguments (see #4861)

OK, I'm convinced. Could we document all this, probably most under https://cabal.readthedocs.io/en/latest/cabal-commands.html?highlight=cabal%20run#cabal-run? In particular, "cabal run sets an environment variable" and, after the addition, "makes build-tool-depends available in the PATH"?

georgefst · 2022-09-21T10:51:17Z

Related: #5411

xsebek · 2023-10-15T18:08:44Z

Hi, I would like to try fix this. 🙂

The motivation is HLS which has tests that need to run the binary in separate process. The built HLS binary needs to be on path, otherwise I would need to pass it through environment variable manually.

Looking at the code, it seems runAction could share the logic of creating environment with execAction.
Indeed this works as desired:

cabal exec $(cabal list-bin func-test) -- -p indefinite  # this puts built HLS on path

I could not find where the testAction eventually creates the environment, I assume "building tests" will run them but my head is dizzy from the ProjectOrchestration logic. 😕

kindaro mentioned this issue Aug 17, 2022

Test suite tests fails when running with cabal run. fourmolu/fourmolu#231

Open

Mikolaj added type: bug re: build-tool Concerning `build-tools` and `build-tool-depends` labels Aug 17, 2022

Mikolaj added the attention: pr-welcome label Aug 19, 2022

xsebek mentioned this issue Oct 15, 2023

Update documentation for running tests haskell/haskell-language-server#3844

Open

xsebek mentioned this issue Oct 15, 2023

Fix run command environment #9341

Closed

4 tasks

jasagredo mentioned this issue Jan 13, 2024

Fix extra-prog-path propagation in the codebase. #9527

Merged

5 tasks

mergify bot closed this as completed in #9527 Jan 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`cabal test` sees the executable, but `cabal run` does not. #8391

`cabal test` sees the executable, but `cabal run` does not. #8391

kindaro commented Aug 17, 2022

Mikolaj commented Aug 17, 2022

Mikolaj commented Aug 17, 2022

georgefst commented Aug 17, 2022

gbaz commented Aug 17, 2022

georgefst commented Aug 17, 2022

Mikolaj commented Aug 17, 2022

georgefst commented Aug 17, 2022 •

edited

Mikolaj commented Aug 17, 2022

ulysses4ever commented Aug 18, 2022 •

edited

Mikolaj commented Aug 18, 2022

fgaz commented Aug 19, 2022

fgaz commented Aug 19, 2022

Mikolaj commented Aug 19, 2022

georgefst commented Sep 21, 2022

xsebek commented Oct 15, 2023

cabal test sees the executable, but cabal run does not. #8391

cabal test sees the executable, but cabal run does not. #8391

Comments

kindaro commented Aug 17, 2022

See for yourself:

Expected:

Actual:

System information:

Mikolaj commented Aug 17, 2022

Mikolaj commented Aug 17, 2022

georgefst commented Aug 17, 2022

gbaz commented Aug 17, 2022

georgefst commented Aug 17, 2022

Mikolaj commented Aug 17, 2022

georgefst commented Aug 17, 2022 • edited

Mikolaj commented Aug 17, 2022

ulysses4ever commented Aug 18, 2022 • edited

Mikolaj commented Aug 18, 2022

fgaz commented Aug 19, 2022

fgaz commented Aug 19, 2022

Mikolaj commented Aug 19, 2022

georgefst commented Sep 21, 2022

xsebek commented Oct 15, 2023

`cabal test` sees the executable, but `cabal run` does not. #8391

`cabal test` sees the executable, but `cabal run` does not. #8391

georgefst commented Aug 17, 2022 •

edited

ulysses4ever commented Aug 18, 2022 •

edited