Add --test and --bench options to build command

[snoyberg]

Motivation: you could write something like the following:

stack build --test --bench --haddock --copy-bins

This just makes the commands far more composable than they are right now. We should not remove or deprecate stack test and stack bench IMO: they are useful shortcuts for common cases.

[borsboom]

What do you think about an --exec option to execute a command after the build (particularly useful in conjunction with --file-watch)? It's a slightly different case than the others because stack exec isn't one of the build commands (unlike stack test etc), and currently uses the execve system call, so the stack build --exec option would probably have to follow a different code path than stack exec.

[snoyberg]

I'm in favor of it, though we will probably need to bikeshed the name to
death.

On Wed, Jul 29, 2015, 8:51 AM Emanuel Borsboom notifications@github.com
wrote:

What do you think about an --exec option to execute a command after the
build (particularly useful in conjunction with --file-watch)? It's a
slightly different case than the others because stack exec isn't one of
the build commands (unlike stack test etc), and currently uses the execve
system call, so the stack build --exec option would probably have to
follow a different code path than stack exec.

—
Reply to this email directly or view it on GitHub
#651 (comment)
.

[snoyberg]

I'd like to write up clearly my plans for overhauling how targets, "wanted"s,
and components will be handled. The goals of this overhaul are to make
tests/benchmarks more modular, make error reporting of bad arguments more
reliable, and make it possible to do things like running a single test suite. This procedure should address the following issues:

• --only-dependencies instead of --only-snapshot  #387
• Add --test and --bench options to build command #651
• Build doesn't happen on implicit global config (with no .yaml file present) #687

1. stack test is a synonym for stack build --test. Similarly, stack bench is a synonym for stack build --bench. At this point: --copy-bins, --haddock, --test, and --bench are composable with each other.
2. When running stack build, you provide a list of targets. Targets can be in one of the following formats:
   • package[:comptype]:compname, e.g. bar:exe:bar-exe1. This will specify as a target the given component of a local package (fails for any non-local package). If the package name is not available locally, this will be an error. comptype can be lib, exe, test, and bench. When using lib, :compname must not be used.
   • package:compname, e.g. bar:bar-exe1. Same as above, but determines the comptype from the compname. This will fail if the same compname is used by two different comptypes in a package (e.g., both a executable and test suite named foobar).
   • :compname, e.g. :bar-exe1. Same as above, but also determines the package name. Will fail for any ambiguity.
   • package, e.g., foo. Fails for non-local package. For a local package, adds all of the library and executables components available in the given package. If --test is passed in, all test suites are also added. If --bench is passed in, all benchmarks are also added.
   • directory, e.g., ./subdir/bar. Finds all of the locals that are in that directory a subdirectory therefore, gets their package names, and then passes each of those as a package
   • package-identifier, e.g., foo-1.2.3: pull that exact version from the package index.
     • If a package by that name is in the locals:
       • Same version: same as passing in foo as a target
       • Different version: results in an error
     • If that package is in extra-deps:
       • Same version: same as passing in foo as a target
       • Different version: results in an error
     • If that package is in the snapshot:
       • Same version: same as passing in foo as a target
       • Different version: implicitly adds foo-1.2.3 to extra-deps and then the same as passing in foo as a target
3. If no targets are specified on the command line, same as passing in the package names of all locals.
4. After parsing input, check for any duplicate package references on the command line, e.g. stack build foo foo-1.2.3. If two different arguments refer to the same package name, errors out.
5. If no targets exist after this parsing, errors out.
6. Take all of these targets (a.k.a. wanteds) and track down their dependencies. For each package depended on, mark its library and all executable components as dependencies. Only mark targets as dependencies if they are depended on by other dependencies, not other targets, so that --only-dependencies has the correct behavior.
7. If --only-dependencies is specified, ignore any packages/components that are not marked as being a dependency.
8. Construct a list of actions to perform with dependencies between each, and execute them in parallel.
9. When building test suite and benchmark components, also run them unless --no-run-tests or --no-run-benchmarks (respectively) is passed in.
10. If any builds fail or test suites fail, error out.
11. If --copy-bins is on: copy all generated executables to the local bin path
12. Run any --exec foo commands

Note: above doesn't explicitly take --haddock into account, assuming it will just follow from the existing codebase.

Pinging @borsboom @chrisdone

[chrisdone]

Sounds good to me. I think that perhaps the stack ghci and stack ide can piggy back onto this target parsing, first for ensuring it builds and second for choosing which targets and components to load.

If --only-dependencies is specified, ignore any packages/components that are not marked as being a dependency.

Presumably this will take into account when you do stack build a b and a depends on b, that it should probably build just the dependencies of a and b.

package:comptype[:compname], e.g. bar:exe:bar-exe1. This will specify as a target the given component of a local package (fails for any non-local package). If the package name is not available locally, this will be an error. comptype can be lib, exe, test, and bench. When using lib, :compname must not be used.

As a would-be-nice enhancement for the command-line implementation of this, we can possibly provide completions via the optparse API. That'd make it easier to write stack foo TAB and get stack foo: with lib, exe, bench listed then stack foo:exe: TAB etc.

[snoyberg]

Presumably this will take into account when you do stack build a b and a depends on b, that it should probably build just the dependencies of a and b.

Good point, and not reflected above. The heuristic above should be "mark something as a dependency if it's a dependency of a dependency, or if it's a dependency of a target and not a target itself." EDIT I updated the comment above to include this.

Providing autocompletion would be nice, but I don't intend to include it in the initial implementation.

[snoyberg]

I've opened up PR #753 with these changes, closing this issue so we can discuss on the PR.

[WillSewell]

When I run stack test mypackage:mytestssuite1 only mytestsuite1 is run, and no other tests suites are. But when I ruck stack bench mypackage:mybenchmarks1, all other benchmarks are also built and run. Am I misunderstanding something?

Stack version 0.1.6.0

[rvion]

@WillSewell Could you open a new issue a reference this one ?

[mgsloan]

@WillSewell Also, please try with version 0.1.8.0, since a lot has changed since 0.1.6.0