[RFC] Allow Pkg.test to pass command-line arguments to test/runtests.jl

[DilumAluthge]

Fixes #518
Fixes #981
Fixes #1241

---

In a source build of Julia, you can run specific test sets as such:

./julia test/runtests.jl bitarray math

In a binary install, you can accomplish the same thing with:

Base.runtests(["bitarray", "math"])

Base.runtests(tests) accomplishes this (see here) by passing the contents of the first positional argument test as the command line arguments to test/runtests.jl.

However, as far as I can tell, there is currently no functionality built into Pkg that allows you to pass any arguments to a package's test/runtests.jl file when running the package tests.

This pull request adds two features:

• Pass flags to the julia process that runs a package's tests. The flags are specified in the julia_flags keyword argument to Pkg.test.
• Pass positional command-line arguments to test/runtests.jl. The arguments are specified in the test_args keyword argument to Pkg.test and are available inside test/runtests.jl as ARGS[1], ARGS[2], etc.

Example usage:

# equivalent to running `julia test/runtests.jl`
Pkg.test("HelloWorld")

# equivalent to running `julia test/runtests.jl foo bar baz`
Pkg.test("HelloWorld"; test_args = ["foo", "bar", "baz"])

# equivalent to running `julia --math-mode=ieee test/runtests.jl`
Pkg.test("HelloWorld"; julia_flags = ["--math-mode=ieee"])

# equivalent to running `julia --math-mode=ieee test/runtests.jl foo bar baz`
Pkg.test("HelloWorld"; test_args = ["foo", "bar", "baz"], julia_flags = ["--math-mode=ieee"])

---

This pull request only adds the functionality for passing the command line arguments to test/runtests.jl. It is up to the author of a package to add the logic for processing the command line arguments inside their test/runtests.jl file. However, this should be relatively straightforward. For example, if this is the current test/runtests.jl file:

using HelloWorld
using Test

@testset "HelloWorld.jl" begin
    @testset "foo" begin
        @test HelloWorld.foo(1) == 2
    end
    @testset "bar" begin
        @test HelloWorld.bar(2) == 6
    end
    @testset "baz" begin
        @test HelloWorld.baz(3) == 12
    end
end

Then you could modify it to look like this:

using HelloWorld
using Test

@testset "HelloWorld.jl" begin
    if isempty(ARGS) || "all" in ARGS
        all_tests = true
    else
        all_tests = false
    end
    if all_tests || "foo" in ARGS
        @testset "foo" begin
            @test HelloWorld.foo(1) == 2
        end
    end
    if all_tests || "bar" in ARGS
        @testset "bar" begin
            @test HelloWorld.bar(2) == 6
        end
    end
    if all_tests || "baz" in ARGS
        @testset "baz" begin
            @test HelloWorld.baz(3) == 12
        end
    end
end

And now if you run Pkg.test("HelloWorld"; test_args = ["foo", "baz"]), then only the foo and baz testsets will be run.

[KristofferC]

I like it. We'll have to discuss a bit on the exact API for the command line args but this feels pretty ok.

[KristofferC]

bors try

[bors]

try

Already running a review

[bors]

try

Build succeeded

• continuous-integration/travis-ci/push
• continuous-integration/appveyor/branch

[KristofferC]

We probably also want to be able to give julia startup command line options. For example --track-allocations. So perhaps there should be julia_args, test_args or something?

[DilumAluthge]

We probably also want to be able to give julia startup command line options. For example --track-allocations. So perhaps there should be julia_args, test_args or something?

I like julia_args and test_args

Actually I wonder if it will be a little confusing to use "args" twice. How about julia_flags and test_args?

We're going with julia_args and test_args.

[DilumAluthge]

I renamed args to test_args, and I added a new keyword argument julia_flags.

We're going with julia_args and test_args.

[fredrikekre]

Is it not better with julia_args and test_args?
We should also think of a Pkg REPL syntax for this. I am thinking only test_args needs to be supported, since julia_args should not be so commonly used. Maybe something like

pkg> test Example -- arg1 arg2

or something, i.e. using -- as a separator. This could be implemented in a separate PR of course.

[DilumAluthge]

We're going with julia_args and test_args.

[DilumAluthge]

We should also think of a Pkg REPL syntax for this. I am thinking only test_args needs to be supported, since julia_args should not be so commonly used.

Could we support both? Maybe something like:

pkg> test Example --julia-args --track-allocation=all --test-args arg1 arg2

But yeah I definitely agree that we should implement this in a separate pull request.

[StefanKarpinski]

That looks quite a bit worse to me. At that point, shouldn't you just be calling the test script manually?

[fredrikekre]

I agree, we have not even had a feature request for customizing julia flags

[DilumAluthge]

That looks quite a bit worse to me. At that point, shouldn't you just be calling the test script manually?

Fair enough. So if we add REPL support, it will only be for test args.

[DilumAluthge]

I've made the requested changes!

Also, in the interest of "eating our own dog food", I've updated the Pkg test suite to support running specific testsets. I've added an example to .travis.yml to demonstrate.

[cmcaine]

That looks quite a bit worse to me. At that point, shouldn't you just be calling the test script manually?

If you want to run a test script manually with julia I think you have to Pkg.add() all of the deps and extras for the package before it will work, so it could be a moderate pain to do.

Kind of off-topic:

Could solve that in another PR by exposing (if they are not already exposed) some functions for getting the right LOAD_PATH from a Project.toml (like activate, but including extras, and possibly not actually changing LOAD_PATH for you, just returning what you should use).

An extra use case for this is running ancillary test scripts (e.g. test/compare_examples.jl in Gadfly (it diffs binary outputs that have been saved to disk from master and the feature branch for regression tests)).

[fredrikekre]

Kind of off-topic:
Could solve that in another PR by exposing (if they are not already exposed) some functions for getting the right LOAD_PATH from a Project.toml (like activate, but including extras, and possibly not actually changing LOAD_PATH for you, just returning what you should use).

This is exposed(ish) in Julia 1.2; you can there run tests as

julia --project=test test/runtests.jl

[cmcaine]

As mentioned on slack, I think that would require copying the Project.toml to the test directory and making some manual changes first.

[DilumAluthge]

I think what @fredrikekre is referring to is that in Julia 1.2 and higher, you specify the test dependencies in test/Project.toml, not in the main Project.toml file.

[fredrikekre]

Looks pretty good now, but

Also, in the interest of "eating our own dog food", I've updated the Pkg test suite to support running specific testsets. I've added an example to .travis.yml to demonstrate.

can we undo that? I don't think that is necessary and just results in a huge diff.

[DilumAluthge]

Also, in the interest of "eating our own dog food", I've updated the Pkg test suite to support running specific testsets. I've added an example to .travis.yml to demonstrate.

can we undo that? I don't think that is necessary and just results in a huge diff.

Alright I think I've reverted all of those changes!

[fredrikekre]

bors try

[fredrikekre]

I simplified the tests, and fixed running the old code-path.

bors try

[bors]

try

Already running a review

[bors]

try

Build failed

• continuous-integration/travis-ci/push

[fredrikekre]

bors r+

[bors]

Build succeeded

• continuous-integration/appveyor/branch
• continuous-integration/travis-ci/push

[rfourquet]

This is really neat, thanks! Also, my +1 to Pkg REPL support for test_args.