Shared code

[davidanthoff]

There will certainly be uses cases where one wants to use some test specific piece of code in more than one @testitem. The only way to deal with this at the moment is to put that code into its own .jl file and then include that file from inside the @testitem. While that works, there are probably better ways to organize this, and this issue is a place to discuss them.

Broadly speaking there are probably different kind of cases: 1) function and type definitions that are meant to be used inside @testitems, 2) initialization type code, 3) maybe more?

Next major question is to what degree this kind of shared code should be handled by Revise vs just re-execution of a piece of code.

One idea for syntax could be something like a @testmodule Foo begin end for declaring a new module that can then be used via using Foo from inside a @testitem, maybe... That probably would work quite well for the function and type definition case, but might not be the right model for init code?

[davidanthoff]

Quick brain dump on my current thinking for this:

Test snippet

The first case is simply code that we want to reuse at the global scope inside a @testitem. This is essentially the case that can already be handled via include, but if the piece of code is very short than the requirement to put it into a separate file becomes cumbersome. For this we could introduce a new macro @testsnippet. It would work like this:

@testsnippet Foo begin
    x = "Something"
    y = x * "And"
end

@testitem "Bar" begin
    @includets Foo

    @test x != y
end

The test item Bar would run the same code as if we had written

@testitem "Bar" begin
    x = "Something"
    y = x * "And"

    @test x != y
end

The main open question here is how we handle namespace issues... Is there one flat namespace per package and all the @testsnippets defined are just dropped into that flat namespace? Is there some more hierarchy? Need to think a little bit more about that...

Shared state

This is trickier, and generally is more in the spirit of the setup and teardown methods in something like NUnit. The problem it attempts to solve is that in some cases it can be useful to generate some state and then reuse that state for the execution of multiple @testitems. An example might be to load a very large dataset that is needed by multiple @testitems and we don't want to reload the same data over and over again during the execution of each @testitem.

I think there are multiple ways we could solve this, here I lay out two.

Test module

The idea around test modules would work like this:

@testmodule Foo begin
    x = load_large_data()
end

@testitem "Something" begin
    @usingts Foo

    @test bar(Foo.x) == "Something"
end

So Foo is like a regular module, and if a @testitem imports that module via the @usingts macro, then Foo gets loaded. If another @testitem then also uses @usingts Foo later on, it will access the already loaded version of Foo.

We could also show all the currently loaded test modules in the Julia Workspace, with an option to terminate them invidiually.

Open question is again the namespace issue. Another question is how code changes would be detected, my best guess is that we could resolve that pretty easily with Revise.jl, though.

Explicit setup and teardown with global data storage

Another option might be to have some explicit setup (and maybe teardown) methods. Things might look like this:

@testsetup Foo begin
    x = load_large_data()
    @ts_store_data :foo_data x # Here we store x under the name :foo for later use by `@testitem`s
end

@testitem "Something" setup=[Foo] begin
    x = @ts_load :foo
    ...
end

So here we would have macros @ts_store and @ts_load to store and load data in a global shared state data structure. An individual @testitem could then also specify which @testsetups must have run before it executes.

[lmiq]

The test module alternative, if it can follow the same syntax of using, import, etc, of standard Julia modules, is the best, IMO.

[NHDaly]

One other thing to consider: if tests allow grouping within a file, along the discussion in #5, the testsetup could be per-test-group.

[bilderbuchi]

I think pytest is in general a good model to take inspiration/ideas from. In this case, shared code is handled by pytest "fixtures". These are basically functions that generate some object (data, function) that tests use by including it as an argument (test cases are functions in pytest).

As a julia newbie, maybe let me describe how pytest works well for the present usecase:

A fixture has a "scope" (function, class, module, package, session) that dictates how often it is invoked. The default is for every test (function), but you can also e.g. have one object that is reused by tests in a whole module, or once per test session. This is a really useful concept, as you can tailor your fixture to the scope you need, and not more.
Additionally, for efficiency, test execution order is automatically made so that there are the least amount of fixture invocations (e.g. all tests using one module fixture are run together/after another, so that you only have to invoke the fixture once).

Setup/teardown is automatically handled by using the Python yield keyword that temporarily hands execution back to the caller:

@pytest.fixture
def sending_user(mail_admin):
    user = mail_admin.create_user()
    yield user # -> at this point the test using the code gets the `user` object and runs
    mail_admin.delete_user(user)

I like that this is very natural, and you are using the same concept of a fixture function without having to learn any additional thing about setup/teardown. (in case you're not aware, the @pytest.fixture is a Python decorator, not a macro)

Fixtures can be cascaded/stacked, so you can neatly assemble hierarchies where one fixture uses another. (See above example,
mail_admin is also a fixture).

The one thing I don't like about pytest fixtures is that their discovery is a bit "magical" -- they are typically defined in a conftest.py file, and if you do that, your test file automatically can use that as a function argument. This breaks the usual flow of having to do import bla if you want to use bla, and your test function arguments are "special" in that their name matters.
I guess all I'm saying, maybe that is an aspect that should not necessarily be picked up, but probably this does not really apply in Julia, anyway.

[schlichtanders]

I just stumbled upon @testsetup in the source code's master branch.
I would like to add another usecase to the table and hope this can still find a place in the discussion before @testsetup is released.

The usecase: Test integration with Pluto.jl

More concretely, Pluto is sometimes used instead of an IDE to code julia files. I myself like it too. There is even PlutoTest which offers a Pluto specific implementation of the @test macro with fancy Pluto interactivity support.

The problem with @testsetup

I now tried to use @testsetup but it is unfortunately clunky inside Pluto. Imagine: In Pluto you like to have small cells so that you can get rich output per cell. In addition everything is interdependent in Pluto. A @testsetup which introduces an extra module scope does not interact well here.

I would vote for @testsnippet

A @testsnippet solution on the other hand would indeed be very helpful to use this within Pluto. You would no longer need to define a single big @testitem cell with everything in it. Instead you could leverage Pluto's cells for rich intermediate outputs inside Pluto, while having everything nicely hidden if used as the plain julia file.

[schlichtanders]

I just thought about a complete alternative approach.

One could support a @if_test (or @iftest) macro which are intended to be collected as additional statements which run before all @testitems.

This would have the benefit that no extra name or tagging is needed.
Also, thanks to precompilation, this might even be quicker than a @testsnippet approach. It is because all @iftest can be included into one overall extra Module, which is then implicitly included next to Test and the original package.

(EDIT: Of course, both @testsnippet or @iftest should include the original package themselves)

[schlichtanders]

@davidanthoff not sure about your timeline. In case you plan to make a new release soon, please take 5 minutes to read up on my thoughts above before releasing.

[schlichtanders]

I just realized that the @testsetup macro was already released (0.2.2 has tests for it)