location where testscripts will be written

[shahzebsiddiqui]

@vsoch The location of the test configuration directory for a particular build can take the relative path of configuration file to build the test path. The relative path is relative to ~/.buildtest/site since buildtest get will clone repos in that sub-directory.

The current setting for testdirectory is defined in settings.yml as follows

build:
  # directory where test scripts will be generated
  testdir: $HOME/.buildtest/testdir

So if user specifies the following buildtest build -c ~/.buildtest/site/github.com/HPC-buildtest/tutorials/compilers/hello.f.yml then the test directory will be the following

test directory: $HOME/.buildtest/testdir/github.com/HPC-buildtest/tutorials/compilers/
test path: $HOME/.buildtest/testdir/github.com/HPC-buildtest/tutorials/compilers/hello.f.yml.SUBTEST-NAME.sh

We can optionally have output and error file for each sub-test.

Output: $HOME/.buildtest/testdir/github.com/HPC-buildtest/tutorials/compilers/hello.f.yml.SUBTEST-NAME.out
Error: $HOME/.buildtest/testdir/github.com/HPC-buildtest/tutorials/compilers/hello.f.yml.SUBTEST-NAME.err

[vsoch]

Hmm, so what if the configuration file isn't relative to the site path?

[shahzebsiddiqui]

Well for now, I take it user has to run buildtest get <repo> to retrieve the files. This should clone under $HOME/.buildtest/site. If user can specify any path outside of this, it may cause issue with test discovery.

[vsoch]

You are envisioning the use case of a site / cluster that does maintain recipes in one location. For a user that is using buildtest on his or her system, it's more of an annoyance to require files to be in the root. It's much more likely they would want to run a one off test, right in their current directory, or even on travis! What if, instead of enforcing that test output goes to this root, we do akin to what snakemake does, and create a local .buildtest (or named something else in case they run it in $HOME) that includes the output? And then the user can use an input arg to specify some (non default) location.

You know, snakemake already supports all of these executors (from SLURM to LSF to cloud executors) plus environment modules, conda...

ENVIRONMENT MODULES:
  --use-envmodules      If defined in the rule, run job within the given
                        environment modules, loaded in the given order. This
                        can be combined with --use-conda and --use-
                        singularity, which will then be only used as a
                        fallback for rules which don't define environment
                        modules.

Serious question - have you considered modifying snakemake to create tests? What is snakemake lacking?

[vsoch]

Is this still an issue? Note that we now use site. You previously mentioned that you don't want to write test results here (but instead in a local .buildtest folder relative to where the test is run) - should we rename the title of this issue to discuss this? I think it's actually a great idea - when I use snakemake I run from a local directory, and a .snakemake folder has the results that I need - right there and easy to find. This seems like a better design than what we are currently doing, and I'd like to change it. It makes sense that if I run buildtest in a folder with test files, I want the output to be right there for me to interact with - I shouldn't have to go poking around some monstrous folder hidden in my home.

[shahzebsiddiqui]

Sorry for the confusion, i have updated the issue. The intent of this ticket was to decide where to write the tests. The test paths need to be unique but I am open to new path. The simple approach i had in mind was take the subdirectory relative to ~/.buildtest/site up to the config file and insert that in testdir defined in settings.yml.

Just to be clear this has nothing to do with where tests-configs (YAML) are stored, we already have that solved.

[vsoch]

I am hugely in support of following snakemake and writing the test files (and logs, run, etc.) in a local directory, .buildtest to the run, unless the user specifies something different.

[vsoch]

Assigning myself - we've discussed this on slack, here is an overview of what will be updated

Client

I would want the test directory to be generated wherever the user runs it, as default. For example, if I'm in my repository with a script hello-world.yml and I run as follows:

buildtest build -c hello-world.yml

I'd want the structure to be created:

hello-world.yml
.buildtest/
  hello-world/test-one/
  hello-world/test-two/

and actually that works much more nicely than the global config, because we can name the test subdirectories according to the yaml file name they are derived from. If that's done in some global folder there is huge chance of name conflict.

I think it's much more likely users will do a command like I just showed above, relative to a file in a local directory. The whole need to maintain this central thing at $HOME/.buildtest makes sense for config, and maybe for some kind of global site, but not for the single user running one off tests.

And then let's say I want to run the test and have the output go somewhere else (other than .buildtest) I would do:

buildtest build -c hello-world.yml --testdir ~/mytests/anotherplace

Organization

Right now the tests were generated under .buildtest/testdir based on the name (Key) in the yaml:

$ tree /home/vanessa/.buildtest/testdir/ -L 1
/home/vanessa/.buildtest/testdir/
├── hello_dinosaur
└── slurm_check

but you can imagine that this breaks easily -what if there is a duplicate name? How do we link back to the file? So for my proposed changes, the structure would be updated to be:

$ /home/vanessa/myrepo
├── hello-slurm.yml
├── .buildtest
  ├── hello-slurm
    ├── hello_dinosaur
    └── slurm_check

So that I can very intuitively see hello-slurm.yml, and then know that my tests are run in .buildtest, and if I want to find the tests for hello-slurm.yml I look in... the hello-slurm folder! And then the keys of the file are the next level.

I have assigned myself, will make some time this week to implement.

[vsoch]

I think we are done here too! Closing issue.