issue with testdirectory structure, duplicate testscripts, output file,

[shahzebsiddiqui]

This issue addresses several issues observed during testing

• Duplicate test scripts generated by rerunning same command
• Incorrect directory structure for test hierarchy
• Not creating testscript when multiple subtest are declared in single YAML file
• Output file name is not working with multiple subtest, only works with one subtest.
• Output and Error are merged in one file, makes sense to have .out for STDOUT and .err for STDERR.
• Lastly we need to have way to clean destination test directory if it exists, some tests will fail if content is already present, this is present when building the compress.yml.

I have sample test that does compression and decompression using zlib

version: 0.0.1

zlib_compression:
  type: script
  run: |
    dd if=/dev/zero of=1mb.txt bs=1M count=1
    tar cvzf 1mb.tar.gz 1mb.txt
    ls -lh 1mb.tar.gz
    gzip -l 1mb.tar.gz

zlib_decompress:
  type: script
  run: |
    dd if=/dev/zero of=1mb.txt bs=1M count=1
    tar cvzf 1mb.tar.gz 1mb.txt
    ls -l 1mb.tar.gz
    gzip -l 1mb.tar.gz
    gzip -d 1mb.tar.gz

When i run the test everything is fine.

(docs) ssi29@ag-mxg-hulk090> buildtest build -c ~/.buildtest/site/github.com/HPC-buildtest/tutorials/script/compress.yml
________________________________________________________________________________
                             start time: 2020-03-18 16:36:08.965846
                     configuration file: /u/users/ssi29/.buildtest/site/github.com/HPC-buildtest/tutorials/script/compress.yml
                                testdir: /u/users/ssi29/.buildtest/testdir/zlib_compression
                               testpath: /u/users/ssi29/.buildtest/testdir/zlib_compression/zlib_compression_script_03-18-2020-16-36-S.sh
                                logpath: None
________________________________________________________________________________



STAGE                                    VALUE
________________________________________________________________________________
[WRITING TEST]                           PASSED
[RUNNING TEST]                           PASSED
Writing results to /u/users/ssi29/.buildtest/testdir/zlib_compression/run/zlib_compression_script_03-18-2020-16-36-S.out
________________________________________________________________________________
                             start time: 2020-03-18 16:36:09.267447
                     configuration file: /u/users/ssi29/.buildtest/site/github.com/HPC-buildtest/tutorials/script/compress.yml
                                testdir: /u/users/ssi29/.buildtest/testdir/zlib_decompress
                               testpath: /u/users/ssi29/.buildtest/testdir/zlib_decompress/zlib_decompress_script_03-18-2020-16-36-S.sh
                                logpath: None
________________________________________________________________________________



STAGE                                    VALUE
________________________________________________________________________________
[WRITING TEST]                           PASSED
[RUNNING TEST]                           PASSED
Writing results to /u/users/ssi29/.buildtest/testdir/zlib_decompress/run/zlib_decompress_script_03-18-2020-16-36-S.out
Finished running 2 total tests.
==============================================================
                         Test summary
Executed 2 tests
Passed Tests: 2 Percentage: 100.0%
Failed Tests: 0 Percentage: 0.0%

One thing i notice is the output file is using the first key decompress_script as the output. When there are two subtest we don't have details on what being output and to which file.
The other issue is output and error stream are showing content of ls -lh in both stream.
The test output is the following:

(docs) ssi29@ag-mxg-hulk090> cat /u/users/ssi29/.buildtest/testdir/zlib_decompress/run/zlib_decompress_script_03-18-2020-16-27-S.out
Test Name:zlib_decompress_script_03-18-2020-16-27-S
Return Code: 0
---------- START OF TEST OUTPUT ----------------
1mb.txt

-rw-r--r-- 1 ssi29 hks 1135 Mar 18 16:27 1mb.tar.gz

         compressed        uncompressed  ratio uncompressed_name
------------ END OF TEST OUTPUT ----------------
---------- START OF TEST ERROR ----------------
1mb.txt

-rw-r--r-- 1 ssi29 hks 1135 Mar 18 16:27 1mb.tar.gz

         compressed        uncompressed  ratio uncompressed_name
------------ END OF TEST ERROR ----------------

Also, i don't like the idea of having a testscript be regenerated with new time-stamp for multiple runs. It's adding extra overhead on files when they are all same. It makes sense the output and logfile have a time-stamp but test name can be zlib_compression.sh and zlib_decompression.sh.

The sub-directory testdir/zlib_decompress should be named by the name of test file (compress.yml) with .yml extension. It would be testdir/compress/{zlib_decompress.sh}{zlib_compression.sh}

(docs) ssi29@ag-mxg-hulk090> tree /u/users/ssi29/.buildtest/testdir/zlib_decompress/
/u/users/ssi29/.buildtest/testdir/zlib_decompress/
├── log
│   ├── zlib_decompress_script_03-18-2020-16-27-S.log
│   ├── zlib_decompress_script_03-18-2020-16-32-S.log
│   └── zlib_decompress_script_03-18-2020-16-36-S.log
├── run
│   ├── zlib_decompress_script_03-18-2020-16-27-S.out
│   └── zlib_decompress_script_03-18-2020-16-36-S.out
├── zlib_decompress_script_03-18-2020-16-27-S.sh
├── zlib_decompress_script_03-18-2020-16-32-S.sh
└── zlib_decompress_script_03-18-2020-16-36-S.sh

[shahzebsiddiqui]

This has been addressed, duplicate test are no longer generated, the output and log file are going to be duplicates which is fine since multiple runs of same command can be issued.

Test names are resolving properly when multiple subtest are defined in one YAML file.