"Test may not be filtered when --dir is enabled"

[billychow68]

It seems I cannot run my tests with both --dir and --verbose flags, otherwise, I get "Test may not be filtered when --dir is enabled.". What does that error message mean?

I see the code that returns that error message in commander/pkg/app/test_command.go, but since I'm new to this project, that block of code doesn't make sense to me.

The current workaround for me is to use --verbose without --dir calling each test suite individually in a shell script. This loses the aggregated test results.

Are there other options or workarounds?

[dylanhitt]

Hey @billychow68,

Can you send me a little more info. An error message or your dir structure would suffice. I currently cannot recreate the issue. Here is my output - I am able to run commander with --dir or --verbose in either order:

commander ((HEAD detached at v2.1.0)) $ ./commander test --dir --verbose  examples/ 
Starting test against directory: examples/...

2020/07/03 12:32:24 title: 'it should validate exit code'  Command:  exit 1
2020/07/03 12:32:24 title: 'it should validate exit code'  Directory:  
2020/07/03 12:32:24 title: 'it should validate exit code'  Env:  []
2020/07/03 12:32:24 title: 'it should validate exit code'  ExitCode:  1
2020/07/03 12:32:24 title: 'it should validate exit code'  Stdout:  
2020/07/03 12:32:24 title: 'it should validate exit code'  Stderr:  
2020/07/03 12:32:24 title: 'it should validate exit code'  Stdout-Expected:  {[] map[]  0 [] map[] map[]}
2020/07/03 12:32:24 title: 'it should validate exit code'  Stdout-Result:  true
2020/07/03 12:32:24 title: 'it should validate exit code'  Stderr-Expected:  {[] map[]  0 [] map[] map[]}
2020/07/03 12:32:24 title: 'it should validate exit code'  Stderr-Result:  true
2020/07/03 12:32:24 title: 'it should validate exit code'  Exit-Expected:  1
2020/07/03 12:32:24 title: 'it should validate exit code'  Exit-Result:  true
✓ [commander.yaml] [local] it should validate exit code
2020/07/03 12:32:24 title: 'echo hello'  Command:  echo hello
2020/07/03 12:32:24 title: 'echo hello'  Directory:  
2020/07/03 12:32:24 title: 'echo hello'  Env:  []
2020/07/03 12:32:24 title: 'echo hello'  ExitCode:  0
2020/07/03 12:32:24 title: 'echo hello'  Stdout:  hello
2020/07/03 12:32:24 title: 'echo hello'  Stderr:  
2020/07/03 12:32:24 title: 'echo hello'  Stdout-Expected:  {[] map[1:hello] hello 1 [] map[] map[]}
2020/07/03 12:32:24 title: 'echo hello'  Stdout-Result:  true
2020/07/03 12:32:24 title: 'echo hello'  Stderr-Expected:  {[] map[]  0 [] map[] map[]}
2020/07/03 12:32:24 title: 'echo hello'  Stderr-Result:  true
2020/07/03 12:32:24 title: 'echo hello'  Exit-Expected:  0
2020/07/03 12:32:24 title: 'echo hello'  Exit-Result:  true
✓ [commander.yaml] [local] echo hello
2020/07/03 12:32:24 title: 'it should fail'  Command:  invalid
2020/07/03 12:32:24 title: 'it should fail'  Directory:  
2020/07/03 12:32:24 title: 'it should fail'  Env:  []
2020/07/03 12:32:24 title: 'it should fail'  ExitCode:  127
2020/07/03 12:32:24 title: 'it should fail'  Stdout:  
2020/07/03 12:32:24 title: 'it should fail'  Stderr:  /bin/sh: invalid: command not found
2020/07/03 12:32:24 title: 'it should fail'  Stdout-Expected:  {[] map[]  0 [] map[] map[]}
2020/07/03 12:32:24 title: 'it should fail'  Stdout-Result:  true
2020/07/03 12:32:24 title: 'it should fail'  Stderr-Expected:  {[/bin/sh: 1: command invalid: not found] map[]  0 [] map[] map[]}
2020/07/03 12:32:24 title: 'it should fail'  Stderr-Result:  false
✗ [commander.yaml] [local] it should fail
2020/07/03 12:32:24 title: 'it should print hello world'  Command:  echo hello world
2020/07/03 12:32:24 title: 'it should print hello world'  Directory:  
2020/07/03 12:32:24 title: 'it should print hello world'  Env:  []
2020/07/03 12:32:24 title: 'it should print hello world'  ExitCode:  0
2020/07/03 12:32:24 title: 'it should print hello world'  Stdout:  hello world
2020/07/03 12:32:24 title: 'it should print hello world'  Stderr:  
2020/07/03 12:32:24 title: 'it should print hello world'  Stdout-Expected:  {[hello world] map[]  0 [] map[] map[]}
2020/07/03 12:32:24 title: 'it should print hello world'  Stdout-Result:  true
2020/07/03 12:32:24 title: 'it should print hello world'  Stderr-Expected:  {[] map[]  0 [] map[] map[]}
2020/07/03 12:32:24 title: 'it should print hello world'  Stderr-Result:  true
2020/07/03 12:32:24 title: 'it should print hello world'  Exit-Expected:  0
2020/07/03 12:32:24 title: 'it should print hello world'  Exit-Result:  true
✓ [commander.yaml] [local] it should print hello world
2020/07/03 12:32:24 Use docker executor
2020/07/03 12:32:24 Pulling image docker.io/simonbaeumer/test-private:latest
✗ [docker.yaml] [ubuntu] cat /etc/lsb-release
2020/07/03 12:32:25 title: 'it should overwrite env'  Command:  echo $KEY
2020/07/03 12:32:25 title: 'it should overwrite env'  Directory:  
2020/07/03 12:32:25 title: 'it should overwrite env'  Env:  [KEY=local]
2020/07/03 12:32:25 title: 'it should overwrite env'  ExitCode:  0
2020/07/03 12:32:25 title: 'it should overwrite env'  Stdout:  local
2020/07/03 12:32:25 title: 'it should overwrite env'  Stderr:  
2020/07/03 12:32:25 title: 'it should overwrite env'  Stdout-Expected:  {[local] map[]  0 [] map[] map[]}
2020/07/03 12:32:25 title: 'it should overwrite env'  Stdout-Result:  true
2020/07/03 12:32:25 title: 'it should overwrite env'  Stderr-Expected:  {[] map[]  0 [] map[] map[]}
2020/07/03 12:32:25 title: 'it should overwrite env'  Stderr-Result:  true
2020/07/03 12:32:25 title: 'it should overwrite env'  Exit-Expected:  0
2020/07/03 12:32:25 title: 'it should overwrite env'  Exit-Result:  true
✓ [environment.yaml] [local] it should overwrite env
2020/07/03 12:32:25 title: 'echo $KEY'  Command:  echo $KEY
2020/07/03 12:32:25 title: 'echo $KEY'  Directory:  
2020/07/03 12:32:25 title: 'echo $KEY'  Env:  [KEY=global]
2020/07/03 12:32:25 title: 'echo $KEY'  ExitCode:  0
2020/07/03 12:32:25 title: 'echo $KEY'  Stdout:  global
2020/07/03 12:32:25 title: 'echo $KEY'  Stderr:  
2020/07/03 12:32:25 title: 'echo $KEY'  Stdout-Expected:  {[global] map[]  0 [] map[] map[]}
2020/07/03 12:32:25 title: 'echo $KEY'  Stdout-Result:  true
2020/07/03 12:32:25 title: 'echo $KEY'  Stderr-Expected:  {[] map[]  0 [] map[] map[]}
2020/07/03 12:32:25 title: 'echo $KEY'  Stderr-Result:  true
2020/07/03 12:32:25 title: 'echo $KEY'  Exit-Expected:  0
2020/07/03 12:32:25 title: 'echo $KEY'  Exit-Result:  true
✓ [environment.yaml] [local] echo $KEY
2020/07/03 12:32:25 title: 'echo hello'  Command:  echo hello
2020/07/03 12:32:25 title: 'echo hello'  Directory:  
2020/07/03 12:32:25 title: 'echo hello'  Env:  []
2020/07/03 12:32:25 title: 'echo hello'  ExitCode:  0
2020/07/03 12:32:25 title: 'echo hello'  Stdout:  hello
2020/07/03 12:32:25 title: 'echo hello'  Stderr:  
2020/07/03 12:32:25 title: 'echo hello'  Stdout-Expected:  {[] map[]  0 [] map[] map[]}
2020/07/03 12:32:25 title: 'echo hello'  Stdout-Result:  true
2020/07/03 12:32:25 title: 'echo hello'  Stderr-Expected:  {[] map[]  0 [] map[] map[]}
2020/07/03 12:32:25 title: 'echo hello'  Stderr-Result:  true
2020/07/03 12:32:25 title: 'echo hello'  Exit-Expected:  0
2020/07/03 12:32:25 title: 'echo hello'  Exit-Result:  true
✓ [minimal_test.yaml] [local] echo hello

Results

✗ [commander.yaml] [local] 'it should fail', on property 'Stderr'

Expected

/bin/sh: invalid: command not found

to contain

/bin/sh: 1: command invalid: not found

✗ [docker.yaml] [ubuntu] 'cat /etc/lsb-release' could not be executed with error message:
could not pull image 'docker.io/simonbaeumer/test-private:latest' with error: 'Error response from daemon: pull access denied for simonbaeumer/test-private, repository does not exist or may require 'docker login': denied: requested access to the resource is denied'

Duration: 0.373s
Count: 8, Failed: 2
Test suite failed, use --verbose for more detailed output

[billychow68]

Hello @dylanhitt,

$ tree
.
├── README.md
├── commander
├── media
│   ├── test1.wav
│   ├── test2.wav
│   └── test3.wav
├── run.sh
└── tests
    ├── adjust_audio_volume.yaml
    ├── animate_builtin.yaml
    ├── exclude
    │   └── post_logs.yaml
    ├── install_audio.yaml
    ├── play_builtin.yaml
    └── scripts
        └── gq.sh

4 directories, 12 files

Run Commander with --dir flag:

$ ./commander test --dir tests/
Starting test against directory: tests/...

✓ [adjust_audio_volume.yaml] [local] GQ0200 - Test method=adjust_audio_volume, volume=0
✓ [adjust_audio_volume.yaml] [local] GQ0201 - Test method=adjust_audio_volume, volume=1
✓ [adjust_audio_volume.yaml] [local] GQ0202 - Test method=adjust_audio_volume, volume=25
... 
(I purposely limited the long output)
...
Duration: 11.743s
Count: 60, Failed: 8
Test suite failed, use --verbose for more detailed output
$

Run Commander with both, --dir and --verbose, flags in that order:

$ ./commander test --dir tests/ --verbose
Test may not be filtered when --dir is enabled

I just learned that if I reverse the order of the flags it works.

$ ./commander test --verbose --dir tests/
Starting test against directory: tests/...

2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25'  Command:  ./tests/scripts/gq.sh 202
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25'  Directory:
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25'  Env:  
...
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25'  ExitCode:  0
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25'  Stdout:  {"success":true}
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25'  Stderr:
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25'  Stdout-Expected:  {[{"success":true}] map[]  0 [] map[] map[]}
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25'  Stdout-Result:  true
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25'  Stderr-Expected:  {[] map[]  0 [] map[] map[]}
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25'  Stderr-Result:  true
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25'  Exit-Expected:  0
2020/07/03 10:11:51 title: 'GQ0202 - Test method=adjust_audio_volume, volume=25'  Exit-Result:  true
✓ [adjust_audio_volume.yaml] [local] GQ0202 - Test method=adjust_audio_volume, volume=25
...
Duration: 12.220s
Count: 60, Failed: 8
Test suite failed, use --verbose for more detailed output

Minor bug: noticed it prints "Test suite failed, use --verbose for more detailed output" even though I ran it with --verbose?

[SimonBaeumer]

The --verbose is interpreted as an argument which is used to filter for a specific test in your test suite.
An error message is displayed because filtering is not enabled with the --dir option.

[dylanhitt]

@SimonBaeumer I think this can be closed. As it will be addressed in #77. I can add a check to not print Test suite failed, use --verbose for more detailed output in #126.

[SimonBaeumer]

Added a draft for the filter implementation. Closes this issue because of #77 as @dylanhitt mentioned above.