bug (regression): venom run DIR_NAME doesn't run anymore test suites in alphabetical order

[nqb]

Hello,

On Venom 1.0.1, it looks like venom run DIR_NAME doesn't run anymore test suites in alphabetical order.
It was the case on 1.0.0-rc-7.

According to our tests, it looks like behavior is different between distributions. We noticed that behavior on Debian 11 but we were not able to replicate on EL8.

Steps to reproduce

Files:

# cat a.yml 
name: a
version: 2
testcases:
- name: a
  steps:
  - type: exec
    script: echo 'a' >> myfile

# cat A.yml 
name: A
version: 2
testcases:
- name: A
  steps:
  - type: exec
    script: echo 'A' >> myfile

Venom run and results on v1.0.0-rc.7:

# venom version
Version venom: v1.0.0-rc.7

# venom run ~/venom/basic/
 • A (/root/venom/basic/A.yml)
        • A SUCCESS
 • a (/root/venom/basic/a.yml)
        • a SUCCESS
# cat ~/venom/basic/myfile
A
a
# rm ~/venom/basic/myfile

Venom run and results on v1.0.1:

# venom version
Version venom: v1.0.1

# venom run ~/venom/basic/
 • a (/root/venom/basic/a.yml)
        • a SUCCESS
 • A (/root/venom/basic/A.yml)
        • A SUCCESS
# cat ~/venom/basic/myfile
a
A

Actual results

Venom v1.0.1 doesn't run test suites in alphabetical order when they are in a directory.

Expected results

Venom v1.0.1 should run test suites in alphabetical order when they are in a directory.

Additional context

Certainly related to #430

[marcaudefroy]

Hum, What is your use case? :)
IMO, if your testsuites depends of the file orders. Maybe the architecture of your tests isn't really good.

[JeGoi]

Hi @marcaudefroy,
Thank you for your answer.
In a directory, we are using numbers in the beginning of the file's name to define the order.
Ex: 00_ (first action) 05_ (second) etc...
Like here: https://github.com/inverse-inc/packetfence/tree/feature/venom-1.0.0/t/venom/test_suites/captive_portal/teardown

Current Output:

    ========================================================================
    =
    =       Running Venom test suite .../test_suites/captive_portal/teardown
    =
     • Delete captive portal connection profile (.../test_suites/captive_portal/teardown/30_delete_connection_profile.yml)
     • Disable MAC Authentication on switch01 (.../test_suites/captive_portal/teardown/05_disable_mac_auth_dot1x_int.yml)
     • Commit changes in configuration (.../test_suites/captive_portal/teardown/10_commit_config.yml)
     • Disable dynamic VLAN on switch01 (.../test_suites/captive_portal/teardown/07_disable_dynamic_vlan.yml)
     • Delete node01 (.../test_suites/captive_portal/teardown/15_delete_node01.yml)
     • Clean httpd.portal cache (.../test_suites/captive_portal/teardown/25_clean_httpd_portal_cache.yml)

Expected:

    ========================================================================
    =
    =       Running Venom test suite .../test_suites/captive_portal/teardown
    =
     • Disable MAC Authentication on switch01 (.../test_suites/captive_portal/teardown/05_disable_mac_auth_dot1x_int.yml)
     • Disable dynamic VLAN on switch01 (.../test_suites/captive_portal/teardown/07_disable_dynamic_vlan.yml)
     • Commit changes in configuration (.../test_suites/captive_portal/teardown/10_commit_config.yml)
     • Delete node01 (.../test_suites/captive_portal/teardown/15_delete_node01.yml)
     • Clean httpd.portal cache (.../test_suites/captive_portal/teardown/25_clean_httpd_portal_cache.yml)
     • Delete captive portal connection profile (.../test_suites/captive_portal/teardown/30_delete_connection_profile.yml)

So, if it is not the good way to do organize tests, please, what is it the good way to do it?
Thank you

[JeGoi]

(NB: tests have been made with v1.0.0 and v1.0.1 even if the branch is just with v1.0.0)

[nqb]

Hello @marcaudefroy,

We used same logic as CDS (https://github.com/ovh/cds/tree/a91511eea8ec7eec6fc27fa0c695dc7f3b9f4433/tests) when we started to write tests with Venom 0.27.0

Our use case is: we want to run several test suites in specific order.

Having everything in one test suite (where order is respected) could be an approach but it think it's more hard to read what is doing test suite. Also before custom executor feature, it was more simple for us to create small test suites that can be reused by all our tests.

[yesnault]

Venom does not sort testsuite anymore, you have to sort them at run.

venom run 01_foo.yml 02_foo.yml will run 01 before 02.
venom run 02_foo.yml 01_foo.yml will run 02 before 01.

if you want to order the testsuite, you have to order them on the command line.

[nqb]

Hello @yesnault,

Thanks for clarification, we will adjust our codebase.
I suggest to add a note about this change because it was not clear for me that 1.0.0 come with that change.

[yesnault]

@nqb release note updated on https://github.com/ovh/venom/releases/tag/v1.0.0, doc on #474