Add pytest option to randomly select and run number/percentage of tests from the collection.

[jameerpathan111]

Problem Statement

• The Automation team requested to add support for pytest option to randomly select and run number/percentage of tests from the collection.

Solution

• Add pytest option --select-random-tests: Number/Percentage of tests to randomly select and run from selected test collection.
• Add pytest option --random-seed: Seed value for random test collection. It should be used with --select-random-tests option.
• To rerun the same test collection use the same seed value, check robottelo.log for seed value generated(if it was not provided before).

Examples

• pytest tests/foreman/ --select-random-tests '5%'
• pytest tests/foreman/ --select-random-tests 4 --random-seed fsdaxv

Related Issues

• See SAT-20829

[jameerpathan111]

trigger: test-robottelo
pytest: tests/foreman/api/ --select-random-tests 2

[Satellite-QE]

PRT Result

Build Number: 7429
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/api/ --select-random-tests 2 --external-logging
Test Result : ======== 2 passed, 2305 deselected, 2336 warnings in 661.87s (0:11:01) =========

[jameerpathan111]

trigger: test-robottelo
pytest: tests/foreman/api/test_location.py --select-random-tests '10%' --component OrganizationsandLocations

[Satellite-QE]

PRT Result

Build Number: 7431
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/api/test_location.py --select-random-tests 10% --component OrganizationsandLocations --external-logging
Test Result : ========== 4 passed, 29 deselected, 65 warnings in 623.06s (0:10:23) ===========

[JacobCallahan]

Can you generate a random seed that would be logged before use? Also, adding the option to pass a specific seed would allow us to repeat a random test selection.

[jameerpathan111]

Can you generate a random seed that would be logged before use? Also, adding the option to pass a specific seed would allow us to repeat a random test selection.

I don't completely understand what you mean here. As per @jyejare's request he'd like user to specify the number/percentage of tests to collect. Or did you mean if no value is provided we should pick random number of tests?

[ogajduse]

Agreed with Jake. It should be really simple to implement. With the current implementation, it should be only a matter of running random.seed(seed).

def random_pick(items, k, seed=None):
    if seed is not None:
        random.seed(seed)
    return random.sample(items, k) 

items = [1,2,3,4,5,6,7,8,9,10]
k = 2
seed = 42

picked_items = random_pick(items, k, seed)
print(f"Picked {k} random items (seed: {seed}):", picked_items)
# Picked 2 random items (seed: 42): [2, 1]
picked_items = random_pick(items, k, seed)
print(f"Picked {k} random items (seed: {seed}):", picked_items)
#Picked 2 random items (seed: 42): [2, 1]
picked_items = random_pick(items, k, seed)
print(f"Picked {k} random items (seed: {seed}):", picked_items)
#Picked 2 random items (seed: 42): [2, 1]

Edit: If you will implement this logic, you will also need to define a new cmd option to allow the user to pass the seed.

[rmynar]

Agree with @JacobCallahan, there should be a way to rerun the same set of tests. There's a good tip how to implement "seed" from @ogajduse. But if you're not going to implement the seed, there should be at least a log entry that contains all the selected test, which should be easily copy&pasted.

[jameerpathan111]

• pytest tests/foreman/ --select-random-tests 4

Ah, now I get what Jake meant. Thanks for explaining:) Implemented your suggestion.

[jameerpathan111]

(env-robottelo3) jpathan@jpathan-mac robottelo % pytest tests/foreman/api/test_activationkey.py --select-random-tests 4 --collect-only --disable-warnings 
=============================================================================================================================================== test session starts ===============================================================================================================================================
collected 64 items / 60 deselected / 4 selected                                                                                                                                                                                                                                                                   

<Dir robottelo>
  <Package tests>
    <Package foreman>
      <Package api>
        <Module test_activationkey.py>
          <Function test_negative_update_limit[2]>
          <Function test_positive_update_name[alphanumeric]>
          <Function test_positive_add_host_collections>
          <Function test_positive_delete[numeric]>

================================================================================================================================== 4/64 tests collected (60 deselected) in 1.40s ==================================================================================================================================

Check the logs/robottelo.log for seed value.
2024-07-02 13:05:03 - robottelo - INFO - Modifying test collection based on --select-random-tests pytest option. Tests collected: 63, Tests to select randomly: 4, Seed value: FxACYYBRwf

(env-robottelo3) jpathan@jpathan-mac robottelo % pytest tests/foreman/api/test_activationkey.py --select-random-tests 4 --random-seed FxACYYBRwf --collect-only --disable-warnings
=============================================================================================================================================== test session starts ===============================================================================================================================================
collected 64 items / 60 deselected / 4 selected                                                                                                                                                                                                                                                                   

<Dir robottelo>
  <Package tests>
    <Package foreman>
      <Package api>
        <Module test_activationkey.py>
          <Function test_negative_update_limit[2]>
          <Function test_positive_update_name[alphanumeric]>
          <Function test_positive_add_host_collections>
          <Function test_positive_delete[numeric]>

================================================================================================================================== 4/64 tests collected (60 deselected) in 1.33s ==================================================================================================================================```

[rmynar]

I'm not sure about Automation team needs, but current solution seems to be able to select testcases randomly and also replay the same set of tests. The replay using seed breaks as soon as test collection is modified, but I don't think this is a problem. ACK

[JacobCallahan]

Very nice!!