Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

killer sensor: improve testing #1948

Merged
merged 8 commits into from
Jan 10, 2024
Merged

killer sensor: improve testing #1948

merged 8 commits into from
Jan 10, 2024

Commits on Jan 9, 2024

  1. tester-progs: compile killer-tester-32 statically

    Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com>
    kkourt committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    d6cc4bf View commit details
    Browse the repository at this point in the history
  2. vmtests: make a note when all tests skipped

    Currently, if all tests are skipped we print success, which is
    misleading. Print a better message.
    
    Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com>
    kkourt committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    007475a View commit details
    Browse the repository at this point in the history
  3. vmtests: listTests now accepts a pattern

    Previously, we always used "." to list tests in listTests. This patch
    adds it as an argument to be used in a later patch.
    
    Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com>
    kkourt committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    f1307a0 View commit details
    Browse the repository at this point in the history
  4. vmtests: improve running tests from a file

    This patch improves loading tests form a file.
    
    First, it allows for having comments with lines starting with '#'.
    
    Second, it lists the tests based on the provided user pattern instead of
    using it as it is. Previously, the test name was defined by the user
    pattern. This lead to inconsistent reporting. This patch fixes this
    because now we report per-test results rather than per-pattern results.
    
    Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com>
    kkourt committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    bf6a653 View commit details
    Browse the repository at this point in the history
  5. vmtests: use a strict pattern for test execution

    It is possible that a test is a substring of another. Use a strict
    pattern so that we execute only the specified test.
    
    Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com>
    kkourt committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    b798f59 View commit details
    Browse the repository at this point in the history

Commits on Jan 10, 2024

  1. tetragon-vmtests-run: support for detailed results

    There are two ways to skip a test in our CI:
     1. add it to CiBlacklist in split-tetragon-gotests
     2. use t.Skip()
    
    CI results report 1., but they do not report 2.
    
    This means that skipped tests can go easily unnoticed.
    
    To report tests skipped with (2), we need to parse the result files.
    This patch does exactly that. It adds a new option
    --enable-detailed-results. If this option is set, we set KeepAllLogs
    because we need to parse all logs after the tests are finished.
    
    The tests we skip with (1) are marked with ⏭️.
    We introduce a new symbol for the tests that are skipped with (2): ⚡
    
    Parsing the detailed results allows us to also report sub-tests (i.e.,
    tests executed with t.Run()).
    
    Here's an output example:
    
    ```
    ✅      pkg.sensors.tracing.Test_Kprobe_DisableEnablePolicy                                     (total:3 failed:0 skipped:0)    2.680496982s    1m33.952671893s
    ├─✅    Test_Kprobe_DisableEnablePolicy/sensor                                                  1.05s                           (1.05s)
    ├─✅    Test_Kprobe_DisableEnablePolicy/tracing-policy                                          1.09s                           (2.14s)
    └─✅    Test_Kprobe_DisableEnablePolicy                                                         2.56s                           (4.7s)
    ⚡      pkg.sensors.tracing.TestKillerOverride32                                                (total:1 failed:0 skipped:1)    130.759191ms    1m34.083431084s
    ⚡      pkg.sensors.tracing.TestKillerSignal32                                                  (total:1 failed:0 skipped:1)    132.393451ms    1m34.215824535s
    ⚡      pkg.sensors.tracing.TestKillerOverrideBothBits                                          (total:1 failed:0 skipped:1)    119.455061ms    1m34.335279596s
    ⚡      pkg.sensors.tracing.TestKillerOverride                                                  (total:1 failed:0 skipped:1)    121.255148ms    1m34.456534744s
    ⚡      pkg.sensors.tracing.TestKillerSignal                                                    (total:1 failed:0 skipped:1)    136.665421ms    1m34.593200165s
    ✅      pkg.sensors.tracing.TestKillerMulti                                                     (total:1 failed:0 skipped:0)    132.761861ms    1m34.725962026s
    ```
    
    Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com>
    kkourt committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    ad9f7ee View commit details
    Browse the repository at this point in the history
  2. gh: add --enable-detailed-results to vmtests

    Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com>
    kkourt committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    e0ee8d3 View commit details
    Browse the repository at this point in the history
  3. chore(deps): update all lvh-images main

    Signed-off-by: cilium-renovate[bot] <134692979+cilium-renovate[bot]@users.noreply.github.com>
    cilium-renovate[bot] authored and kkourt committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    cf6695d View commit details
    Browse the repository at this point in the history