T14397 baseline test plan

[gctucker]

This is an initial PR to start adding the new baseline test plan. See previous thread on the mailing list for some background:
https://groups.io/g/kernelci/topic/29693068

Initially this would be to run bootrr on a few platforms with a dedicated script to check the device drivers are initialised properly on them. Then it would be necessary to have a way to automatically determine the expected devices in order to scale better with the number of platforms tested on kernelci.org, probably by reading the device tree or some other info which the kernel itself uses to load drivers and probe devices.

Another part of this test plan would be to look for kernel errors rather than plainly relying on LAVA's attempt to reach a login prompt.

This PR relies on the bootrr scripts to be added to a buildroot image, so it depends on the following ones to be resolved first:

• Fix google,kevin script in bootrr Google kevin fixes linux-msm/bootrr#24
• Fix make install in bootrr T14637 fix install directory bootrr#1
• Add bootrr package to buildroot T14637 add bootrr buildroot#3
• Build a baseline buildroot rootfs with bootrr in Jenkins and use it on staging

[gctucker]

I've run this with my test tree and got this report:

gtucker/kernelci-local baseline: 122 tests, 0 regressions (kernelci-local-snapshot-037-1-gce3292b9c9eb)

Test results summary
--------------------

  Test:    baseline
  Tree:    gtucker
  Branch:  kernelci-local
  Kernel:  kernelci-local-snapshot-037-1-gce3292b9c9eb
  URL:     https://gitlab.collabora.com/gtucker/linux.git
  Commit:  ce3292b9c9eb51981c8a8d86ea74d5b871a6bc1f


1  | rk3288-veyron-jaq      | arm   |  61 total:  61 PASS   0 FAIL   0 SKIP
2  | rk3288-veyron-jaq      | arm   |  61 total:  61 PASS   0 FAIL   0 SKIP  

There are 2 lines because they were with kernels built with gcc and clang. The rk3399-gru-kevin results are missing due to some infrastructure error (console was down), next round should have them in principle.

Some sample jobs in LAVA:
https://lava.collabora.co.uk/scheduler/job/1594445 (rk3288-veyron-jaq)
https://lava.collabora.co.uk/scheduler/job/1595670 (rk3399-gru-kevin)

[khilman]

This change (specifically the last patch) is also using a modified buildroot ramdisk. What's missing in our buildroot images, and shouldn't we just update them instead?

[gctucker]

The bootrr scripts are not in the buildroot images. I'm waiting for some changes to get merged into bootrr itself, then I have some patches in buildroot to add a bootrr package using that. We can then decide to either have that enabled directly in the regular buildroot builds or have a variant for the "baseline" test plan (i.e. bootrr + other fast system-wide checks).

Ultimately I think the "baseline" test plan should replace our current "boot" tests (see thread on the mailing list). We can keep 2 separate rootfs images until that happens. I've found that the bootrr tests add about 10s to the LAVA job run times with buildroot, on rk3399-gru-kevin and rk3288-veyron-jaq, but using Debian (Stretch) would have a much bigger impact.

[khilman]

OK, thanks for the clarification. I think getting merging boot and baseline (or getting rid of boot) is the right strategy.

So, I'm fine with this change, but I'd prefer to keep everything that's staging only in the last patch. As I pointed out above, it's not quite a clean separation.

[gctucker]

@khilman So if I understand correctly what you mean, you prefer to have the bootrr scripts and any other potential tool needed for the baseline test plan to be added to the base image rather than have a separate baseline one?

I was initially planning to add a config fragment in buildroot to add bootrr and may other small things if necessary and keep building 2 flavours for each architecture, with and without this fragment. That's why I added the buildroot_baseline_ramdisk and that was not intended to be only for staging, only the temporary image URL was. But I'm OK with directly adding bootrr to the base image and simplify things a bit.

In any case I'll update this PR once other things it depends up have been resolved, I'll add a checklist to keep track of them.

[gctucker]

There's a small issue with how test results from LAVA callbacks are stored in the database: the test suite results are converted into a test group containing some results, but it doesn't take into account the case where there are several test suites in a single job or if the test suite doesn't have the same name as the test plan. In this case, the plan is "baseline" and the test suite is "bootrr", so the results are stored as "bootrr" rather than "baseline". This needs to be fixed so the plan name is used instead, and if there are several test suites in the job (or if subsequent jobs send more results for the same test plan) then the results for each test suite should be kept within a separate group.

[gctucker]

The backend has been fixed to handle LAVA jobs with test suites that have a different name than the test plan, or multiple test suites.

@khilman I think it's best to keep building 2 flavours of buildroot during the transition phase between boot and baseline: the one used for current boot tests, and a "baseline" one with extra scripts installed for bootrr. Once the "baseline" is running in production, we can decide to merge the changes and have only one buildroot file system or drop current boot tests once baseline is running on all the devices. If there's any issue with deploying "baseline" tests, current boot tests won't be affected.

[khilman]

@gctucker: OK with me to have both.

[gctucker]

Tested OK on staging with buildroot file systems from here:
http://staging-storage.kernelci.org/images/rootfs/buildroot/kci-2019.02-2-gefc755ba4a/

Sample boot and baseline runs:
https://lava.collabora.co.uk/scheduler/job/1637437 (boot on x86 Minnowboard Turbot)
https://lava.collabora.co.uk/scheduler/job/1637426 (baseline on arm64 rk3399-gru-kevin)

[gctucker]

Email report for the baseline test plan showing no failures:

kernelci/staging.kernelci.org baseline: 217 tests, 0 regressions (v5.2-rc3-25-g4bf231fa7f04)

Test results summary
--------------------

  Test:    baseline
  Tree:    kernelci
  Branch:  staging.kernelci.org
  Kernel:  v5.2-rc3-25-g4bf231fa7f04
  URL:     https://github.com/kernelci/linux.git
  Commit:  4bf231fa7f042454850eba9d9d8e44060c8a26bb


1  | rk3399-gru-kevin       | clang-8  | arm64 |  78 total:  78 PASS   0 FAIL   0 SKIP
2  | rk3288-veyron-jaq      | gcc-8    | arm   |  61 total:  61 PASS   0 FAIL   0 SKIP
3  | rk3399-gru-kevin       | gcc-8    | arm64 |  78 total:  78 PASS   0 FAIL   0 SKIP  

[gctucker]

@khilman The "base" version of the file system failed for risc-v with this error:

18:37:21 Makefile:609: arch/riscv/Makefile: No such file or directory
18:37:21 make[1]: *** No rule to make target 'arch/riscv/Makefile'.  Stop.

However the "baseline" seems to have worked, so maybe that was a glitch in the Jenkins job. All the other combinations worked.

[gctucker]

For the record, this is how the Jenkins job is configured to run the build script for staging.kernelci.org, with PARAM_ARCH and PARAM_VARIANT set as Matrix axis values:

#!/bin/bash                                                                                                                                                                           

rm -rf kernelci-core
git clone --depth 1 --branch staging.kernelci.org https://github.com/kernelci/kernelci-core

export PATH=$PATH:$PWD/kernelci-core:$PWD/kernelci-core/jenkins
buildroot-builder.sh $PARAM_ARCH $PARAM_VARIANT

Values for PARAM_ARCH:

arc
armel
armeb
arm64
arm64be
mipsel
riscv
x86

Values for PARAM_VARIANT:

base baseline

[gctucker]

@khilman The "base" version of the file system failed for risc-v with this error:

18:37:21 Makefile:609: arch/riscv/Makefile: No such file or directory
18:37:21 make[1]: *** No rule to make target 'arch/riscv/Makefile'.  Stop.

However the "baseline" seems to have worked, so maybe that was a glitch in the Jenkins job. All the other combinations worked.

I ran a build again this morning and it failed in the same way, for some reason the "base" fragment fails with risc-v but not the "baseline" one o.O

[khilman]

The arg order is important here, we want options from $arch to override options from $variant (especially when variant == base) so this should be:
./configs/frags/build ${variant} ${arch}

[gctucker]

OK thanks @khilman, the latest update of this PR worked. I've now changed the staging Jenkins config to this:

#!/bin/bash                                                                                                                                                                           

rm -rf kernelci-core
git clone --depth 1 --branch staging.kernelci.org https://github.com/kernelci/kernelci-core
export PATH=$PWD/kernelci-core:$PATH
./kernelci-core/jenkins/buildroot-builder.sh $ARCH $FRAG

[gctucker]

All the builds passed, I also added a step to delete the output directory before starting each buildroot build to avoid hard to track issues. The binaries can be found here:
http://staging-storage.kernelci.org/images/rootfs/buildroot/kci-2019.02-2-gefc755ba4a/

Email reports for boot and baseline using these file systems:

kernelci/staging.kernelci.org boot: 89 boots: 1 failed, 87 passed with 1 untried/unknown (v5.2-rc3-38-gfb9efe9dfd51)

Full Boot Summary: https://staging.kernelci.org/boot/all/job/kernelci/branch/staging.kernelci.org/kernel/v5.2-rc3-38-gfb9efe9dfd51/
Full Build Summary: https://staging.kernelci.org/build/kernelci/branch/staging.kernelci.org/kernel/v5.2-rc3-38-gfb9efe9dfd51/

Tree: kernelci
Branch: staging.kernelci.org
Git Describe: v5.2-rc3-38-gfb9efe9dfd51
Git Commit: fb9efe9dfd51e94129ef0342785a8bf3c907c4b2
Git URL: https://github.com/kernelci/linux.git
Tested: 34 unique boards, 15 SoC families, 3 builds out of 9

Boot Regressions Detected:

arm:

    multi_v7_defconfig:
        clang-8:
          bcm2836-rpi-2-b:
              lab-collabora-dev: failing since 10 days (last pass: v4.14-11011-ga39fa8b3e663 - first fail: v5.2-rc2-1-gf7e5586b71bd)

Boot Failure Detected:

arm:
    multi_v7_defconfig:
        clang-8:
            bcm2836-rpi-2-b: 1 failed lab

---
For more info write to <info@kernelci.org>

kernelci/staging.kernelci.org baseline: 278 tests, 0 regressions (v5.2-rc3-38-gfb9efe9dfd51)

Test results summary
--------------------
  Test:    baseline
  Tree:    kernelci
  Branch:  staging.kernelci.org
  Kernel:  v5.2-rc3-38-gfb9efe9dfd51
  URL:     https://github.com/kernelci/linux.git
  Commit:  fb9efe9dfd51e94129ef0342785a8bf3c907c4b2


1  | rk3399-gru-kevin       | gcc-8    | arm64 |  78 total:  78 PASS   0 FAIL   0 SKIP
2  | rk3288-veyron-jaq      | gcc-8    | arm   |  61 total:  61 PASS   0 FAIL   0 SKIP
3  | rk3399-gru-kevin       | clang-8  | arm64 |  78 total:  78 PASS   0 FAIL   0 SKIP
4  | rk3288-veyron-jaq      | clang-8  | arm   |  61 total:  61 PASS   0 FAIL   0 SKIP  

[gctucker]

Updated Jenkins job script, for the record:

#!/bin/bash                                                                                                                                                                           

set -e

rm -rf kernelci-core
git clone --depth 1 --branch staging.kernelci.org https://github.com/kernelci/kernelci-core
export PATH=$PWD/kernelci-core:$PATH
rm -rf output
./kernelci-core/jenkins/buildroot-builder.sh $ARCH $FRAG