add build config and test-plan for PREEMPT_RT by khilman · Pull Request #475 · kernelci/kernelci-core

khilman · 2020-08-25T20:02:47Z

Add minimal set of build configurations, including PREEMPT_RT=y config for the rt-stable branches.
Also add a test-plan based on LAVA test-definition from Daniel Wagner's fork of LInaro test-definition repo.

NOTE: PR currently based on the kselftest PR #445 due to conflicts in adding test-plan to test-configs and lab-configs.

Things to fix

move parameters to device-specific part of test-configs (e.g. num threads will be SoC specific)

broonie · 2020-08-25T20:04:57Z

Makes sense to me.

khilman · 2020-08-25T23:02:32Z

Manually triggered LAVA job: https://lava.baylibre.com/scheduler/job/31249
and results on staging: https://staging.kernelci.org/test/plan/id/5f4581c91902afaf5f544a7b/

@gctucker something kind of curious. The frontend shows some of the results as unknown, but the LAVA_SIGNAL sent says fail. Here's the relevant part of the LAVA job log:

- {"dt": "2020-08-25T22:56:06.243696", "lvl": "target", "msg": "t0-min-latency pass 21 us"}
- {"dt": "2020-08-25T22:56:06.243948", "lvl": "target", "msg": "t0-avg-latency pass 42 us"}
- {"dt": "2020-08-25T22:56:06.248144", "lvl": "target", "msg": "t0-max-latency fail 4344 us"}
- {"dt": "2020-08-25T22:56:06.248434", "lvl": "target", "msg": "t1-min-latency pass 24 us"}
- {"dt": "2020-08-25T22:56:06.248614", "lvl": "target", "msg": "t1-avg-latency pass 39 us"}
- {"dt": "2020-08-25T22:56:06.248743", "lvl": "target", "msg": "t1-max-latency fail 4782 us"}
- {"dt": "2020-08-25T22:56:06.254064", "lvl": "target", "msg": "+ ../../utils/send-to-lava.sh ./output/result.txt"}
- {"dt": "2020-08-25T22:56:06.340646", "lvl": "target", "msg": "<LAVA_SIGNAL_TESTCASE TEST_CASE_ID=t0-min-latency RESULT=pass UNITS=us MEASUREMENT=21>"}
- {"dt": "2020-08-25T22:56:06.350176", "lvl": "debug", "msg": "Received signal: <TESTCASE> TEST_CASE_ID=t0-min-latency RESULT=pass UNITS=us MEASUREMENT=21"}
- {"dt": "2020-08-25T22:56:06.350818", "lvl": "results", "msg": {"case": "t0-min-latency", "definition": "1_cyclictest", "measurement": !!python/object/apply:decimal.Decimal ["21"], "result": "pass", "units": "us"}}
- {"dt": "2020-08-25T22:56:06.420675", "lvl": "target", "msg": "<LAVA_SIGNAL_TESTCASE TEST_CASE_ID=t0-avg-latency RESULT=pass UNITS=us MEASUREMENT=42>"}
- {"dt": "2020-08-25T22:56:06.423155", "lvl": "debug", "msg": "Received signal: <TESTCASE> TEST_CASE_ID=t0-avg-latency RESULT=pass UNITS=us MEASUREMENT=42"}
- {"dt": "2020-08-25T22:56:06.423739", "lvl": "results", "msg": {"case": "t0-avg-latency", "definition": "1_cyclictest", "measurement": !!python/object/apply:decimal.Decimal ["42"], "result": "pass", "units": "us"}}
- {"dt": "2020-08-25T22:56:06.498716", "lvl": "target", "msg": "<LAVA_SIGNAL_TESTCASE TEST_CASE_ID=t0-max-latency RESULT=fail UNITS=us MEASUREMENT=4344>"}
- {"dt": "2020-08-25T22:56:06.499538", "lvl": "debug", "msg": "Received signal: <TESTCASE> TEST_CASE_ID=t0-max-latency RESULT=fail UNITS=us MEASUREMENT=4344"}
- {"dt": "2020-08-25T22:56:06.500106", "lvl": "results", "msg": {"case": "t0-max-latency", "definition": "1_cyclictest", "measurement": !!python/object/apply:decimal.Decimal ["4344"], "result": "fail", "units": "us"}}
- {"dt": "2020-08-25T22:56:06.593601", "lvl": "target", "msg": "<LAVA_SIGNAL_TESTCASE TEST_CASE_ID=t1-min-latency RESULT=pass UNITS=us MEASUREMENT=24>"}
- {"dt": "2020-08-25T22:56:06.594449", "lvl": "debug", "msg": "Received signal: <TESTCASE> TEST_CASE_ID=t1-min-latency RESULT=pass UNITS=us MEASUREMENT=24"}
- {"dt": "2020-08-25T22:56:06.595037", "lvl": "results", "msg": {"case": "t1-min-latency", "definition": "1_cyclictest", "measurement": !!python/object/apply:decimal.Decimal ["24"], "result": "pass", "units": "us"}}
- {"dt": "2020-08-25T22:56:06.680753", "lvl": "target", "msg": "<LAVA_SIGNAL_TESTCASE TEST_CASE_ID=t1-avg-latency RESULT=pass UNITS=us MEASUREMENT=39>"}
- {"dt": "2020-08-25T22:56:06.681478", "lvl": "debug", "msg": "Received signal: <TESTCASE> TEST_CASE_ID=t1-avg-latency RESULT=pass UNITS=us MEASUREMENT=39"}
- {"dt": "2020-08-25T22:56:06.682296", "lvl": "results", "msg": {"case": "t1-avg-latency", "definition": "1_cyclictest", "measurement": !!python/object/apply:decimal.Decimal ["39"], "result": "pass", "units": "us"}}
- {"dt": "2020-08-25T22:56:06.936498", "lvl": "target", "msg": "<LAVA_SIGNAL_TESTCASE TEST_CASE_ID=t1-max-latency RESULT=fail UNITS=us MEASUREMENT=4782>"}
- {"dt": "2020-08-25T22:56:06.936910", "lvl": "target", "msg": "+ set +x"}
- {"dt": "2020-08-25T22:56:06.937279", "lvl": "target", "msg": "<LAVA_SIGNAL_ENDRUN 1_cyclictest 31249_1.6.2.4.5>"}
- {"dt": "2020-08-25T22:56:06.937505", "lvl": "target", "msg": "<LAVA_TEST_RUNNER EXIT>"}

gctucker · 2020-08-26T08:27:52Z

@khilman Yes that's because tests that have always been reporting a "FAIL" are not considered as regressions. It's an artifact of the UI in the frontend, we should show the status differently to reflect failures that aren't regressions. Something like this maybe:

Green check: pass
Orange cross: always failed
Red cross: regression
Grey question mark: unknown

That deserves an issue in kernelci-frontend.

gctucker

A few minor things - and yes that will need to be rebased once #445 has been merged. Or, the dependency could be inverted if we wanted to merge this first and then #445 with kselftests.

build-configs.yaml

gctucker · 2020-09-01T08:26:43Z

build-configs.yaml

+    configs:
+      - 'CONFIG_EXPERT=y'
+      - 'CONFIG_PREEMPT_RT=y'
+      - 'CONFIG_PREEMPT_RT_FULL=y' # <= v4.19


I guess it won't hurt to have this config in the fragment with newer kernel, as it should just result in a warning from merge_config.sh. If however we start having strict checks and all the config options must appear in the resulting .config then we would need to create a separate fragment, e.g. preempt_rt_4.19 with that RT_FULL config option and the default one without it.

khilman · 2020-09-16T15:35:16Z

Fixed alphabetical order issues mentioned above and rebased onto master now that kselftest is merged.
Should be ready for broader testing in staging now.

gctucker · 2020-09-16T15:54:53Z

Great, the PR had been conflicting for a while so let's hope this gets tested in the next staging job.

gctucker · 2020-09-16T15:57:43Z

templates/preempt-rt/preempt-rt.jinja2

+      name: preempt-rt-prereq
+      path: inline/preempt-rt-prereq.yaml
+
+    - repository: https://github.com/igaw/test-definitions.git


We should be getting those test definitions from kernelci/test-definitions now, with the kernelci.org branch. Let's get a couple of staging jobs to run with this repo first but then we would need to get it changed before deploying in production.

I don't think we'll get any runs on staging because there are no rt-stable trees built in staging. Maybe you add rt-stable v5.4 at least temporarily?

Looks like I need to switch to the kernelci repo right now. Daniel submitted his stuff upstream and removed his preempt-rt branch.

Starting tree monitor jobs for some stable-rt build configs manually is fine for now, we shouldn't be building that all the time as it's a significant load. If we do want to keep testing real-time on staging, we should make a special build config i.e. kernelci_staging-rt or pull from stable-rt into the kernelci staging kernel branch.

test-configs.yaml

khilman · 2020-09-17T17:31:05Z

Updated to use test-definitions repo/branch from kernelci, and dropped rootfs pattern based on comments.

Results from manually submitted jobs: https://staging.kernelci.org/test/job/rt-stable_v5.4-rt/branch/HEAD/kernel/v5.4.61-rt37/plan/preempt-rt/

LAVA jobs:

original (odroid-n2): https://lava.baylibre.com/scheduler/job/117611#bottom
added (khadas-vim3l): https://lava.baylibre.com/scheduler/job/117612#bottom

gctucker · 2020-09-17T20:02:10Z

https://staging.kernelci.org/test/case/id/5f63999274b6645009581060/
9223372036854776000us lacenty? That's 292271 years :p

I don't know what went wrong, looks like an unsigned integer overflow. What is in the original LAVA results? That would help narrow down the problem, between LAVA, kernelci-backend or kernelci-frontend.

khilman · 2020-09-17T20:48:17Z

The results sent to the backend match the results from LAVA: https://lava.baylibre.com/scheduler/job/117611#L1293

So this looks like a cyclictest issue, not something related to the test-definition or this PR. IMO, we can merge this and then get rt-tests folks like Daniel Wagner to have a look.

gctucker · 2020-09-17T20:58:08Z

Great, thanks for confirming.

How about the item in the description to add parameters such as number of threads?

khilman · 2020-09-17T21:18:19Z

How about the item in the description to add parameters such as number of threads?

I meant for that to be a future work item as more platforms are added, not a prereq for this PR.

khilman · 2020-09-17T21:22:32Z

Here's some results from the khadas-vim3l (arm64: 2x A53): https://lava.baylibre.com/scheduler/job/118418#L1361

Those look more normal than the odroid-n2 (arm64: big.LITTLE 2x A53, 4x A72)

gctucker · 2020-09-21T15:24:05Z

Tested OK on staging: https://staging.kernelci.org/test/job/rt-stable_v5.4-rt/branch/HEAD/kernel/v5.4.61-rt37/plan/preempt-rt/

templates/preempt-rt/preempt-rt.jinja2

gctucker · 2020-09-21T15:29:06Z

templates/preempt-rt/preempt-rt.jinja2

+          description: Pre-requisites for PREEMPT_RT
+        run:
+          steps:
+          - apt-get update && apt-get install -y procps rt-tests


Looks like something to add to the rootfs image as a follow-up - either buster-rt if we want to keep adding test suites for real-time, if not this could be added to the basic buster one to avoid rootfs proliferation.

Yes, this can be done if/when creating a test-plan specific rootfs.

1) add config fragments for building RT-enabled kernels for the rt-stable tree and branches. 2) Add preempt_rt_variant for minimal set of builds. Based on minimal_variant, and adds preempt_rt fragment. 3) Add v4.19-rt and v5.4-rt branches Signed-off-by: Kevin Hilman <khilman@baylibre.com>

Add preempt-rt test-plan based on the Linaro test-definitions repo. Uses vanilla KernelCI debian buster NFS root filesystem, and installs rt-tests (and dependencies) via apt-get before running the test plan. Signed-off-by: Kevin Hilman <khilman@baylibre.com>

Signed-off-by: Kevin Hilman <khilman@baylibre.com>

khilman linked an issue Aug 25, 2020 that may be closed by this pull request

Real-Time test coverage #474

Closed

khilman force-pushed the dev/preempt-rt branch from 7b66a16 to 904c057 Compare August 25, 2020 21:21

khilman changed the title ~~build-configs: update RT preempt configurations~~ and build config and test-plan for PREEMPT_RT Aug 25, 2020

khilman mentioned this pull request Aug 25, 2020

Real-Time test coverage #474

Closed

khilman force-pushed the dev/preempt-rt branch 2 times, most recently from 6e05adc to 00489c0 Compare August 25, 2020 22:33

khilman changed the title ~~and build config and test-plan for PREEMPT_RT~~ add build config and test-plan for PREEMPT_RT Aug 25, 2020

gctucker requested a review from a team September 1, 2020 08:20

gctucker reviewed Sep 1, 2020

View reviewed changes

khilman force-pushed the dev/preempt-rt branch from 00489c0 to 67ea625 Compare September 16, 2020 15:34

gctucker reviewed Sep 16, 2020

View reviewed changes

test-configs.yaml Outdated Show resolved Hide resolved

khilman force-pushed the dev/preempt-rt branch from 67ea625 to 229750a Compare September 17, 2020 17:27

gctucker reviewed Sep 21, 2020

View reviewed changes

khilman added 3 commits September 22, 2020 09:00

preempt-rt: add lab-baylibre and khadas-vim3l

d19b9e0

Signed-off-by: Kevin Hilman <khilman@baylibre.com>

khilman force-pushed the dev/preempt-rt branch from 229750a to d19b9e0 Compare September 22, 2020 16:01

gctucker approved these changes Sep 23, 2020

View reviewed changes

gctucker merged commit 5fa4394 into kernelci:master Sep 23, 2020

khilman deleted the dev/preempt-rt branch January 18, 2023 18:09

Conversation

khilman commented Aug 25, 2020 • edited by gctucker Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

broonie commented Aug 25, 2020

Uh oh!

khilman commented Aug 25, 2020

Uh oh!

gctucker commented Aug 26, 2020

Uh oh!

gctucker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gctucker Sep 1, 2020

Choose a reason for hiding this comment

Uh oh!

khilman commented Sep 16, 2020

Uh oh!

gctucker commented Sep 16, 2020

Uh oh!

gctucker Sep 16, 2020

Choose a reason for hiding this comment

Uh oh!

khilman Sep 17, 2020

Choose a reason for hiding this comment

Uh oh!

khilman Sep 17, 2020

Choose a reason for hiding this comment

Uh oh!

gctucker Sep 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

khilman commented Sep 17, 2020

Uh oh!

gctucker commented Sep 17, 2020

Uh oh!

khilman commented Sep 17, 2020

Uh oh!

gctucker commented Sep 17, 2020

Uh oh!

khilman commented Sep 17, 2020

Uh oh!

khilman commented Sep 17, 2020

Uh oh!

gctucker commented Sep 21, 2020

Uh oh!

Uh oh!

gctucker Sep 21, 2020

Choose a reason for hiding this comment

Uh oh!

khilman Sep 22, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

khilman commented Aug 25, 2020 •

edited by gctucker

Loading

gctucker Sep 21, 2020 •

edited

Loading