test-configs.yaml: Enable additional preempt-rt tests #2397

igaw · 2024-02-19T14:55:25Z

While cyclictest is the classic preempt-rt test, cyclicdeadline and the rtla tests are also interesting to run.

Since these are all smoke test there is no point in running them too long. Thus reduce the runtime per test to one minute. This should keep the total preempt-rt runtime roughly in the same time frame.

jan-kiszka · 2024-02-19T15:06:37Z

@patersonc @aliceinwire, can CIP help out on this?

broonie · 2024-02-19T16:24:01Z

I've created a PR to add @igaw to the list of people who's PRs get automatically included in staging tests: kernelci/kernelci-deploy#132

igaw · 2024-02-20T07:29:58Z

Figured out how to resolve all the dependencies to run make test locally.

Also, I think it would make sense to figure out if we could run a bunch of different workloads. hackbench is mostly stressing the scheduler. E.g. I am running stress-ng in my lab as workload. But first, I try to figure out the basic stuff how configure kernelci.

igaw · 2024-02-20T07:36:14Z

Also one thing I would like to look into, is an artifact storage service like

https://archive.validation.linaro.org/

so that the rt-tests data can be analyzed after the run in more detail.

nuclearcat · 2024-02-20T08:45:47Z

Let me know please when PR is ready for testing on staging, as it will require some adjustments for staging patches.

igaw · 2024-02-20T09:37:40Z

@nuclearcat I am ready. I fixed all reports from make test and it should at least pass this level of checks.

nuclearcat · 2024-02-20T09:45:46Z

Thanks, i will do staging run in few hours, today (need to test few more pending PR as well).

nuclearcat · 2024-03-13T09:01:45Z

I am sorry for delay, last time i tried to run on staging, it seems something missing on staging specific to CIP instance. (ii tried just by adding one of trees in monitor).
@aliceinwire do you have experience on running CIP-specific tests?
I might try again this week.

igaw · 2024-03-13T10:17:39Z

FWIW, it's not just for cip test plan, this should be enabled for the rt-stable test plan too.

Do you have some logs, so I can look into?

nuclearcat · 2024-03-13T13:39:42Z

One of errors i spotted:


Traceback (most recent call last):
  File "/usr/local/bin/kci_test", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kernelci/scripts/kci_test.py", line 409, in main
    status = opts.command(configs, opts)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kernelci/scripts/kci_test.py", line 346, in __call__
    job = runtime.generate(
          ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kernelci/legacy/lava/__init__.py", line 84, in generate
    template = jinja2_env.get_template(short_template_file)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1010, in get_template
    return self._load_template(name, globals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 969, in _load_template
    template = self.loader.load(self, name, self.make_globals(globals))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/loaders.py", line 125, in load
    source, filename, uptodate = self.get_source(environment, name)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/loaders.py", line 204, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: preempt-rt-rtla-timerlat/generic-uboot-tftp-ramdisk-preempt-rt-rtla-timerlat-template.jinja2

igaw · 2024-03-13T15:00:25Z

Okay, I'll look into it.

igaw · 2024-03-20T09:33:56Z

I'm feeling a bit stupid. make check complains all over the place. I really thought I did test my submission. Well, I'm cleanup up the mess now and restructure some more while I am at it and make the preempt-rt tests a bit more flexible to configure. This will be useful in future.

Use a template for preempt-rt cyclictest. This is a preparation step to enable more tests from the rt-test suite. Signed-off-by: Daniel Wagner <wagi@monom.org>

igaw · 2024-03-20T11:05:14Z

Changes:

rebased
fixed all the warning reported by 'make check'
made it possible to provide different parameters for each rt-tests

Because some of the rt-tests are CPU/board depended the last point is kind of important. I haven't looked into the different board configurations available, just assumed that the current default work fine. That is every board tested has at least 2 CPUs.

broonie · 2024-03-21T12:33:53Z

We do currently schedule the preempt-rt tests on Beaglebone Black which is single processor, mainly just because there's lots of boards available in my lab so we can easily schedule tests there. I think it'd be fine to just not use that board so long as we've got something else providing the coverage, or to set up a nosmp variant which we select to run on single processor systems.

igaw · 2024-03-21T15:04:22Z

I think it makes sense to keep the Beaglebone Black in the loop. I am also using this board locally for testing, so good for reproducing results locally.

I haven't figured out yet how to per board type configurations are expressed. I stare a bit at the code and try to figure out what is there or to do it.

igaw · 2024-03-21T15:10:49Z

Alternatively, I could look into extending the test suite so it automatically selects all available CPUs. In the end we are talking about cyclictest, cyclicdeadline and signaltest. The rest of the test don't have any additional requirements. I think this makes more sense.

broonie · 2024-03-21T15:20:00Z

There's not really per board configurations at the minute as far as I'm aware, modulo the tests reading things from the running system and self configuring.

broonie · 2024-03-21T15:23:51Z

Support for automatic detection is probably going to be the easiest thing to get working here, yeah.

igaw · 2024-03-21T17:58:34Z

BTW, I haven't changed the defaults in this PR. cyclictest was and is still hard coded to use two threads.

Thus we don't have to wait for my PR for test-definition to land first. I'll update the kernelci configuration as soon as my test-definition is available in kernelci.

nuclearcat · 2024-03-25T10:52:43Z

After trying to generate tests:

+ kci_test generate --install-path=/data/workspace/bot.staging.kernelci.org/test-runner/workspace/test-runner/artifacts --runtime-json=/data/workspace/bot.staging.kernelci.org/test-runner/workspace/test-runner/artifacts/lab-broonie.json --storage=http://storage.staging.kernelci.org/ --runtime-config=lab-broonie --user=kernel-ci --runtime-token=**** --output=/data/workspace/bot.staging.kernelci.org/test-runner/workspace/test-runner/jobs/lab-broonie --callback-id=kernel-ci-callback-staging --callback-url=https://api.staging.kernelci.org/
Traceback (most recent call last):
  File "/usr/local/bin/kci_test", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kernelci/scripts/kci_test.py", line 409, in main
    status = opts.command(configs, opts)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kernelci/scripts/kci_test.py", line 346, in __call__
    job = runtime.generate(
          ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kernelci/legacy/lava/__init__.py", line 85, in generate
    data = template.render(params)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/etc/kernelci/lava/preempt-rt/generic-uboot-tftp-ramdisk-preempt-rt-template.jinja2", line 1, in top-level template code
    {% extends 'boot/generic-uboot-tftp-ramdisk-boot-template.jinja2' %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/etc/kernelci/lava/boot/generic-uboot-tftp-ramdisk-boot-template.jinja2", line 1, in top-level template code
    {% extends 'base/kernel-ci-base-tftp-deploy.jinja2' %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/etc/kernelci/lava/base/kernel-ci-base-tftp-deploy.jinja2", line 1, in top-level template code
    {% extends 'base/kernel-ci-base.jinja2' %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/etc/kernelci/lava/base/kernel-ci-base.jinja2", line 79, in top-level template code
    {% block actions %}{% endblock %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/etc/kernelci/lava/preempt-rt/generic-uboot-tftp-ramdisk-preempt-rt-template.jinja2", line 5, in block 'actions'
    {% include 'preempt-rt/preempt-rt.jinja2' %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/etc/kernelci/lava/preempt-rt/preempt-rt.jinja2", line 22, in template
    path: automated/linux/{{ tst_group | tst_cmd }}/{{ tst_cmd }}.yaml
^^^^^^^^^^^^^^^^^^^^^^^
jinja2.exceptions.TemplateAssertionError: No filter named 'tst_cmd'.

Allow to configure all parameters because these are board specific, e.g. how many CPUs are available. Thus, it doesn't make sense to hard code them. Furthermore, the various rt-tests (cyclictest, cyclicdeadline, pmqtest, ...) do have some common parameters but also unique ones. Signed-off-by: Daniel Wagner <wagi@monom.org>

Add the missing tests from the rt-tests suite. Signed-off-by: Daniel Wagner <wagi@monom.org>

While cyclictest is the classic preempt-rt test, cyclicdeadline and the rtla tests are also interesting to run. Since these are all smoke test there is no point in running them too long. Thus reduce the runtime per test to one minute. This should keep the total preempt-rt runtime roughly in the same time frame. Signed-off-by: Daniel Wagner <wagi@monom.org>

igaw · 2024-03-25T12:56:12Z

I've tried hard to get kci_test working locally but failed. Thus I tried to use a jinja2 template test and yamllint to figure out if the configuration gets rendered correctly. As it turns out there were some more typos. They should be fixed now.

Sorry for this back and forth. I'll try to get kernelci eventually running in my own lab, so that I can test everything locally.

igaw · 2024-03-25T13:00:29Z

plan: preempt-rt-cyclictest
job_timeout: 10
tst_cmd: 'cyclictest'
tst_group:

- test:
    name: preempt-rt-cyclictest
    timeout:
      minutes: 10

    definitions:
    - from: inline
      repository:
        metadata:
          format: Lava-Test Test Definition 1.0
          name: preempt-rt-prereq
          description: Pre-requisites for PREEMPT_RT
        run:
          steps:
          - apt-get update && apt-get install -y procps rt-tests
      name: preempt-rt-prereq
      path: inline/preempt-rt-prereq.yaml

    - repository: https://github.com/kernelci/test-definitions.git
      from: git
      revision: kernelci.org
      path: automated/linux/cyclictest/cyclictest.yaml
      name: preempt-rt-cyclictest
      parameters:
        ARTIFACTORIAL_TOKEN: 
        ARTIFACTORIAL_URL: 
        AFFINTIY: 0-1
        DURATION: 300s
        BACKGROUND_CMD: hackbench
      
        HISTOGRAM: 
        INTERVAL: 1000
        PRIORITY: 98
        THREADS: 2

igaw · 2024-03-25T13:05:20Z

The PR for LAVA has been merged. I am not really familiar how the LAVA test suite is intergrated into kernel-ci. Does it always run against the latest version or is there some sort of sync involved. Just asking if I should update this PR to use the new feature which lets cyclictest run on all CPUs and not just two.

igaw · 2024-04-12T15:32:45Z

@nuclearcat I've update the series and it's ready for the integration test. I really hope i got it right this time

igaw · 2024-05-03T11:50:52Z

@nuclearcat friendly ping.

igaw · 2024-06-04T12:15:03Z

Is there anything I can do to get this moving?

igaw force-pushed the preempt-rt branch from 64a4fb6 to d0c0d69 Compare February 20, 2024 07:25

nuclearcat added the staging-skip Don't test automatically on staging.kernelci.org label Feb 20, 2024

igaw force-pushed the preempt-rt branch 2 times, most recently from fde5e66 to 1ecc7a2 Compare February 20, 2024 09:36

igaw marked this pull request as ready for review February 20, 2024 09:37

nuclearcat removed the staging-skip Don't test automatically on staging.kernelci.org label Feb 21, 2024

igaw force-pushed the preempt-rt branch from 1ecc7a2 to f5e7842 Compare March 20, 2024 10:58

test-config.yaml: Use template for preempt-rt cyclictest

734114b

Use a template for preempt-rt cyclictest. This is a preparation step to enable more tests from the rt-test suite. Signed-off-by: Daniel Wagner <wagi@monom.org>

igaw force-pushed the preempt-rt branch from f5e7842 to e99a6df Compare March 20, 2024 11:02

igaw mentioned this pull request Mar 21, 2024

preempt-rt: add support to run a thread per cpu Linaro/test-definitions#503

Merged

igaw added 3 commits March 25, 2024 13:52

test-configs.yaml: Add test jobs for all rt-tests

6270729

Add the missing tests from the rt-tests suite. Signed-off-by: Daniel Wagner <wagi@monom.org>

igaw force-pushed the preempt-rt branch from e99a6df to d80a441 Compare March 25, 2024 12:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test-configs.yaml: Enable additional preempt-rt tests #2397

test-configs.yaml: Enable additional preempt-rt tests #2397

igaw commented Feb 19, 2024 •

edited

jan-kiszka commented Feb 19, 2024

broonie commented Feb 19, 2024

igaw commented Feb 20, 2024

igaw commented Feb 20, 2024

nuclearcat commented Feb 20, 2024

igaw commented Feb 20, 2024

nuclearcat commented Feb 20, 2024

nuclearcat commented Mar 13, 2024

igaw commented Mar 13, 2024 •

edited

nuclearcat commented Mar 13, 2024

igaw commented Mar 13, 2024

igaw commented Mar 20, 2024

igaw commented Mar 20, 2024 •

edited

broonie commented Mar 21, 2024

igaw commented Mar 21, 2024

igaw commented Mar 21, 2024

broonie commented Mar 21, 2024

broonie commented Mar 21, 2024

igaw commented Mar 21, 2024

nuclearcat commented Mar 25, 2024

igaw commented Mar 25, 2024 •

edited

igaw commented Mar 25, 2024

igaw commented Mar 25, 2024

igaw commented Apr 12, 2024

igaw commented May 3, 2024

igaw commented Jun 4, 2024

test-configs.yaml: Enable additional preempt-rt tests #2397

Are you sure you want to change the base?

test-configs.yaml: Enable additional preempt-rt tests #2397

Conversation

igaw commented Feb 19, 2024 • edited

jan-kiszka commented Feb 19, 2024

broonie commented Feb 19, 2024

igaw commented Feb 20, 2024

igaw commented Feb 20, 2024

nuclearcat commented Feb 20, 2024

igaw commented Feb 20, 2024

nuclearcat commented Feb 20, 2024

nuclearcat commented Mar 13, 2024

igaw commented Mar 13, 2024 • edited

nuclearcat commented Mar 13, 2024

igaw commented Mar 13, 2024

igaw commented Mar 20, 2024

igaw commented Mar 20, 2024 • edited

broonie commented Mar 21, 2024

igaw commented Mar 21, 2024

igaw commented Mar 21, 2024

broonie commented Mar 21, 2024

broonie commented Mar 21, 2024

igaw commented Mar 21, 2024

nuclearcat commented Mar 25, 2024

igaw commented Mar 25, 2024 • edited

igaw commented Mar 25, 2024

igaw commented Mar 25, 2024

igaw commented Apr 12, 2024

igaw commented May 3, 2024

igaw commented Jun 4, 2024

igaw commented Feb 19, 2024 •

edited

igaw commented Mar 13, 2024 •

edited

igaw commented Mar 20, 2024 •

edited

igaw commented Mar 25, 2024 •

edited