test-configs.yaml: Enable additional preempt-rt tests

[igaw]

While cyclictest is the classic preempt-rt test, cyclicdeadline and the rtla tests are also interesting to run.

Since these are all smoke test there is no point in running them too long. Thus reduce the runtime per test to one minute. This should keep the total preempt-rt runtime roughly in the same time frame.

[jan-kiszka]

@patersonc @aliceinwire, can CIP help out on this?

[broonie]

I've created a PR to add @igaw to the list of people who's PRs get automatically included in staging tests: kernelci/kernelci-deploy#132

[igaw]

Figured out how to resolve all the dependencies to run make test locally.

Also, I think it would make sense to figure out if we could run a bunch of different workloads. hackbench is mostly stressing the scheduler. E.g. I am running stress-ng in my lab as workload. But first, I try to figure out the basic stuff how configure kernelci.

[igaw]

Also one thing I would like to look into, is an artifact storage service like

https://archive.validation.linaro.org/

so that the rt-tests data can be analyzed after the run in more detail.

[nuclearcat]

Let me know please when PR is ready for testing on staging, as it will require some adjustments for staging patches.

[igaw]

@nuclearcat I am ready. I fixed all reports from make test and it should at least pass this level of checks.

[nuclearcat]

Thanks, i will do staging run in few hours, today (need to test few more pending PR as well).

[nuclearcat]

I am sorry for delay, last time i tried to run on staging, it seems something missing on staging specific to CIP instance. (ii tried just by adding one of trees in monitor).
@aliceinwire do you have experience on running CIP-specific tests?
I might try again this week.

[igaw]

FWIW, it's not just for cip test plan, this should be enabled for the rt-stable test plan too.

Do you have some logs, so I can look into?

[nuclearcat]

One of errors i spotted:

Traceback (most recent call last):
  File "/usr/local/bin/kci_test", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kernelci/scripts/kci_test.py", line 409, in main
    status = opts.command(configs, opts)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kernelci/scripts/kci_test.py", line 346, in __call__
    job = runtime.generate(
          ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kernelci/legacy/lava/__init__.py", line 84, in generate
    template = jinja2_env.get_template(short_template_file)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1010, in get_template
    return self._load_template(name, globals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 969, in _load_template
    template = self.loader.load(self, name, self.make_globals(globals))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/loaders.py", line 125, in load
    source, filename, uptodate = self.get_source(environment, name)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/loaders.py", line 204, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: preempt-rt-rtla-timerlat/generic-uboot-tftp-ramdisk-preempt-rt-rtla-timerlat-template.jinja2

[igaw]

Okay, I'll look into it.

[igaw]

I'm feeling a bit stupid. make check complains all over the place. I really thought I did test my submission. Well, I'm cleanup up the mess now and restructure some more while I am at it and make the preempt-rt tests a bit more flexible to configure. This will be useful in future.

[igaw]

Changes:

• rebased
• fixed all the warning reported by 'make check'
• made it possible to provide different parameters for each rt-tests

Because some of the rt-tests are CPU/board depended the last point is kind of important. I haven't looked into the different board configurations available, just assumed that the current default work fine. That is every board tested has at least 2 CPUs.

[broonie]

We do currently schedule the preempt-rt tests on Beaglebone Black which is single processor, mainly just because there's lots of boards available in my lab so we can easily schedule tests there. I think it'd be fine to just not use that board so long as we've got something else providing the coverage, or to set up a nosmp variant which we select to run on single processor systems.

[igaw]

I think it makes sense to keep the Beaglebone Black in the loop. I am also using this board locally for testing, so good for reproducing results locally.

I haven't figured out yet how to per board type configurations are expressed. I stare a bit at the code and try to figure out what is there or to do it.

[igaw]

Alternatively, I could look into extending the test suite so it automatically selects all available CPUs. In the end we are talking about cyclictest, cyclicdeadline and signaltest. The rest of the test don't have any additional requirements. I think this makes more sense.

[broonie]

There's not really per board configurations at the minute as far as I'm aware, modulo the tests reading things from the running system and self configuring.

[broonie]

Support for automatic detection is probably going to be the easiest thing to get working here, yeah.

[igaw]

BTW, I haven't changed the defaults in this PR. cyclictest was and is still hard coded to use two threads.

Thus we don't have to wait for my PR for test-definition to land first. I'll update the kernelci configuration as soon as my test-definition is available in kernelci.

[nuclearcat]

After trying to generate tests:

+ kci_test generate --install-path=/data/workspace/bot.staging.kernelci.org/test-runner/workspace/test-runner/artifacts --runtime-json=/data/workspace/bot.staging.kernelci.org/test-runner/workspace/test-runner/artifacts/lab-broonie.json --storage=http://storage.staging.kernelci.org/ --runtime-config=lab-broonie --user=kernel-ci --runtime-token=**** --output=/data/workspace/bot.staging.kernelci.org/test-runner/workspace/test-runner/jobs/lab-broonie --callback-id=kernel-ci-callback-staging --callback-url=https://api.staging.kernelci.org/
Traceback (most recent call last):
  File "/usr/local/bin/kci_test", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kernelci/scripts/kci_test.py", line 409, in main
    status = opts.command(configs, opts)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kernelci/scripts/kci_test.py", line 346, in __call__
    job = runtime.generate(
          ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kernelci/legacy/lava/__init__.py", line 85, in generate
    data = template.render(params)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/etc/kernelci/lava/preempt-rt/generic-uboot-tftp-ramdisk-preempt-rt-template.jinja2", line 1, in top-level template code
    {% extends 'boot/generic-uboot-tftp-ramdisk-boot-template.jinja2' %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/etc/kernelci/lava/boot/generic-uboot-tftp-ramdisk-boot-template.jinja2", line 1, in top-level template code
    {% extends 'base/kernel-ci-base-tftp-deploy.jinja2' %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/etc/kernelci/lava/base/kernel-ci-base-tftp-deploy.jinja2", line 1, in top-level template code
    {% extends 'base/kernel-ci-base.jinja2' %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/etc/kernelci/lava/base/kernel-ci-base.jinja2", line 79, in top-level template code
    {% block actions %}{% endblock %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/etc/kernelci/lava/preempt-rt/generic-uboot-tftp-ramdisk-preempt-rt-template.jinja2", line 5, in block 'actions'
    {% include 'preempt-rt/preempt-rt.jinja2' %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/etc/kernelci/lava/preempt-rt/preempt-rt.jinja2", line 22, in template
    path: automated/linux/{{ tst_group | tst_cmd }}/{{ tst_cmd }}.yaml
^^^^^^^^^^^^^^^^^^^^^^^
jinja2.exceptions.TemplateAssertionError: No filter named 'tst_cmd'.

[nuclearcat]

As referenced in PR we are creating -RT testing support in new system, called Maestro. Probably it is completed by now, feel free to open PR or issue in kernelci-pipeline repository if you want to send any updates.
Legacy system is not being updated, so i am closing this PR.

[igaw]

Is the new dashboard already online, to see if the RT builds/tests are running?

[igaw]

Nevermind, found the docs to it: https://docs.kernelci.org/maestro/api/early-access/

[nuclearcat]

We have also internal tools to view results:
Staging: https://staging.kernelci.org:9000/viewer
Production: https://kernelci-api.westus3.cloudapp.azure.com/viewer
I will make sure also today to merge remaining -rt related patches, if that will be possible. I will reopen PR meanwhile to keep discussion.

[padovan]

@igaw you can also see the results on the WIP grafana dashboard from Helen Koike: https://grafana.kernelci.org/d/OKXc44EIz/wip3a-koike?orgId=1&var-origin=maestro&var-tree=stable-rt&var-branch=All&var-test_path_regex=kernelci_baseline&var-platform=%25&var-config=%25&var-datasource=default

[igaw]

I've shared the dashboard with my fellow stable-rt maintainers and we played a bit around. It seems everyone like the direction this project takes.

Anyway, one remark was that there was no rt config used for the builds, just the default ones. Which is good as it already found a bug in one of our rt patch series. So the question is if it would be possible to enable the rt configs back as well? Thanks!

[musamaanjum]

Thank you so much for reporting.

The preempt_rt config builds are already enabled. There is some problem on the dashboard side and those results are missing. I'll investigate.

[padovan]

@igaw thanks for the valuable feedback! Let's also setup email notifications for your trees and tests (cc @JenySadadia)

[igaw]

For the email notification, could these emails send to stable-rt@vger.kernel.org too? We use this mailing list for stable work, so it would be the right place to post any build results etc. Thanks!

[JenySadadia]

Hello @igaw
For starters, I'll start working on reporting build failures and rt-tests failures in the email notification.
Is there any other test failure would you like to see in the report? Such as boot failures?

[igaw]

Hi @JenySadadia, cool! Yes, boot failures would also be good to get.

[JenySadadia]

Hi @JenySadadia, cool! Yes, boot failures would also be good to get.

Okay, thanks.

[nuclearcat]

Opened issue #2643
As it's better to continue discussion there and close PR that is no more relevant. Thank you!