YAML configuration design for new Pipeline

[gctucker]

Along with the new KernelCI API & Pipeline currently under active development, the YAML pipeline configuration is also going through a redesign. This discussion is to go through the proposals and collect feedback before we can start working on a final implementation and documentation.

Here's some related documentation to provide some context:

• Current YAML configuration: https://kernelci.org/docs/core/config/
• New API & Pipeline overview: https://kernelci.org/docs/api/

The new kci config tool should also provide ways to manipulate the YAML configuration, for example to check which jobs would be run upon receiving a particular event or to list things in a more human-readable way.

Then ideally some schema should be created using e.g. pydantic for the YAML configuration objects, which could also serve as a basis for documentation. We should also have a version for this schema and remove the YAML configuration from the code in kernelci-core to actually make it user application data.

See also the email thread: [RFC] KernelCI YAML pipeline configuration

[gctucker]

APIs

These config entries are about where the new API instances can be found. While a lot of things can be done offline such as running jobs locally, the main use-case is to use the API to store / retrieve data and send / receive events via the Pub-Sub interface. It relies on API tokens which are managed as secrets so not in the YAML configuration file. So right now, we just need the URL. For example:

APIs:

  docker-host:
    url: http://172.17.0.1:8001

  staging.kernelci.org:
    url: https://staging.kernelci.org:9000

I believe it's a bit unusual to have uppercase letters in YAML tags, but apis doesn't look good and api_configs is not great either. It would be good to know if people are generally OK with APIs or if someone wants to suggest something else.

Note: Unlike the current production system, only one type of API is now supported. This is a design decision to simplify client-side code and draw the line for abstracting the backend at the API level. So in theory, if someone wanted to use a different API implementation, it would just work as long as the HTTP entry points were compatible. It's also possible to specify the API version, which could be used to implement extensions.

[gctucker]

There hasn't been any other suggestions or feedback on this so far, and it's very unusual to have capital letters in YAML keys. So I think api: will be fine in this case even though we can have several. It's more like the name of the section, just like we have storage:.

[gctucker]

storage

The new API doesn't provide any built-in storage solution, instead it just requires all artifacts to be available via an http:// or https:// URL. So any arbitrary storage solution can be used, by default the docker-compose setup comes with an SSH container for uploading files and an Nginx container for serving them using a shared Docker volume. A production system might have an S3-compatible storage or anything that provides a way to retrieve the artifacts over HTTP.

This is where the "storage" YAML configuration comes into play, with a "type" attribute to specify which kind of storage this is. The type is then used to look up an implementation module based on an abstract interface Storage class.

storage:

  docker-host:
    type: ssh
    host: 172.17.0.1
    port: 8022
    base_url: http://172.17.0.1:8002/

  staging.kernelci.org:
    type: ssh
    host: staging.kernelci.org
    port: 9022
    base_url: http://staging.kernelci.org:9080/

[nuclearcat]

Maybe we need some options to specify authentication type? Where to search for key (file, agent socket, etc), or maybe key is embedded in "secret" variable.
For S3 as i recall token also required.

[gctucker]

Yes there's already a way to specify credentials separately from the YAML config as we can't keep secrets directly there. But it's true we could have something to know the name of the secret, like credentials. One thing to note though, is that different users might be using the same storage config from the same YAML file but with different credentials, so it might not really work. It's the same thing as with the API tokens or Kubernetes any kind of authentication in fact.

[gctucker]

jobs

A "job" is essentially anything that can be run. This is a lot more generic than the test and build configurations used with the current Jenkins pipeline. There are still a few assumptions about "jobs" though as they typically come with a Jinja2 template to generate a job definition. This is in principle independent of the runtime environment, so a same template can be used to run the job in a local shell, in a Docker container, in Kubernetes, in LAVA etc. It will do this by including a different base template that's specific to the chosen runtime. Jobs are also expected to require an "image" name which can be pretty much anything that's usable by the runtime environment, typically a Docker image or a root file system.

Another difference with the current Jenkins pipeline is that jobs have explicit dependencies on other jobs. For example, a KUnit job or a kernel build job will require a source code tarball. Then a kselftest job will require a particular kernel build. This hinges on the Pub/Sub interface: every time a job node changes state an event is sent. Then a pipeline service orchestrating the jobs will look for jobs that have requirements that match the event to run them. This is what the run_on attribute is for.

Note: Jobs can also have arbitrary parameters that are passed directly to the template engine via the params configuration attribute. See also runtimes and platforms in this discussion.

# Not directly loaded into the config, only used for YAML aliases in this file
_anchors:

  checkout: &checkout-node
    - channel: node
      name: checkout
      result: pass

  kbuild: &kbuild-node
    - channel: node
      name: kbuild
      result: pass


jobs:

  baseline:
    template: baseline.jinja2
    run_on: *kbuild

  checkout:
    template: checkout.jinja2
    image: kernelci/kernelci
    run_on:
      - channel: trigger
        name: kernel-tree

  kbuild-gcc-10-arm: &gcc-10-arm
    template: kbuild.jinja2
    name: kbuild
    image: kernelci/gcc-10:arm-kernelci
    run_on: *checkout-node
    params:
      arch: arm64
      compiler: gcc-10
      defconfig: '*_defconfig'

  kbuild-gcc-10-arm-kselftest:
    <<: *gcc-10-arm
    params:
      arch: arm64
      compiler: gcc-10
      defconfig: multi_v7_defconfig
      fragments: [kselftest]

  kbuild-gcc-10-x86:
    template: kbuild.jinja2
    name: kbuild
    image: kernelci/gcc-10:x86-kernelci
    run_on: *checkout-node
    params:
      arch: x86_64
      compiler: gcc-10
      defconfig:
        - x86_64_defconfig
        - allmodconfig

  kselftest-landlock:
    template: kselftest.jinja2
    run_on:
      <<: *kbuild-node
      data.defconfig: kselftest

  kunit: &kunit
    template: kunit.jinja2
    name: kunit
    image: kernelci/gcc-10:x86-kunit-kernelci
    run_on: *checkout-node

  kunit-x86_64:
    <<: *kunit
    params:
      arch: x86_64

[gctucker]

A follow-up from this could be to have nested jobs, for example to define steps as part of a single job but with different entries in the database. This could be used for grouping tests in a same job, or breaking down jobs into sub-sections directly in the YAML configuration rather than at the job implementation level. From a YAML point of view, I guess this could be done simply by adding a jobs attribute and recursively have the same job schema inside, for example:

jobs:
  hello:
    template: hello.jinja2
    run_on: *kbuild
    jobs:
      foo:
        image: something/foo
        params:
          key: value
      bar:
        image: something/bar
        params:
          key: value

Then he sub-jobs would also be passed to the parent template, typically this could be rendered as a multi-stage job in LAVA (several boots on the same platform) or multiple containers in Kubernetes. The run_on attribute might not make sense in this particular case, but that's up for discussion.

[JenySadadia]

How about renaming run_on to runs_on? It seems more suitable to me.

[gctucker]

How about renaming run_on to runs_on? It seems more suitable to me.

run_on means it's the intention, like "please run on the occurrence of this event". It still needs to be implemented by something, and it might not actually be run if the implementation decides to discard it for any reason. With runs_on I think it implies it's guaranteed to always run and kind of looks like a built-in feature that happens implicitly.

[gctucker]

I think moving run_on to the scheduler config would make more sense, also having a single entry rather than a list since the scheduler config is already a list. For example:

jobs:
  kbuild-gcc-10-x86:
    template: kbuild.jinja2
    image: kernelci/staging-gcc-10:x86-kselftest-kernelci
    params:
      arch: x86_64
      compiler: gcc-10
      defconfig: x86_64_defconfig

scheduler:
  - job: kbuild-gcc-10-x86
    runtime:
      type: kubernetes
    run_on:
      channel: node
      name: checkout
      state: available

[gctucker]

Implemented by #1974

[gctucker]

runtimes

Jobs are run in runtime environments, or "runtimes". Currently, the only runtimes implemented are a local shell directly on the host, Docker containers and Kubernetes. The next ones should include LAVA and other hardware labs such as LabGrid or custom ones that don't rely on common frameworks. Like the storage configurations, runtimes have a type to look up a matching implementation module. The configuration attributes are therefore also specific to the runtime type. Then also like storage, runtime implementations should be based on the abstract Runtime interface to provide standard methods for generating jobs and running them.

Like jobs, runtimes can have arbitrary parameters passed to the template engine via the params configuration attribute.

Note: Jobs are actually run on a platform in a runtime environment. Platforms have their own configuration entries separate from the runtimes, which is covered under the "platforms" section.

runtimes:

  docker:
    type: docker
    env_file: .docker-env
    volumes:
      - '/home/user/kernelci/data:/home/kernelci/data'  # example

  lava-broonie:
    type: lava
    url: https://lava.sirena.org.uk/RPC2/
    job_priority:
      min: 0
      max: 40
    queue_timeout:
      days: 1

  k8s-gke-eu-west4:
    type: kubernetes
    context: gke_android-kernelci-external_europe-west4-c_kci-eu-west4
    params:
      spec:
        ttlSecondsAfterFinished: 30
        template:
          spec:
            tolerations:
              - key: "kubernetes.azure.com/scalesetpriority"
                operator: "Equal"
                value: "spot"
                effect: "NoSchedule"

  shell:
    type: shell

[gctucker]

platforms

Runtime environments often provide a variety of platforms which can be used to, run jobs. For example, Kubernetes clusters may provide different kinds of pods, and hardware test labs may provide different types of boards or servers. This, is covered by the platform configurations.

Each platform is tied to a runtime environment type so that the services, running jobs can tell which ones need to be submitted for which runtime, environment. Then specific fields can be provided depending on the platform's, associated runtime type. For example, hardware platforms have a particular CPU, architecture that the runtime implementation needs to use to determine whether, it's compatible with a particular kernel build.

platforms:
  kubernetes:
    runtime-type: kubernetes

  bcm2837-rpi-3-b:
    runtime-type: lava
    arch: arm64
    boot_method: uboot
    oem: broadcom
    params:
      context:
        console_device: ttyS1
        test_character_delay: 10

  qemu_x86_64:
    runtime-type: lava
    name: qemu
    arch: x86_64
    boot_method: qemu
    oem: qemu
    params:
      context:
        arch: x86_64
        cpu: qemu64
        guestfs_interface: ide

  shell:
    runtime-type: shell

  shell-dash:
    runtime-type: shell
    name: shell
    params:
      shebang: /bin/dash

[gctucker]

scheduler

Finally, a list of scheduler combinations is used to tie things together. This is like a switchboard to connect jobs, runtimes and platforms. As such, each entry is anonymous so it takes the form of a list of dictionaries. The initial proposal is to have at least a job and runtime defined in each scheduler entry, but we might need to refine it with additional criteria. The idea being that job, runtime and platform configurations should be self-contained and all the "wiring" between them is done in the scheduler.

Note: The current YAML configuration has the concept of filters. So far they have been ignored in this proposal as it's a better if the design explicitly describes the pipeline. For the scheduler, it might make sense to reintroduce them in potentially a simplified form and with a well-defined set of filter keys.

scheduler:
  - job: kunit
    runtime:
      name: k8s-gke-eu-west4

  - job: kbuild-gcc-10-x86
    runtime:
      type: kubernetes

  - job: baseline
    runtime:
      type: lava

  - job: ltp-crypto
    runtime:
      type: lava
    platforms:
      - bcm2837-rpi-3-b

  - job: v4l2-compliance
    runtime:
      name: lava-lkft
    platforms:
      - bcm2837-rpi-3-b