Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kci_docker #1320

Merged
merged 12 commits into from
Aug 1, 2022
Merged

kci_docker #1320

merged 12 commits into from
Aug 1, 2022

Conversation

gctucker
Copy link
Contributor

Add a kci_docker tool to produce a new set of Docker images using Jinja2 templates. This is needed in order to provide the flexibility required when running builds and tests in Kubernetes, and to facilitate producing images with special combinations of tools for local development.

See the README on the incoming branch: https://github.com/gctucker/kernelci-core/tree/kci_docker/config/docker-new#readme

The tree of files looks like this:

docker-new
├── base
│   ├── clang.jinja2
│   ├── debian.jinja2
│   ├── host-tools.jinja2
│   └── python.jinja2
├── clang-11.jinja2
├── clang-12.jinja2
├── clang-13.jinja2
├── clang-14.jinja2
├── clang-15.jinja2
├── fragment
│   ├── azure-cli.jinja2
│   ├── gcloud.jinja2
│   ├── kernelci.jinja2
│   ├── kselftest.jinja2
│   ├── kubernetes.jinja2
│   └── kunit.jinja2
├── gcc-10-arc.jinja2
├── gcc-10-arm64.jinja2
├── gcc-10-arm.jinja2
├── gcc-10-armv5.jinja2
├── gcc-10-mips.jinja2
├── gcc-10-riscv64.jinja2
├── gcc-10-x86.jinja2
├── k8s.jinja2
├── kci_docker
├── kernelci.jinja2
├── README.md
└── requirements.txt

Add a new kci_docker tool to generate Dockerfiles from Jinja2
templates and build images.  This adds the possibility to reuse
fragments of Dockerfiles and combine them to create a set of images,
typically to have a basic set of toolchain images and then also add
dependencies for kselftest and the kernelci Python package.

Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add the base/debian.jinja2 template to create images with Debian
Bullseye as well as the common list of host tools for kernel builds.
Also add all the gcc-10 templates for each supported architecture.

Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add a base/clang.jinja2 template for the common packages required by
Clang as well as a collection of Clang versions from 11 to 15 (oldest
supported in mainline Linux to latest release).

Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add a fragments block in base/debian.jinja2 to iterate over a list of
fragment names and include them.  It sets is_debian so fragment
templates can adjust accordingly and also shares a sub_arch variable
which the base templates can set (typically for cross-compiler
toolchains).

Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Set the sub_arch variable in all the ARM templates for gcc-10.

Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add a kselftest.jinja2 fragment to add kselftest build dependencies to
any base toolchain image.

Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add a kunit.jinja2 fragment to add the KUnit build dependencies for an
x86 base toolchain image.  As KUnit results should not vary with the
CPU architecture and it's normally running in user-mode linux, only
x86 is supported by this fragment for now.

Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add a kernelci.jinja2 base template as well as a fragment to either
create a plain Python image with the kernelci package or add it on top
of another Debian base image.

Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add a k8s.jinja2 template with a set of fragments to install all the
Kubernetes tools for GCE and Azure.

Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add requirements.txt with the list of requirements to use kci_docker,
which should only be the Docker Python SDK and Jinja2.

Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add initial README.md file to explain how to use kci_docker.

Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add --build-arg command line argument to kci_docker to work in the
same way as the 'docker build' command, to pass an arbitrary number of
key-valye build argument pairs.

Also add a note in the README.md with an example.

Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
@gctucker gctucker marked this pull request as ready for review July 29, 2022 06:55
@gctucker
Copy link
Contributor Author

Things to do before this can replace the current docker directory (as follow-up PRs):

  • add missing images: qemu, debos, cros-*
  • add support for building with caching disabled
  • update automated build & push to use the new image names
  • update Jenkins to use new naming convention

@gctucker gctucker mentioned this pull request Aug 1, 2022
10 tasks
@gctucker gctucker linked an issue Aug 1, 2022 that may be closed by this pull request
10 tasks
Copy link
Contributor Author

@gctucker gctucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested OK on staging with new pipeline and KUnit (gcc-10 x86).

The full set of images will be tested when the other parts of the code have been updated to use the new images on staging.

@gctucker gctucker merged commit df6106e into kernelci:main Aug 1, 2022
KernelCI project board automation moved this from In Progress to Done Aug 1, 2022
@gctucker gctucker deleted the kci_docker branch August 1, 2022 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

kci_docker
1 participant