Userspace builds without docker server (buildah)

[distorhead]

As a werf user I want to have an ability to run werf builds in Kubernetes itself. To run werf builds in Kubernetes werf should have a mode to work without docker server (as https://github.com/GoogleContainerTools/kaniko does).

[donbeave]

Maybe by adding buildpacks support?

https://buildpacks.io/

[tumauwt]

Is this on the radar for future support?

[distorhead]

Currently implementing first version of werf with buildah included to build only dockerfiles in this PR: #3735

TODO check list:

• Check werf binaries ldd dependencies;
  • Try to build werf with buildah and cgo enabled as statical binary
• Try to run werf-build in native-rootless buildah mode in Kubernetes.
• buildah-login should work for docker-with-fuse buildah mode.
• Investigate ci-env + buildah-mode login issue.
• Fix native-rootless buildah mode initialization procedure for werf.
• Debug native-rootless buildah mode in werf-build.
• Performance testing.
• E2E testing.
• Temporarily forbid usage of :local stages storage (werf-build command without --repo arg) with buildah container runtime.

[ilya-lesikov]

Current performance testing results for building sample Dockerfile are:

• Linux:
  • old Docker builder: 516 sec
  • new Buildah native builder: 481 sec
  • new Buildah non-native builder: 515 sec
• macOS:
  • old Docker builder: 242 sec
  • new Buildah non-native builder: 347 sec
• Windows
  • old Docker builder: 385 sec
  • new Buildah non-native builder: 434 sec

We are continuing to work on it.

[develar]

Will be Stapel covered or it is another issue? (ability to werf build without running Docker).

[ilya-lesikov]

@develar Yes, it will be, but after implementing Dockerfile builder with Buildah.

[distorhead]

Status update

Werf currently supports building of Dockerfile-based images using buildah container runtime by setting WERF_CONTAINER_RUNTIME_BUILDAH=auto|native-rootless|docker-with-fuse.

Next main steps:

1. Run werf+buildah in Kubernetes (for Dockerfiles only): Running werf in buildah mode in containers and Kubernetes #3933
2. Add buildah support for stapel images builder: Buildah support for stapel builder #3936

Upd.

Corrected initial error: not WERF_BUILDAH_CONTAINER_RUNTIME, but WERF_CONTAINER_RUNTIME_BUILDAH environment variable.

[ilya-lesikov]

Current progress

Now werf can be run in Kubernetes or in standalone containers. We provide preconfigured images for this. Only available for Dockerfile builds for now, next step is to allow stapel builds with Buildah, thus allowing to run stapel builds in containers/Kubernetes too.

What's done and lands soon in all release channels up to stable:

• Now WERF_BUILDAH_MODE env var used instead of WERF_CONTAINER_RUNTIME_BUILDAN to set Buidah mode/isolation. WERF_BUILDAH_MODE="auto" is enough to enable builds with Buildah, without Docker.
• native-chroot Buildah mode is the only mode available in containers.
• Only Linux: native-rootless (oci) mode now used by the default if not in a container (previously native-chroot). For macOS/Windows the same docker-with-fuse mode is used.
• VFS storage driver available in addition to OverlayFS. Any OS, any mode, containerized or not. To enable: WERF_BUILDAH_STORAGE_DRIVER=vfs
• Preconfigured images with werf provided to run werf in containers/Kubernetes. Based on Alpine, Ubuntu and CentOS. ghcr.io/werf/werf:1.2-(alpha|beta|ea|stable)-(alpine|ubuntu|centos)
• If native kernel OverlayFS support is available it is used automatically. If not hen fuse-overlayfs will be used.
• Now you can run werf in unprivileged containers/Pods.
• Documentation added to explain how to run werf in Kubernetes/containers and how to integrate it with your CI.

Next

• Investigate ci-env with buildah login #3935
• Buildah support for stapel builder #3936

[shurup]

#4146 sheds some light on why buildah was chosen for werf instead of some other alternatives available (e.g. kaniko).

[distorhead]

Update. In this update we will refresh plans of implementing buildah.

Full support of buildah will be introduced eventually, and includes:

• Buildah is the main container backend both for CI/CD usage and local development.
• Buildah backend is crossplatform, on Windows and MacOS platforms it could work through virtualization layer.
• werf commands support:
  • Delivery commands (main and secondary):
    • werf build
    • werf converge
    • werf bundle publish
    • werf render
    • werf export
    • werf kube-run
  • Cleaning commands:
    • werf cleanup
    • werf purge
    • werf host cleanup
  • Other commands:
    • werf ci-env
    • werf compose
• Not supported commands:
  • werf run
• Local development without --repo param is supported.

Current buildah support:

• Building of Dockerfiles with buildah.
• Only repo stages storage supported (it means --repo CONTAINER_REGISTRY_REPO param is mandatory to use buildah).
• Buildah working through docker-server on MacOS and Windows platforms.

Next stage of buildah support will include:

• Support of building Dockerfiles and stapel images with buildah.
• Only repo stages storage supported (it means --repo CONTAINER_REGISTRY_REPO param is mandatory to use buildah).
• Buildah working through docker-server on MacOS and Windows platforms.
• All main werf commands work with buildah backend.

See progress of this stage in the issue: #3936 (comment).

[alexey-igrychev]

Solved. The user can build images using Stapel and Dockerfile syntax with Docker or Buildah backends 🎉