Stateless engine or stateful engine?

[shykes]

Question

I want to start a discussion on an important aspect of Dagger's architecture: whether to make the engine stateful or stateless. This is a complex topic with important ramifications, that affect users in very tangible ways.

This has been a known unresolved question for a while, which we collectively shelved to deal with more urgent things. But I think now is the time to unshelve it, because more organizations are now running, or looking to run, Dagger on their CI infrastructure. As a result they face new questions:

1. What is the architecture? What are the components, where do they run, how are they installed?
2. What is the lifecycle? What components need to be updated when, and what are the dependencies between them?
3. Who's in charge? When self-hosted CI runners are involved, Dagger touches the CI platform (eg. Github Actions), and the underlying compute platform (eg. Kubernetes). These platforms are often managed by different teams, sometimes in completely different orgs, with very different priorities and timelines. Who ultimately is in charge of installing Dagger in CI? Are they the same people who experience the value of Dagger? If not, how to make harmonious collaboration between those two teams possible?

At the moment it is very difficult to answer these questions clearly and definitively, because our architecture is not clear, and not yet defined. For various pragmatic engineering reasons, the engine is split in two parts:

1. A stateless frontend that lives in the dagger CLI. This includes both the user-facing interface and GraphQL API.
2. A stateful backend that lives in a container. This includes our custom build of buildkit, containerd and associated tooling

These two pieces must eventually be reunited - we have strong consensus on that. The unresolved question is where?

• Does the entire engine become stateless and live in the dagger CLI + purely ephemeral companion containers spawned as needed?
• Or does it become stateful and move entirely into a long-running OCI container?

This is a simplified question designed to provoke conversation. I believe both options are possible, and both come with a long list of difficult tradeoffs and nuance. But, let's start simple for now :) What does everyone think?

Answer

After much discussion (see comments below), we have reached the following tentative answer:

Where we want to go

• We should move towards making the engine stateless and ephemeral
• The benefits are 1) it simplifies the versioning matrix, 2) it simplifies the operational model, 3) it makes it easier to adopt Dagger.

How to get there

• We should start moving towards this architecture in incremental steps
• We should take the first step as soon as possible
• The first step is detailed by @sipsma as follows:

I actually think we can+should jump immediately to doing this such that the changes are to our engine image, rather than the k8s daemonset, and thus benefit every user rather than just k8s users.

The outline being (in approximate order of implementation steps, can be done iteratively rather than all at once):

Switch our engine's executor backend from oci worker -> containerd worker
This means we have to include containerd in our engine image, which will increase the size a bit, but highly unlikely to be so much it's a real concern.
Also containerd may be a preferable backend to use anyways since we could benefit from the (small but growing) ecosystem of containerd runtimes.
Change the engine image's entrypoint to no longer be the dagger engine but instead to be containerd
When the image starts, it will start the engine as a container in containerd w/ the containerd socket mounted in.
Note that since containerd is the engine's backend, this change does NOT result in crazy container-in-container-in-container behavior; any containers created by the engine will actually be "sibling" containers to it in containerd. So it's the same level of nestedness as today.
Add support for clients (i.e. the CLI) to optionally start older/newer versions of the engine in containerd rather than just one single version
We'd likely want to do something like include the latest version of the engine in the image and then require that any other engine images be pulled at runtime, which feels like a reasonable tradeoff in terms of functionality and not bloating the image. But there's many other variations possible here too.
This is also the point where we'll need to start considering making the engine containers more short-running, which would make them available for cleanup and thus prevent versions from accumulating indefinitely.
That would technically just be a start to the overall grand plan, but it's extremely appealing in that

It's surprisingly low-hanging fruit; probably somewhere between medium and large in terms of t-shirt sized effort
It would dramatically cut back on how often version mismatches between CLI<->Runner happen, likely to the point of those being very rare events.

Packaging and naming considerations

One important detail to consider: this will determine product packaging, as in: what are the parts of our product and what do our users call them?

I think we reserve “engine” for the upper part, soon to be fully orchestrated by the CLI, soon the be stateless.
We need a new name for the lower part: the substrate that acts as privileged container runner + data plumbing.
Both need to be distributed as OCI images, at least until we can fully split buildkit in higher/upper parts (so: for a good while)
Consequently we may need to distribute 2 distinct OCI images? One called “engine” (upper) and one called maybe “runner” for the lower part / substrate? Would be consistent with the use of “RUNNER” in the env variable.

[shykes]

Paging @aluzzardi, @sipsma, @vito, @kpenfound, @jlongtine, @gerhard, @d3rp3tt3 @jpadams :)

[sipsma]

If we have an engine that is capable of running for a long time and handling multiple clients, then we inherently have an engine that is capable of running ephemerally and handling only one client, right? The second case is just a subset of the first afaict.

If however our only mode of operation is to always run ephemerally and handle only one client, that seems pretty unnecessarily limiting. Are there any pros of exclusively running ephemerally vs. supporting long-running "stateful" engines that can also be executed as one-off instances? I can think of some short term pros like possibly making it easier for us as Dagger devs to implement version matching between client<->server, but in the long run those pros diminish and eventually disappear.

[shykes]

If we have an engine that is capable of running for a long time and handling multiple clients, then we inherently have an engine that is capable of running ephemerally and handling only one client, right? The second case is just a subset of the first afaict.

In theory, yes.. but in practice no (I think), because of caching. An ephemeral engine means an ephemeral buildkit state as well. Which means the engine has to redownload all content for all cache hits once per session. That results in either less cache hits, or more cycles and IO spent downloading cache. Either way pipelines run slower in ephemeral engines unless we engineer their caching specifically for ephemeral.

[sipsma]

In theory, yes.. but in practice no (I think), because of caching. An ephemeral engine means an ephemeral buildkit state as well. Which means the engine has to redownload all content for all cache hits once per session. That results in either less cache hits, or more cycles and IO spent downloading cache. Either way pipelines run slower in ephemeral engines unless we engineer their caching specifically for ephemeral.

Right but even in a long-running engine the cache is not stateful in the same way that e.g. a database is, it's just best effort. The local cache gets automatically pruned once it takes up too much disk space (with the pruning policy being configurable, even though we haven't exposed that yet), and if you have very little disk space it's possible that the vast majority of the cache would get pruned after each run, despite the engine being long-running and "stateful".

Similarly, even if you have a big local cache, you still benefit from having a super optimized + intelligent remote cache because in many use cases there's gonna be stuff that other engines have run and remotely cached that your particular engine has not run yet.

That compounds even further when you start to consider that "having a big local cache" does not always come cheap. E.g. on AWS you either pay extra for beefier instances w/ nvme storage or you pay extra-extra for fast EBS drives, which are charged hourly for as long they exist, whether or not they are being used or are even attached to any ec2 instance. So even for long-running engine cases, the better the remote caching the more money you can potentially save (with specifics depending on the use case and pricing of everything involved of course).

The overall point being that while the ephemeral case has really strong motivation to optimize remote caching, the long-running case also benefits significantly from an optimized remote cache too, even if it's of a somewhat smaller magnitude than the ephemeral case.

---

So I don't currently view this is a fork-in-the-road decision in terms of our architecture. We should absolutely optimize remote caching no matter what, it will benefit both long-running and ephemeral case.

Then, when it comes to what we recommend to users, we have both options available and working well to choose from for whatever makes sense in their particular execution environment, requirements, opinions/desires, etc.

In fact, I think there's honestly a whole spectrum between ephemeral and long-running, all of which can make sense in different use cases.

The extreme end on the ephemeral side is an engine that spawns just-in-time for one single workflow with no local cache and shuts down right after (managed GHA is pretty much this). The extreme end on the long-running side is an engine that just never shuts down and has a big local cache (whatever that may cost).

But in between you have something similar to ARC+Karpenter where there can be e.g. a "baseline" of a couple engines that run for a pretty long time, but when demand increases the system autoscales in some extra spot instances for you that last until the demand dies down and they are shut down.

If we just optimize the remote caching, and in the future start tossing in stuff like distributed builds where the engines can coordinate between in each and share load, we become agnostic to concerns like how long the engine runs and provide good coverage to the whole spectrum of different execution environments.

[marcosnils]

In fact, I think there's honestly a whole spectrum between ephemeral and long-running, all of which can make sense in different use cases.

This sentence summarizes pretty much my thoughts. My opinion is that I don't think we should take a one-way direction, and offer support for both models. There's always going to be "that user" which really needs / wants to use Dagger but because of architectural reasons (stateless and/or stateful) they can't.

Again, it doesn't seem to me that the current state of CI deployments have converged towards an award winning solution. Basically the same thing that happens with Monoliths/Microservices and Cloud/On-prem. There's always going to be users in both sides and if we can offer both with justified motives, why not do it?

[shykes]

If however our only mode of operation is to always run ephemerally and handle only one client, that seems pretty unnecessarily limiting. Are there any pros of exclusively running ephemerally vs. supporting long-running "stateful" engines that can also be executed as one-off instances? I can think of some short term pros like possibly making it easier for us as Dagger devs to implement version matching between client<->server, but in the long run those pros diminish and eventually disappear.

I think there are several benefits of a stateless engine (as well as drawbacks of course).

I set out to write them down but got interrupted for today @sipsma . I will finish my thought tomorrow.

[shykes]

Update. I just had a live conversation with @marcosnils and @sipsma which partially melted my brain. I will now attempt to use what is left of my brain to write down a few thoughts on this topic.

Benefits of stateless engine

I see several benefits to a stateless engine:

1. It simplifies the versioning matrix

If the engine is stateless, it can be spawned by the CLI. That collapses the "engine version" dimension into the "CLI version" of the matrix. Meanwhile Project Zenith is working on collapsing "SDK version" into "CLI version".

In other words, if you combine a) Project Zenith with b) making the engine stateless, we go from a 3-dimensional versioning matrix (bad) to no matrix at all: the CLI version becomes the Dagger version, period.

2. It simplifies the operational model

Right now there is not one, but two completely different operational models for Dagger, depending on how your CI is setup:

• On ephemeral VMs (either managed CI platforms, or self-hosted microvm-based platforms like Actuated.dev): the engine is short-lived, spawned either from the dagger CLI, or by a custom script using docker, podman or nerdctl. This is all configured from the CI configuration in the application repository.
• On long-lived machines (self-hosted VMs or bare metal machines, typically combined with a container runtime like containerd and/or kubernetes): the engine is longer-lived, spawned separately by the machine or cluster administrator. This is done separatelty from the CI configuration in the app repo.

As a result, the answer to many important questions like what's the architecture? What's the lifecycle? Who's in charge is "it depends".

If the engine is always ephemeral, then the operational model would be simplified, and we could give more clear cut answers to those questions. This would make Dagger easier to understand and trust.

Specifically, operating Dagger would always be like it is today on ephemeral VMs: the engine is always spawned from the CLI; it's always updated in lockstep with the CLI; and it's all configured from the CI configuration in the app repo.

3. It makes it easier to adopt Dagger

One of the mistakes Tekton made (in my opinion) is that it's a good CI product that is too tied to Kubernetes. Kubernetes is infrastructure software, but CI is fundamentally a developer-facing tool. By tying CI too much to the underlying infrastructure, Tekton makes it hard for developers to adopt. We should be careful to avoid that mistake. Making the engine ephemeral reduces the surface area of dependency on the underlying infrastructure.

In other words, Dagger should run great on your infrastructure. But it should not become part of your infrastructure.

The need for an infrastructure substrate

As discussed with @sipsma and @marcosnils, on some infrastructure, running an ephemeral Dagger engine is not possible without running some sort of stateful service. So far we have assumed the solution was to run the Dagger engine itself as that stateful service (hence the bifurcation of operational model we discussed above). I propose instead that there should be a lower level infrastructure substrate - something that acts as "polyfill" for the underlying infrastructure so that it can then correctly run the same ephemeral engine.

Crucially this infrastructure substrate should not include buildkit. 90% of buildkit is high-level logic that belongs in the engine, coupled with the graphql server as @sipsma is currently doing. The remaining 10% is low-level plumbing to a) run containers (via containerd/runc) and b) move data to and from containerd. That 10% of plumbing should be gradually spun out of the dagger and buildkit codebase (not an easy task) and spun out into a separate component, with its own name. that component is what infrastructure teams would run on their long-running VMs, baremetal machines, etc. Basically containerd + a basic data layer.

We started discussing incremental steps towards that. Starting with perhaps changing the kubernetes daemonset, to run a "bootstrap containerd" instead of a long-running engine. This would allow the Dagger CLI itself to spawn the dagger engine on that containerd, the same way it does on ephemeral VMs and laptops. That would not make the engine ephemeral per se, but it be a first step to align the operational model across all runner machines (because it would always be managed by the CLI).

@sipsma @marcosnils did I miss anything important from our discussion?

[shykes]

note, the above 👆 conversation was 2 days ago, but I got busy 😁

[sipsma]

I think the main point of prior confusion that was clarified by the live discussion was that I previously thought ephemeral+stateless engine was referring to the CLI being always responsible for starting a privileged container executor (i.e. our default "auto-run in docker" hack, except we'd only support hacks like that).

But the goal isn't to eliminate long-running everything, it's just to make the long-running component not tied to a particular version of the dagger API, dagger-specific configuration, etc. while still retaining our current feature-set.

Once that was clear we were on the same page the whole time 😄

---

We started discussing incremental steps towards that. Starting with perhaps changing the kubernetes daemonset, to run a "bootstrap containerd" instead of a long-running engine.

I actually think we can+should jump immediately to doing this such that the changes are to our engine image, rather than the k8s daemonset, and thus benefit every user rather than just k8s users.

The outline being (in approximate order of implementation steps, can be done iteratively rather than all at once):

1. Switch our engine's executor backend from oci worker -> containerd worker
   • This means we have to include containerd in our engine image, which will increase the size a bit, but highly unlikely to be so much it's a real concern.
   • Also containerd may be a preferable backend to use anyways since we could benefit from the (small but growing) ecosystem of containerd runtimes.
2. Change the engine image's entrypoint to no longer be the dagger engine but instead to be containerd
   • When the image starts, it will start the engine as a container in containerd w/ the containerd socket mounted in.
   • Note that since containerd is the engine's backend, this change does NOT result in crazy container-in-container-in-container behavior; any containers created by the engine will actually be "sibling" containers to it in containerd. So it's the same level of nestedness as today.
3. Add support for clients (i.e. the CLI) to optionally start older/newer versions of the engine in containerd rather than just one single version
   • We'd likely want to do something like include the latest version of the engine in the image and then require that any other engine images be pulled at runtime, which feels like a reasonable tradeoff in terms of functionality and not bloating the image. But there's many other variations possible here too.
   • This is also the point where we'll need to start considering making the engine containers more short-running, which would make them available for cleanup and thus prevent versions from accumulating indefinitely.

That would technically just be a start to the overall grand plan, but it's extremely appealing in that

1. It's surprisingly low-hanging fruit; probably somewhere between medium and large in terms of t-shirt sized effort
2. It would dramatically cut back on how often version mismatches between CLI<->Runner happen, likely to the point of those being very rare events.

[shykes]

@sipsma

I actually think we can+should jump immediately to doing this such that the changes are to our engine image, rather than the k8s daemonset, and thus benefit every user rather than just k8s users.

The outline being (in approximate order of implementation steps, can be done iteratively rather than all at once):

Even better if we can start shipping towards this early and often!

One important detail to consider: this will determine product packaging, as in: what are the parts of our product and what do our users call them?

• I think we reserve “engine” for the upper part, soon to be fully orchestrated by the CLI, soon the be stateless.
• We need a new name for the lower part: the substrate that acts as privileged container runner + data plumbing.
• Both need to be distributed as OCI images, at least until we can fully split buildkit in higher/upper parts (so: for a good while)
• Consequently we may need to distribute 2 distinct OCI images? One called “engine” (upper) and one called maybe “runner” for the lower part / substrate? Would be consistent with the use of “RUNNER” in the env variable.

[shykes]

FYI @d3rp3tt3 👆

[sipsma]

The "Runner" and "Engine" terms sound good to me, makes perfect sense.

I was originally hoping we could maybe still get away with a single image that's just invoked different ways (one entrypoint for the runner, one for the engine), but after thinking more it would probably actually be more confusing in the long run when talking about versioning and compatibility, so yeah distributing 2 images is the right route I think.

Just to outline a bit for anyone else catching up, the versioned components would be:

1. An SDK
2. The CLI (either dagger run or dagger session as called internally by SDKs by default)
   • After the recent re-architecture, this exists only to facilitate SDK<->Engine communication for the pieces of functionality that are too hard to re-implement in every SDK (e.g. filesync). GraphQL API server no longer lives here.
3. The Runner
   • OCI Image that contains pretty much just containerd (TBD if anything else besides vanilla containerd)
4. An Engine
   • OCI Image(s) that is started by the CLI in the runner and serves the graphql API that SDKs are clients to.

SDKs and the Engine are both expected to change often whenever the graphql API is updated, with the same versioning requirements that exist today. This is improved relative to today though since the introduction of the Runner essentially allows an SDK to spin up a compatible engine in any infra (with the CLI facilitating that spin up).

The CLI and Runner can change, but are not expected to very often, with backwards incompatible changes being especially rare. Changes requiring a version upgrade would only happen when there's an "internal protocol" change that impacts SDK<->Engine communication. So while it's important these are versioned, the versioning requirements will not need to be nearly as stringent as today (where we basically advise a one-to-one-to mapping between SDK+CLI+Engine).

[RealHarshThakur]

I've been hacking at setting up Buildkit on K8s in a way that it auto-scales to 0(using Envoy to queue connection reqs) but still caches.

We should absolutely optimize remote caching no matter what, it will benefit both long-running and ephemeral case

I am trying to figure out the best caching strategy too and 100% agree with this. The ideal scenario looks like p2p containerd caching backend(still figuring out how to do this) and the stop gaps look like local block storage or S3-fuse setups so far. So I really like that it aligns with long term goals of Dagger here :)

Having containerd socket mounted looks like a tempting solution as containerd may not be configurable in a lot of environments easily(managed clusters), but I'm not sure of the security implications and if it'll work in environments where policies are strict.