shell: builtins vs. core API

[shykes]

Context

The shell introduces a new concept of builtins (.help, .login etc). Builtins can be contextually in a pipeline (for example .container | from alpine | .doc) which makes them very useful.

Problem

The relationship between shell builtins and core API is unclear: some builtins map directly to core API calls, others don't.

This creates the risk of a "shadow API". We should clarify the relationship that we want.

Solution

A solution is still being discussed. Please participate below.

Initial thoughts from @sipsma, on the possibility of builtins becoming a shadow API:

my main thought is that it would be better to avoid too much of that since, all else being equal, it's better to have functionality in the core API. Then it's available to e.g. module function implementations too (which use the various SDKs directly and don't go through dagger shell)

However, for functionality that absolutely only ever makes sense in the context of dagger shell, then it should be fine. So I guess that raises the question what sort of builtins you are thinking about specifically here

Initial thoughts from @shykes:

Empirically, when I'm tempted to add a builtin today, it's for one of the following reasons:

1. It involves accessing the client context (filesystem; env variables; commands; network) and therefore feels "weird and special" - stuff like Directory.export, Service.up, etc. It's part of the Dagger API today, but it's a special and weird part. So moving it out can be tempting.
2. It's "middleware" - a cross-cutting action that I want to inject into any type. For example .doc. I can't just add a doc function to every single type - it wouldn't belong there. It would require adding a new dimension to the schema.
3. It's faster to iterate. Sometimes I want to add polish on an existing call. Or try a new syntax for doing the same thing. Or add a bunch of small but useful calls. I feel a freedom and a velocity adding builtins to the shell, that I just don't know how to get when contributing to the core API. There's the complexity of the code; the overhead of building and testing; the bikeshedding and having to argue with lots of people; the concerns about breaking changes; etc. Below a certain size of change, I won't even bother with the overhead.
4. The API got bloated. In the early days we carefully bikeshedded every detail of the API, with the assumption that it was user-facing. Over time we started treating it more like a hidden implementation detail. As a result, we can't just expose the core API as-is as shell builtins. It's too bloated. So no matter what, some hiding has to happen. Then from there, it's a slippery slope. It used to be the API itself was treated like a self-sufficient UX domain - you designed it end-to-end so that interacting with it would make sense. Now that it's not the case, I kind of need a another self-sufficient UX domain.

[sipsma]

One analogy that seems potentially useful here is between systems shells like bash and the Linux kernel:

• dagger shell is obviously bash in that it's a userspace wrapper around low level syscalls
• the engine + core API are the kernel and syscalls

From that sort of perspective, it makes sense to me that there's going to be cases where it's faster+easier to just "do it in userspace" and add functionality to the shell. But some features either have to be in the engine/core or might be better there because it can be more optimized/used in more use cases/etc.

It also may often make sense to start with something in the shell and then move it to the engine API later once a need for it is proven there.

Not a flawless analogy, but might be a useful framing for thinking about this sort of stuff. Either way I'd say that:

• Let's not block shell features for the sake of purity, if it's faster to get going there then let's do it
• We can look at putting features in the engine on a case-by-case basis and can also migrate shel features to "engine native" features at a later time if that makes sense
  • I suspect we'll end up hitting cases like "I can do this cool thing in the shell but not from my module code"; that would be a good time to consider either spinning out the feature to the core engine or perhaps even to a module implementation that's depended on by the shell

[helderco]

I agree with @sipsma, but currently that's cobra commands, not shell builtins. There's currently 3 shell builtins that are proxying cobra commands and these could be in core as much as possible:

dagger/cmd/dagger/shell.go

Lines 1711 to 1716 in 4c40887

	h.addBuiltin(
	// TODO: Add uninstall command when available.
	cobraToShellCommand(moduleInstallCmd),
	cobraToShellCommand(loginCmd),
	cobraToShellCommand(logoutCmd),
	)

Each of those is a single line of code that can be removed if they end up in the core API, and they'd still be there since core functions appear as a built-in right now.

The rest is just very shell specific helpers or wrappers:

• .config - sets a module's constructor arguments
• .doc, .help, and .deps - these can be merged and output a CLI specific --help kind of thing
• .load, and .use - they're like calling multiple dagger call with different -m flags, there's also a lot of introspection being loaded into memory at runtime, to compensate for the lack of client bindings.

We're thinking about adding .print, .json, and .list (names not set in stone), but they're also CLI specific helpers.

I any case, for the future, I agree it's best to move things to the core API if it makes sense.

I also think that Query may have too much, but that could just be organized differently. Like engine stuff you place in the Engine object. A login and logout could be placed in a Cloud object. All the typedef stuff could be under a common object too. All the SDK specific stuff could be marked as internal so it doesn't have to be hardcoded in the CLI (current function call, etc). Similar thing with loadXXXFromID, but marked as hidden so we can decide not to show by default but still make it available if accessed directly (or listed unfiltered like with .core in the shell).

[shykes]

This is addressed by #9035 .