Docs: future DX, version 4

[shykes]

4th iteration of describing a DX. This continues from #74, and incorporates ideas from a testing session with @samalba.

[samalba]

It'd be more intuitive to have a source being a directory. Here build.sh will need access to its own directory I guess. So I would specify "." as the source, and build.sh would be an argument of the "bash sdk", like "entrypoint" or "script".

[shykes]

The idea was to allow source being either a file or directory, for simplicity. Easy to try a few different formats though.

[dolanor]

In the case of it is a file, would . be imported by default, then?
I agree with nothing else, this build.sh would be not so interesting as a pure bash function.

[shykes]

I agree with nothing else, this build.sh would be not so interesting as a pure bash function.

Didn't understand that, sorry.

[dolanor]

well, this script by itself is useless without any IO. I bet it access some source files, it uses another binary to build, or minifies assets, using some options in an env var.

I guess the IO (or the client, in the Europa version) is what is coined the privileges here, and if we put workdir: true, it will automatically mount/copy the directory where this build.sh file is stored in? Unless it is the workdir of the CLI, which will depend where you execute your workflow from. Which is it?

[shykes]

well, this script by itself is useless without any IO. I bet it access some source files, it uses another binary to build, or minifies assets, using some options in an env var.

Yes, correct.

I guess the IO (or the client, in the Europa version) is what is coined the privileges here

Yes also correct.

if we put workdir: true, it will automatically mount/copy the directory where this build.sh file is stored in?

No, that may be useful in the future (similar to core.#Source) but unspecified for now.

Unless it is the workdir of the CLI, which will depend where you execute your workflow from. Which is it?

Either the workdir of the CLI, or the root of the project directory. Not sure yet, but leaning towards the latter. If it's the latter, it may be an implicit privilege: it seems reasonable that every workflow would have access to its project dir (like context directory for docker build).

[dolanor]

Unless it is the workdir of the CLI, which will depend where you execute your workflow from. Which is it?

Either the workdir of the CLI, or the root of the project directory. Not sure yet, but leaning towards the latter. If it's the latter, it may be an implicit privilege: it seems reasonable that every workflow would have access to its project dir (like context directory for docker build).

What to do in case of a giant monorepo where the build/run context would be too big for only a subset of what is used during the run?

[shykes]

What to do in case of a giant monorepo where the build/run context would be too big for only a subset of what is used during the run?

I guess place your project file in the relevant subdirectory (similar to dockerfile and docker-compose.yaml)

[shykes]

This is by no means finished, but I think we have consensus that it's directionally correct. I'm going to merge as a checkpoint, then we can continue in follow-up PRs, OK with everyone watching this?

[samalba]

Given that an SDK is used by an extension to "extend" the API, I don't why a workflow needs target an SDK. As you wrote above:

When a workflow is run, it simply queries the Dagger API

At the point, the extensions are already loaded into the API.

[shykes]

Since workflows are run by cloak in containers, they need to be built also, just like extensions. The API for extensions and workflows is not the same, but they both require an SDK.

[shykes]

(The SDK is also needed to generate client stubs to talk to the API)

[samalba]

That's the part I find confusing. It's simpler to think of a workflow only on the client side: a query + host resources. On the other side, an extension + sdk live in a container.

[shykes]

That was my proposal in v2 :) But it has several problems which @sipsma can explain better than me.

One of them is that a single graphql query is not powerful enough to express all workflows. For example todoapp deploy needs to 1) get a netlify token from somewhere, 2) call yarn install then yarn build, 3) call netlify deploy with the results of 1 and 2. There is no clean way to do this in a single query. I proposed a "pipeline" which would be a sequence of several graphql queries, but that breaks other things, for example you can't use query parameters since there are now several queries: which one do you use? etc.

In the end @sipsma and Andrea (no mention to protect his vacation) asked "why not just run the workflow in code", and then "might as well run that code in cloak". And it all clicked for me.

/braindump

[samalba]

One of them is that a single graphql query is not powerful enough to express all workflows.

It should be considered a limitation by design to encourage people to move this complexity in extension with their language of choice, instead of carrying complex gql queries.

I think it's much better to have a complex workflow with a lots of dependencies in an extension, that way the corresponding workflow will have a very simple query (similar to the current todoapp example). And that's also a way to abstract gql for the end user, because a simple query could be expressed without gql (like I explored in #79).

[shykes]

There are several problems with this:

• Extensions can't access the host system, everything needs to be passed explicitly, so users will probably write wrapper shell scripts anyway (equivalent to workflows, but worse DX)
• Extensions are harder to write: you need to write a graphql schema, generate code etc. Bash may not be powerful enough to implement an extension, so you won't be able to use the bash SDK. This makes the learning curve steeper.
• We can still encourage moving complex logic into an extension. But it's much easier to adopt dagger if you're not forced to do it right away.

[shykes]

@samalba to be clear I'm asking myself the same questions as you. I consider this part to be not fully resolved. But I'm glad we have a placeholder design that we can try, it will help us iterate IMO

[sipsma]

I think that it is useful (at least for now) to have two conceptual "boxes", one for workflow and one for extension, but I also have been wondering to what extent they will end up being different on a technical level. It seems plausible that a workflow could end up just being a special case of an extension. E.g. if we pretend workflows didn't exist as a separate concept and instead wanted to accomplish the same goals with just the extension concept, it may just mean supporting the following for an extension:

1. Being able to optionally skip defining schema+operations for the extension, in which case default ones are generated for you
   • This difference between workflow+extension in this respect could become even more blurry in the future if we have SDKs that implement code-first schema support, at which point defining the schema of an extension could be so seamless that it's almost as nice as the DX of workflows (where you can skip defining them)
2. Being able to grant the extensions privileged access to directly request client resources (basically, whether that schema with privileged apis is presented to them by the cloak server).
3. Being able to control whether the extension's schema is included when used as a dependency.
   • e.g. workflows aren't supposed to be importable as dependencies, but maybe this could just be a configuration option of an extension; essentially whether it is "private" to a project or not.

I think in theory all of those points could be made configuration knobs on an extension (and or sdks), at which point the concept of a workflow could just be a particular "template" of settings applied to the more general extension concept.

Whether this is actually works in practice, results in a better DX, etc. is all unknown to me at this point. I think it makes sense to start with them as separate entities in our API and then see if they end up so similar that we could just merge them together. Probably easier to merge them later than to start with them as the same thing and later disentangle them.

The good thing is that choosing one or another option here doesn't mean we will have invalidated all the docs and all the implementation. If we later decide to merge the concepts it's just a matter of migrating workflow configuration to be a special case extension configuration, which is more like a minor refactor than a "re-write everything" both in terms of docs+code.