Call Dagger Functions from an external client

[sipsma]

Right now if you want to call functions available in a module, the only reasonable ways are from another module or from the CLI.

Otherwise you are on your own to load the module and construct graphql queries for it.

We can add support for codegening clients that include module deps but are intended to run on a host rather than as part of another module.

The implementation would just involve a hybrid of the existing "standard client codegen" and the module codegen, is most likely fairly straightforward other than deciding on how to expose it in terms of ux/dx.

[levlaz]

Today its pretty clear how to use dagger modules from other modules, but its not obvious to me how to use dagger modules from "original dagger" code, I was experimenting with things yesterday and ran into some strange behavior that I cant fully describe.

One question that came out of this for me was what happens to dagger run in a world of dagger call

Ultimately, do we imagine there being a clear integration or onramp from current dagger to modules, or does this assume people rewrite everything as a module.

[sipsma]

Ultimately, do we imagine there being a clear integration or onramp from current dagger to modules, or does this assume people rewrite everything as a module.

@levlaz Our current running assumption is that while the existing dagger use case of running SDK code directly on the host will continue to exist and be supported, it will become an "advanced use case" and not be the primary focus of attention. We'll encourage every one to write modules (and to convert existing code over to modules).

However, in our current plans there still would be occasional genuine use cases for non-module code so it does make sense to at least support calling module code from non-module code.

I'm mainly onboard with that because it's fairly straightforward afaict. Internally, the codegen that runs that allows you to call other modules from your module code is almost entirely the same codegen that we use to allow non-module code to call the core API today. So it just needs some slight adjustments to support the use case described in this issue.

• Specifically, we'd need to add a "hook" for this case that causes the modules the caller depends on to be automatically loaded when the client connects. Not a problem for non-module clients today because they only rely on the core API, which is always inherently loaded. And not a problem for module clients today because the dagger engine pre-loads the module's dependencies when the module starts. So this use case of non-module code depending on modules just requires one extra special case handling.

If it wasn't the case that this was fairly easy (or if we hit some unexpected bumps once we get to actually trying to implement this), then I'd start to question more whether we even should do it. But right now it's in a state of just easy-enough and potentially useful-enough that I feel we may as well.

One question that came out of this for me was what happens to dagger run in a world of dagger call

This specific subtopic will be decided on as part of this issue for simplifying the current CLI command set: #6229

[DMajrekar]

To add my thoughts on this, I would like to see support to call / use modules from standard dagger code. I have a test suite that will either allow developers to connect to a local database when working locally, or a dagger based service when in the CI/CD pipeline.

At least while we migrate all users over to use dagger modules locally!

To expand on our setup, we use the ExperimentalPrivilegedNesting flag within our main CI pipeline to wrap each module into its own pipeline. This main CI pipeline runs go test ./... with a workdir of the module to be tested. Within the suite_test.go for that module, we initialise any dagger services that the individual module needs.

The aim is to have fairly generic test code (maybe a module in the future) with all customisation left to the individual module developers.

Happy to continue the conversation here or on discord

[sipsma]

Going to pick this one up in the near future. Have some new thoughts on how to approach:

1. I think this could actually be implemented as a module itself, rather than as a 100% builtin feature to the engine. I really like the idea of moving as much functionality as possible to "user-space" (and thus making it a dog-fooding exercise) and keeping the core engine API minimal. I will probably start here as an experiment and see how it goes.
   • e.g. (as a strawman) dagger call -m github.com/dagger/daggerverse/host-codegen -o ./my-host-app --source-module github.com/foo/bar/my-mod --sdk go results in code being exported to ./my-host-app that can be executed via dagger run and has go sdk bindings to github.com/foo/bar/my-mod included.
2. Theoretically, in addition to exporting source code executable with dagger run ... we could also support exporting executable binaries/packages that can be executed directly on a user's host. The user would write code with the bindings to their module but then execute it directly on their host.
   • This offers a very intriguing alternative to dagger call is tedious with too many flags+args #6723 and Make passing caller environment variables to modules more seamless #6112
   • The appeal of this is that it avoids the problems of the user needing to have host-specific resources installed (the right interpreter, the right go compiler, etc.)
   • This would be pretty obvious for Go, somewhat less so for python/ts/interpreted langs, though there's plenty of options available there (including super generic ones like appimage et. al.)
   • This is likely too much to bite off all at once, but I may do some preliminary investigation to see how viable it is. Implementing as a module rather than engine-builtin (as mentioned above) also makes experiments like this a bit less risky.

[DMajrekar]

If I read this correctly, the go side for point 1 would be:

import myMod "github.com/foo/bar/my-mod"

func test() {
   myMod.functionCall(args)
}

If so, at least for golang, that would be great. I'd personally prefer that then having to exec out to a sub processes

[shykes]

User request: https://discord.com/channels/707636530424053791/1221631015743918121

[smolinari]

I'm very interested in this. My use case is to put Dagger functions inside Temporal Activities. I was hoping it would be a generally easy coding task and thought it not being easy was because of my lack of experience with Go and more in particular working with different contexts. I guess it isn't so easy. 🤔

I believe I can get around this by using exec.Command to run Dagger CLI commands. It's just... well.... I was hoping to avoid that. Feels kinda icky. LOL! 😁

Scott

[helderco]

You can do it with a GraphQL query, you just won't have the client bindings in dag beyond the core API, like you have when you use a dependent module.