Repository navigation
Call Dagger Functions from an external client #5993
Description
Activity
- changed the title
[-]Zenith: support codegen of modules that's exported to non-module clients [/-][+]Call a Zenith module from non-Zenith code[/+]on Nov 21, 2023 Today its pretty clear how to use dagger modules from other modules, but its not obvious to me how to use dagger modules from "original dagger" code, I was experimenting with things yesterday and ran into some strange behavior that I cant fully describe.
One question that came out of this for me was what happens to
dagger runin a world ofdagger callUltimately, do we imagine there being a clear integration or onramp from current dagger to modules, or does this assume people rewrite everything as a module.
Ultimately, do we imagine there being a clear integration or onramp from current dagger to modules, or does this assume people rewrite everything as a module.
@levlaz Our current running assumption is that while the existing dagger use case of running SDK code directly on the host will continue to exist and be supported, it will become an "advanced use case" and not be the primary focus of attention. We'll encourage every one to write modules (and to convert existing code over to modules).
However, in our current plans there still would be occasional genuine use cases for non-module code so it does make sense to at least support calling module code from non-module code.
I'm mainly onboard with that because it's fairly straightforward afaict. Internally, the codegen that runs that allows you to call other modules from your module code is almost entirely the same codegen that we use to allow non-module code to call the core API today. So it just needs some slight adjustments to support the use case described in this issue.
- Specifically, we'd need to add a "hook" for this case that causes the modules the caller depends on to be automatically loaded when the client connects. Not a problem for non-module clients today because they only rely on the core API, which is always inherently loaded. And not a problem for module clients today because the dagger engine pre-loads the module's dependencies when the module starts. So this use case of non-module code depending on modules just requires one extra special case handling.
If it wasn't the case that this was fairly easy (or if we hit some unexpected bumps once we get to actually trying to implement this), then I'd start to question more whether we even should do it. But right now it's in a state of just easy-enough and potentially useful-enough that I feel we may as well.
One question that came out of this for me was what happens to dagger run in a world of dagger call
This specific subtopic will be decided on as part of this issue for simplifying the current CLI command set: #6229
To add my thoughts on this, I would like to see support to call / use modules from standard dagger code. I have a test suite that will either allow developers to connect to a local database when working locally, or a dagger based service when in the CI/CD pipeline.
At least while we migrate all users over to use dagger modules locally!
To expand on our setup, we use the
ExperimentalPrivilegedNestingflag within our main CI pipeline to wrap each module into its own pipeline. This main CI pipeline runsgo test ./...with a workdir of the module to be tested. Within the suite_test.go for that module, we initialise any dagger services that the individual module needs.The aim is to have fairly generic test code (maybe a module in the future) with all customisation left to the individual module developers.
Happy to continue the conversation here or on discord
Reacted by Erik Sipsma and I Putu AriyasaGoing to pick this one up in the near future. Have some new thoughts on how to approach:
- I think this could actually be implemented as a module itself, rather than as a 100% builtin feature to the engine. I really like the idea of moving as much functionality as possible to "user-space" (and thus making it a dog-fooding exercise) and keeping the core engine API minimal. I will probably start here as an experiment and see how it goes.
- e.g. (as a strawman)
dagger call -m github.com/dagger/daggerverse/host-codegen -o ./my-host-app --source-module github.com/foo/bar/my-mod --sdk goresults in code being exported to./my-host-appthat can be executed viadagger runand has go sdk bindings togithub.com/foo/bar/my-modincluded.
- e.g. (as a strawman)
- Theoretically, in addition to exporting source code executable with
dagger run ...we could also support exporting executable binaries/packages that can be executed directly on a user's host. The user would write code with the bindings to their module but then execute it directly on their host.- This offers a very intriguing alternative to
dagger callis tedious with too many flags+args #6723 and Make passing caller environment variables to modules more seamless #6112 - The appeal of this is that it avoids the problems of the user needing to have host-specific resources installed (the right interpreter, the right go compiler, etc.)
- This would be pretty obvious for Go, somewhat less so for python/ts/interpreted langs, though there's plenty of options available there (including super generic ones like
appimageet. al.) - This is likely too much to bite off all at once, but I may do some preliminary investigation to see how viable it is. Implementing as a module rather than engine-builtin (as mentioned above) also makes experiments like this a bit less risky.
- This offers a very intriguing alternative to
- I think this could actually be implemented as a module itself, rather than as a 100% builtin feature to the engine. I really like the idea of moving as much functionality as possible to "user-space" (and thus making it a dog-fooding exercise) and keeping the core engine API minimal. I will probably start here as an experiment and see how it goes.
If I read this correctly, the go side for point 1 would be:
import myMod "github.com/foo/bar/my-mod" func test() { myMod.functionCall(args) }
If so, at least for golang, that would be great. I'd personally prefer that then having to exec out to a sub processes
- changed the title
[-]Call a Zenith module from non-Zenith code[/-][+]Call Dagger Functions from an external client[/+]on Mar 25, 2024 I'm very interested in this. My use case is to put Dagger functions inside Temporal Activities. I was hoping it would be a generally easy coding task and thought it not being easy was because of my lack of experience with Go and more in particular working with different contexts. I guess it isn't so easy. 🤔
I believe I can get around this by using
exec.Commandto run Dagger CLI commands. It's just... well.... I was hoping to avoid that. Feels kinda icky. LOL! 😁Scott
You can do it with a GraphQL query, you just won't have the client bindings in
dagbeyond the core API, like you have when you use a dependent module.
Right now if you want to call functions available in a module, the only reasonable ways are from another module or from the CLI.
Otherwise you are on your own to load the module and construct graphql queries for it.
We can add support for codegening clients that include module deps but are intended to run on a host rather than as part of another module.
The implementation would just involve a hybrid of the existing "standard client codegen" and the module codegen, is most likely fairly straightforward other than deciding on how to expose it in terms of ux/dx.