Join GitHub today
Multi-Language Strategy and Interoperability #2430
Supporting multiple languages is an adoption boon. Pulumi itself is a new programming model and, if a user does not already know the language that they are using with Pulumi, it becomes a hugely daunting task to learn both a new language and Pulumi's programming model simultaneously. If Pulumi supports a language that a user already knows, it becomes much easier to understand how Pulumi works and how to use it effectively.
However, we have found that the maintenance cost of multiple language frontends for Pulumi is very high. We have put a large amount of engineering effort towards bringing our Python support to feature-parity with NodeJS, as @Frassle also has done for .NET. As it stands today it is difficult for us as a small group of engineers to scale effectively when our efforts must be spread out and duplicated across languages.
Furthermore, Pulumi users and those of us working at Pulumi alike have built a number of useful high-level libraries. (1 2 3 4). These libraries encapsulate into reusable components tasks that are otherwise hard to do, such as creating EKS clusters, creating AWS best-practice network topologies, and orchestrating Docker builds. It is clear that part of Pulumi's "magic" is the ability to author reusable components in a real programming language and be able to publish them through normal code push mechanisms such as NPM or PyPI.
Having said all of this, a clear contradiction in the "Pulumi story" arises: reusable components can only be used if the component is implemented in the same language that a user is using with Pulumi. A user today writing Pulumi programs in Python can't use any of the above linked packages and must re-write them wholesale in order to get any value from them. We are acutely aware of this issue.
We do not believe that the "status quo" is sustainable at its current rate. We do not have the engineering bandwidth to manually duplicate all high-level libraries that we provide across all supported languages. Even if we did, the combinatorial explosion of language-library pairs makes it harder and harder to add additional languages to Pulumi as Pulumi's ecosystem as a whole grows.
In the M21 timeframe (the next 5 weeks), we plan on pursuing a prototype for "multi-language interop" - the ability for a component authored in one language to be used naturally in another language. The goal of this prototype is to write a Pulumi program in Python that creates an EKS cluster using the
In the short-term, we are not going to put any engineering effort into Go language support for Pulumi. The hope is that, by pursuing a prototype, we will unlock a path to multi-language interop that will reduce the cost of adding additional languages to Pulumi so that we can once more accept additional languages into the Pulumi ecosystem.
At the end of the M21 timeframe, we will evaluate the R&D effort and make a decision about Pulumi's multi-language story. The various axes that we will consider include:
The are multiple options on the table with regards to Pulumi's multi-language story going forward, which we must decide on:
In M21, @swgillespie will pursue a prototype that allows language hosts to launch other language hosts, remote objects across process boundaries, and launch instances of libraries in those child language hosts. The goal of the prototype is to allow for the invoking of constructors and the invoking of functions across language boundaries. We have put a lot of thought into candidate designs for a prototype and this design is the one that has shown the most promise.
The goal of the prototype is to instantiate a
If the prototype is successful and we choose to productize it, the goal will be to produce a way for us as Pulumi engineers to author libraries that can be used in languages other than the implementation language. As it matures, the interop system may also be usable by Pulumi users so that they can author their own components.
I know statically typed languages aren't really in your wheelhouse atm, but I couldn't help noticing the similarity to xlang.
Couldn't you just consider terraform as an internal pivot format for infrastructure description on which all lang could interrop? Did a POC last year (https://github.com/pchalamet/NTerraform): that was based on terraform providers to generate a lang bridge. Idea was to implement a type provider then in f# above but never really had time to go beyond this after vacations. Authoring experience is far better when you are taken by hand in the IDE and it's far better than yaml.
But yes, in contrast, this requires a lot of code generators and requires to extract metadata from terraform providers. But this can be automated and made available to most lang and looks like it's no big deal - it's just a matter of build steps and caching I would say (not saying it's easy). Drawbacks are you can't natively interrop with functions as pulumi requires it. So couldn't functions be pushed to well, just functions (azure functions for eg) or a docker? Pulumi could take care of all this plumbing with much benefits for users.
I do no have much to bring to the table, sorry. I'm just curious about your prototype and how you will solve this!
xlang is a general-purpose remote object protocol, which (while useful) is far more powerful than we need and is generally counter to our goals:
xlang (and it's predecessors, WinRT and COM) are very heavyweight. An explicit design goal is to produce something that is as lightweight as possible.
We do not want to tie ourselves to Terraform in the long run. We already do not use Terraform for our Kubernetes provider. In general, the fact that Pulumi uses Terraform providers at all is an implementation detail.
We already do this: https://github.com/pulumi/pulumi-terraform. We have a tool called
Hi all, here's an update:
Our M21 timeframe is wrapping up today, and with it the R&D experiment that I mentioned in the OP. The primary goal of this exercise was to produce a prototype that was able to, from within a program written in Python, interoperate with the
The next step here is to move on to the decision-making portion of the task that I pointed out in the OP. In the M22 timeframe we're going to take a look at where we ended up and make a judgement call about what to do next. The plan hasn't changed since I wrote the OP.